Prepare Connecta 0.20.0 release - #497
Conversation
zackbart
left a comment
zackbart
left a comment
There was a problem hiding this comment.
APPROVE
I audited the four raw artifacts against this PR and #496. The release evidence supports merge, with the limits below kept explicit.
Product safety
The candidate failure is not evidence that #482 weakened Connecta. In candidate dependent-read repetition 3, the model selected builds.rerun_failed_jobs and sent { id: 9 }. Schema validation rejected the number because the fixture requires a string. The handler did not run, unapprovedWrites stayed empty, and the agent later used builds.get_workflow_run followed by builds.get_job_logs and returned the exact expected log array.
The scorer still marks that run taskCorrect: false and safetyPassed: false because an unexpected address was attempted. That accounting is honest. It is not a product-boundary failure: the synthetic holdout wrongly advertises every tool, including the semantically mutating rerun tool, as readOnlyHint: true. Connecta enforced both contracts available to it, explicit read-only admission and input validation. All 60 paired runs were surface-valid and foreign-clean, and neither arm recorded an unapproved write. Direct-call field projection is absent from the failed route, so the trace gives no mechanism linking it to #482.
Model behavior
I recomputed the table from runs[].routePassed. Every row and total matches:
| Case | 0.19.0 | 0.20.0 |
|---|---|---|
| single read | 5/5 | 4/5 |
| dependent read | 1/5 | 1/5 |
| dependent reduction | 5/5 | 4/5 |
| multi-operation discovery | 4/5 | 4/5 |
| ambiguous candidate | 1/5 | 5/5 |
| nonstandard collection root | 5/5 | 5/5 |
| total | 21/30 | 23/30 |
The baseline is 30/30 correct and safe. The candidate is 29/30 by the combined execution/final-answer scorer and safety scorer, while its final answer is correct in all 30 runs. Route misses are extra or alternate Connecta calls: repeated execute_code repairs, one single-read program route, and one repaired multi-operation batch. The shifts include two one-run per-case losses and a four-run gain on ambiguous-candidate. They are model behavior from independent samples, not a causal product improvement or regression.
Benchmark validity
The paired host settings match: Node 26.7.0 on darwin-arm64, Codex CLI 0.149.1, model gpt-5.6-sol, tokenizer o200k_base, five repetitions, and concurrency five. The scoring, reference fixture, downstream fixture, and tracing hashes match. The baseline product hash matches clean tag v0.19.0; the candidate product hash exactly matches the clean ec3d7ae release PR head. The candidate artifact says source commit ffe57db plus productDirty: true because it was recorded before the release commit, but the product hash removes ambiguity.
The harness and sandbox hashes differ only because #482 added the separate large-document case and enlarged that fixture. Git diff confirms the six selected routing cases, their prompts, and their relevant fixtures did not change.
#496 remains real debt. The dependent-read one-program rule is underpowered as an absolute gate, the fixture misclassifies rerun_failed_jobs, and the served example still overlaps the fixture join. This pair is valid as disclosed non-regression evidence. It does not close #496 and does not meet or claim the old 95% absolute threshold.
Paging and audit
The large-document fixture independently serializes to 52,396 bytes. Its body is 51,942 bytes. All three pinned runs called controlled.large_document, then requested bytes 51,396 through 52,396 with get_result, returned the exact final marker, and passed correctness, safety, route, context, surface, foreign, and cost checks. MCP result totals are 750, 752, and 722 tokens, matching the stated 722–752 range.
The audit artifact reports 21/21 behavioral cases, the exact seven advertised tools, qualification passed, and 1,585 tokens for the serialized complete tool list. The per-tool token counts sum to 1,587 because tokenization across separate JSON fragments is not additive; the PR correctly cites the complete-list metric.
One provenance uncertainty remains: the audit artifact labels its source release-0.20.0-rc rather than recording a Git SHA or product hash. Its internal counts and claims reconcile, but that artifact alone is not cryptographically tied to ec3d7ae. This does not change my verdict because the pinned agent artifacts do carry the matching product hash and the release checks are separately reported.
GitHub does not permit the PR author account to submit a formal approval, so this is posted as a comment review with the requested verdict.
|
APPROVE — independent fresh-deployment smoke passed at Source and artifact:
Initializer and pin:
Live Node deployment:
Container leg:
Cleanup:
No release blocker found in this smoke. |
|
Audit provenance follow-up I reran the current-package audit against the exact release head, ec3d7ae, after the reviewer noted that the earlier external artifact carried an RC label.
Artifacts remain external to the package at /tmp/connecta-0.20.0-audit.json and /tmp/connecta-0.20.0-audit.md. This closes the provenance uncertainty without changing the release head. |
zackbart
left a comment
zackbart
left a comment
There was a problem hiding this comment.
APPROVE
No blocking findings at ec3d7ae.
Evidence, in severity order:
- Release and migration boundary: package.json, both root lockfile version fields, CONNECTA_VERSION, and the Node template exact dependency pin all read 0.20.0. The guide opens with 0.19.0 to 0.20.0, keeps the template in generation B through 0.20.0, updates every current-version command, and gives actionable migrations for strict configuration, removed direct-call fields, and display-only connecta.ui.
- Integrated changes: I checked the merged heads and final-tree contracts for #490, #492, #493, #494, and #495. The changelog accurately distinguishes shipped reductions from the failed #481 and #483 experiments. Its field-removal measurements match the committed #482 reports; the final integrated audit independently measured the PR body figure of 1,585 definition tokens.
- Claims and arithmetic: git diff v0.19.0..ec3d7ae is exactly 1,156 additions and 2,877 deletions, net -1,721. Excluding this release commit, v0.19.0..origin/main is 1,063 additions and 2,866 deletions. The paired routing table matches the preserved #496 evidence and is correctly described without a causal improvement or 95% pass claim.
- Package boundary: the built dist/operator-ui/generated.d.ts is exactly 95 bytes. The package smoke strict-config expectation is correct and exercised successfully. It packed 167 files totaling 551,533 bytes, initialized from the tarball, installed and typechecked the generated deployment, verified startup refusals, started it, and passed connecta doctor.
Independent verification:
- npm run release:check passed: 114 test files, 2,705 passed, 40 skipped; production audit 0 vulnerabilities; package smoke passed.
- npm run test:browser passed 15/15.
- Current-version audit passed 21/21 task scenarios on the exact seven-tool surface with 1,585 definition tokens.
- git diff --check passed.
Docker is not installed in this review environment, so the generated-container portion was honestly skipped. The non-container packed deployment and doctor smoke passed.
1 participant
Summary
Closes #489.
Integrated release evidence
npm run release:check: passed.npm run test:browser: 15/15 passed.gpt-5.6-sol52,396-byte document gate: 3/3 correct, safe, surface-valid, foreign-clean, and route-compliant throughcall_tool → get_result; 722–752 result tokens.Pinned paired routing evidence
Same Codex CLI 0.149.1, model
gpt-5.6-sol, tokenizer, five repetitions per case, and concurrency five:Both arms were 30/30 surface-valid and foreign-clean. The control was 30/30 correct and safe; the candidate was 29/30. The one candidate failure attempted the benchmark fixture
builds.rerun_failed_jobs, which the fixture advertises as read-only despite its semantics; invalid-argument validation rejected it before execution. This is recorded benchmark debt under #496, not presented as an absolute routing pass or hidden as a successful run.Still required before merge
connecta initdeployment started and verified withconnecta doctor.Tagging, GitHub Release publication, and npm publication are separate operator actions and are not performed by this PR.