chore(skills): gen3 measurement lands as HOLD — verifier invalidated the promotion - #726
Merged
Conversation
One live surface in the tree: SKILL.md is the only skill text checked in; every prior generation's text is recoverable from git history via the sha256s pinned in generations/ and the new IMPROVE.md version-history table. baseline-v1.json and gen2.json move (git mv, history preserved) to generations/gen1-baseline.json and generations/gen2.json.
Both target the residual under-graphing failure (mission text dropped from the deliverable when the brief sounds cheap) without reusing any holdout brief phrasing. The scorer gains an explicit correctAnswerIsGraph check, equal-weight like every other expect key; checkIsMechanical and reason remain annotations, noted not scored. Also carries the improve.mts output path onto generations/gen1-baseline.json following the reorg.
… loop k=5 per case per surface, 7 train cases (5 prior + the two new mission-in-deliverable-cluster cases), holdout unchanged and prompt-leak asserted. v2 re-measured: train 0.817, holdout 0.600. v3: train 0.937, holdout 0.900 — mission-in-deliverable went 0/5 to 5/5. Gate ship: holdout strictly better, train +0.12, anti-over-graphing cases flat at 1.0. One transient router 503 recovered by the bounded per-cell retry (receipt in gen3.json); zero lost cells, worst-case-0 imputation leaves the verdict unchanged. Full per-rep tables, tally of v2 failing checks, and surface sha256s pinned in generations/gen3.json.
…promotion The script gate said ship (holdout 0.600->0.900, k=5, all cells present) and the arithmetic is correct. The verdict is not, for two reasons the verifier proved rather than argued: 1. Case-design contamination. The new train case that produced the gain is a template-level paraphrase of the holdout case it was built to fix, designed FROM that case's measured failure. Banned-phrase hygiene passed; the leak happened one level up, in case design. The entire +0.300 holdout improvement rests on that single case, which makes it evidence of targeting, not of generalization. 2. Scorer leniency. A live probe caught v3 authoring a graph runGraph refuses (invalid perWorker.maxIterations) while scoring 1.00 -- the case scores only the mission check. A graph the runtime rejects is not a graph. The v3 text even bans the option the author contract offers. SKILL.md reverts to v2 (sha 4c6615b6). The k=5 measurements stay: they are real evidence, including v2's holdout 0.600 at n=15 confirming gen2's 0.611. gen3.json now records the hold, the four reasons, and the five gen4 requirements. A held generation with its reasons on the record is the loop working, not failing.
tangletools
approved these changes
Aug 3, 2026
tangletools
left a comment
tangletools
left a comment
Contributor
There was a problem hiding this comment.
✅ Auto-approved drewstone PR — 343e1af4
This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.
tangletools · auto-approval · reason: drewstone_author · 2026-08-03T20:06:52Z
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Gen3 of the agent-graphs improvement loop, landing as a hold record, not a promotion.
SKILL.mdstays at v2.What happened
The script gate said ship — holdout 0.600 → 0.900 at k=5, all 100 cells present, arithmetic independently recomputed and correct. The adversarial verifier then invalidated the verdict:
runGraphrefuses (invalidperWorker.maxIterations) while scoring 1.00 — that case scores only the mission check. A graph the runtime rejects is not a graph. v3's own text bans the option the author contract offers.What this PR keeps
generations/reorganization (git-mv'd, history-following) and the version-history table, with gen3 recorded as HOLD and why.gen3.jsonnow carriesverifierHold: the four reasons and five gen4 requirements (validationError ⇒ 0 for graph decisions; author contract aligned with the skill text; blind-authored holdout refresh; revision-prompt inputs persisted; in-loop gate wiring fixed).A held generation with its reasons on the record is the loop working. Promoting a flattering number past a failed verification would be the failure.