feat(campaign): family-GEPA infrastructure + the selection-dominance negative — the grader is the bottleneck - #540
Conversation
tangletools
left a comment
tangletools
left a comment
There was a problem hiding this comment.
✅ Auto-approved drewstone PR — fc7dac5e
This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.
tangletools · auto-approval · reason: drewstone_author · 2026-08-04T01:59:14Z
tangletools
left a comment
tangletools
left a comment
There was a problem hiding this comment.
🟢 Value Audit — sound
| Verdict | sound |
| Concerns | 2 (1 low, 1 weak-concern) |
| Heuristic | 0.0s |
| Duplication | 0.0s |
| Interrogation | 182.6s (2 bridge agents) |
| Total | 182.6s |
💰 Value — sound
Extends an established iterating GEPA campaign script with a new oht2-family experiment mode plus an honest negative result, reusing all library primitives and adding only campaign-local split/budget logic in the repo's grain.
- What it does: Adds
--scenario-family oht2to scripts/gepa-analyst-campaign.ts: runs GEPA prompt-optimization over mixed OpenHands+Terminus2 labeled-positive dev rows with per-family stratified sampling (12 train + 8 selection per family), commits the resulting traj_id split to benchmarks/trace-analysis/oht2-family-gepa-20260803/split-plan.json as a GEPA-SPENT record, and re-derives + drift-checks that split o - Goals it achieves: (1) Test whether the analyst prompt generalizes to longer/harder oht2 trajectories on tuning-legal data. (2) Make the data split auditable and tamper-evident — sha-pinned labels + a projection-based drift check — so certification runs elsewhere can provably exclude spent rows even when the winner is rejected. (3) Surface macro-vs-micro metric divergence explicitly so a winner gaming the per-case g
- Assessment: Good change, in the grain of the codebase. The repo convention is campaign-specific parameters live in scripts/ drivers while reusable primitives live in src/, and this script composes exactly the right primitives via import: gepaOptimizationMethod, the pre-existing GepaOptimizationRecipe union (engine/sequential/adaptive-sequential/best-of/vote/omni all already defined in src/campaign/gepa-optimi
- Better / existing approach: none — this is the right approach. Searched for an existing generic split-plan / SPENT-record abstraction to extend (patterns: split-plan, splitPlan, SPENT, GEPA-SPENT under benchmarks/ and src/) and found none — this is the first campaign that needs a committed, drift-checked spent record, so campaign-local is correct. The omni recipe is already a first-class library type (src/campaign/gepa-optim
- Model: opencode/zai-coding-plan/glm-5.2
- Bridge attempts: 2
- Bridge warning: opencode/kimi-for-coding/k2p7: opencode: opencode error
🎯 Usefulness — sound
A well-wired extension of the canonical GEPA campaign script: new --scenario-family oht2 and --recipe engine|omni modes, a committed split-plan that pins labels-SHAs and traj_ids with drift-refusal, plus a real budget-overshoot bug fix — all riding existing framework primitives; the omni recipe is e
- Integration: Reachable and canonical. scripts/gepa-analyst-campaign.ts is the documented campaign entry point (benchmarks/trace-analysis/codetracebench-glm52-certified-20260801/README.md:18 cites it as the producer of the certified round-1 instructions). New imports all resolve to real exports: GepaOptimizationRecipe/GepaEngineRun from src/campaign/gepa-optimization-method.ts:77,91; loadPublicBenchmarkRows fro
- Fit with existing patterns: Follows the established single-entry-point pattern. Rides gepaOptimizationMethod (gepa-optimization-method.ts:188) with the framework's recipe discriminated union exactly as designed, reuses preparePublicAnalystBenchmark, scoreAnalystFindings, traceAnalystQualityJudge, adaptPublicBenchmarkFindings, and codeTraceBenchCase — no new primitive invented where one exists. Per-family stratified sampling
- Real-world viability: Happy and error paths both covered. The budget-geometry note (scripts:1026-1046) documents a real measured failure (gepa 0.1.4 checks budget once per iteration, so a callback cap equal to the upstream target overshoots mid-iteration, hits HTTP 429, and raise_on_exception kills the run — measured twice); the fix is one-iteration headroom via boundaryBurst = 2*REFLECTION_MINIBATCH_SIZE + selectionSc
- Model: opencode/zai-coding-plan/glm-5.2
- Bridge attempts: 1
🔎 Heuristic Signals
🟡 Cruft: console debug added scripts/gepa-analyst-campaign.ts
console.log(`[gepa-analyst-r${ROUND}] plan-only: wrote ${FAMILY_SPLIT_PLAN_PATH}`)
💰 Value Audit
🟡 Budget headroom is coupled to gepa 0.1.4's internal budget-check cadence [maintenance] ``
The boundaryBurst math (scripts/gepa-analyst-campaign.ts:1047-1052) repurposes maxConcurrency as a headroom knob because the bridge derates the upstream limit by maxConcurrency-1, and the callback cap = upstream + 2*minibatch + selectionSize is sized to one gepa-0.1.4 iteration overshoot. This is documented inline (lines 1026-1046) and measured twice per the PR, so the author is aware. The fragility: a gepa upgrade that changes reflection-minibatch handling or the per-iteration budget-check poin
What this audit checks
It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.
| Pass | What it asks |
|---|---|
| Heuristic | Vague title? Whitespace-only or cruft-bearing diff? (content signals only) |
| Duplication | Do added function/class names already exist elsewhere in the repo? |
| Value Audit | What does it do? What goal does it achieve? Is it good? Better architecture or already-exists? |
| Usefulness Audit | Does it integrate and fit? Will it hold up in real use and actually get used? |
Findings are concerns, not blocks — the human reviewer decides what to do with them.
|
tangletools
left a comment
tangletools
left a comment
There was a problem hiding this comment.
✅ Auto-approved drewstone PR — fc7dac5e
This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.
tangletools · auto-approval · reason: drewstone_author · 2026-08-04T02:27:14Z
--scenario-family oht2 retargets the analyst GEPA campaign at the TUNING-LEGAL OpenHands + Terminus2 dev pools: labeled-positive rows only, per-family seeded sample (12 train + 8 selection per family, disjoint), recorded in a committed split-plan that every run re-derives and verifies before spending. The optimized metric is the plain per-case judge composite; selection readouts report plain macro and pooled micro plus a per-family breakdown. --recipe engine runs one bounded GEPA engine; --recipe omni runs the official Omni composition on the same evaluation budget via the gepa-source venv. Family evaluations run at concurrency 3 (the provider seat 429-limits at ~5 concurrent); the mini-SWE round-2 path is unchanged and stays serial.
… one-iteration callback headroom gepa 0.1.4 checks its metric-call budget once per iteration-loop pass, so after a passing check it can still spend a full iteration (parent minibatch + proposed minibatch + accepted full-valset evaluation). When the callback's hard cap equals GEPA's own target, that boundary overshoot draws an HTTP 429 from the callback and raise_on_exception kills the run — measured on both smoke shapes. The family recipes now give the callback cap exactly one worst-case iteration burst of headroom above the upstream target, using the bridge's maxConcurrency derate as the headroom knob while engine parallel=false keeps execution strictly serial (also holding the z.ai seat far below its ~5-concurrent 429 limit). The family cost ceiling rises to $30 to keep ~2x margin over the larger worst-case call count, and the smoke prefers unsolved trajectories so it demonstrates the nonzero-score path. The mini-SWE round-2 recipe is byte-identical to its completed run.
fc7dac5 to
22011c4
Compare
2 participants
What this is
The family-GEPA round's infrastructure and its honest negative. The campaign script gains
--scenario-family oht2(mixed OpenHands/Terminus2 scenario pools from the tuning-legal dev labels, per-family stratified splits, committed GEPA-SPENT record) and--recipe engine|omni, plus a real bug fix found by the smoke phase: gepa 0.1.4 checks its budget once per iteration, so the callback's hard cap could be hit mid-iteration → run death (measured twice); fixed with one-iteration headroom + strictly serial metric calls (also keeps the seat under its measured ~5-concurrent 429 ceiling).The run and the verdict
56 evals, $6.42, winner changed — and rejected at selection by dominance: plain macro +2.4pp (0.163→0.187) while pooled micro collapsed 0.340→0.190 (tp 26→11, n=16 stratified scenarios). The winner games the per-case grader by improving small-gold cases and abandoning gold-heavy ones; the certification metric is micro, so certifying would fail by construction — $30 of sealed-instrument spend declined.
The finding that matters: both selection-metric shapes have now failed once each (round-2's gold-mass weighting overfit n=12; round-4's plain macro diverges from micro). The grader, not the optimizer, is the bottleneck — no further GEPA spend (including the prepared omni recipe) until a micro-aligned, non-overfitting selection metric exists. Companion finding from the same round: the framing prompt is family-divergent (+2.2pp Terminus2 / −8.7pp OpenHands, clean pairs both) — per-family prompt dispatch is the new ranked hypothesis.
Proof
Battery: lint (666 files), typecheck, suite 4721/4721, build all exit 0;
--planbyte-deterministic; both recipe smokes green on real provider calls; split-plan committed (24 train / 16 selection, labels shas pinned, runs refuse to start on drift). Run artifacts durable:~/bench-cache/ctb-20260801/fgepa-full-engine/. Ledger rows r4-e1 in.evolve/experiments.jsonl.