eval: compare st2-managed and harness-native delegation across 12 matched cells

[schickling]

Blocked on compoundingtech/st2#247 and compoundingtech/st2#248.
The design is preregistered and the free gate is green, but the tournament cannot be run for real yet — the
st2 message bus does not deliver in these catalogs. One cell was run for money and the result was void, not
failed (see One real run below). Do not spend the ~$120 until both upstream issues close.

The question

Are st2-managed sub-agents worse than harness-native sub-agents on real delegation work? This is a regression
check on an existing fleet capability, not a feature demo: if handing a scoped slice to a peer seat over the bus
produces fewer correct outcomes — or the same outcomes far more slowly — than letting a harness fan out to its own
sub-agents, that is a defect in the managed path and we want it measured rather than argued.

Four arms, and why four rather than three

cells/delegation-{sweep,review,implement}-{claude-st2,claude-native,codex-st2,codex-native}/ — 4 arms × 3 tasks
= 12 cells.

A three-arm shape (one managed arm plus two native arms, or one managed plus one native) confounds the delegation
mechanism with the model family. If claude-st2 underperforms codex-native, that difference is unattributable.
So every native arm has a same-harness managed counterpart, and every comparison the pass criterion makes is
within a harness: claude-st2 vs claude-native, codex-st2 vs codex-native. The mechanism is the only thing
that varies inside a comparison.

Within a task the four arms are byte-identical except for the seat declarations and the coordinator persona. The
managed arms declare worker seats and a coordinator whose persona names its delegates over the bus; the native arms
declare a single seat whose persona fans out to harness sub-agents. Task text, graders, fixture repository, frozen
baseline manifest, deliverable contract, kickoff target, and timeout are all shared bytes, and
bin/check-delegation-parity.sh proves it.

Three tasks

task	work	gating judges
sweep	a call-site audit: name exactly the modules that reach legacyTitle(), delivered as two attributed slices plus a summary	outcome, slices, isolation, delegation
review	review a supplied proposal patch against the repository it applies to	outcome, slices, isolation, delegation
implement	make slugs collision-safe while preserving old behaviour, with a mutation-valid regression test	outcome, suite, regression, slices, isolation, delegation

Each task ships one mechanical grader used by all four arms. The delegation judge takes an arm-kind argument
(st2 / native) so a single grader reads a bus delegation and a harness fan-out identically — the outcome graders
never learn which arm produced the deliverable.

Metrics and pass criterion

Gating judges produce a PASS/FAIL per run. A non-gating signal judge (judges/observe.sh) records descriptive
per-run metrics into a durable metric sink: kickoff timestamp, confirmation timestamp, derived bus latency, bus
message count, bus seat count, delegate labels, and the deliverable list.

Managed delegation is "worse" only if, in ≥2 of 3 tasks, the managed arm has strictly fewer PASS runs than its
same-harness native arm — or outcomes tie and its median bus latency exceeds 2× native.

Anything else is a null result. The criterion is preregistered in
evidence/delegation-parity-design-20260812.md so the threshold
cannot be chosen after the numbers land.

Cost

n=3 per cell → 36 runs. The one real run cost $3.36, so the full tournament is ~$120.

One real run — VOID, not failed

delegation-sweep-claude-st2 was run once for real. The result is void: the coordinator could not have used the
bus at all, because st2 message send hard-fails in that catalog.

The two "delegation" messages present in the run were hand-written files — named s1send and s2send, whereas
a real send gets a random suffix like ffxvrm. The agents, finding the bus broken, wrote inbox files by hand. So
the delegation judge graded a filesystem workaround, not st2 bus delivery. The run cannot speak to the
question the eval asks, in either direction.

Root cause chain

1. A fixture package.json contains {"type":"module"}.
2. st2 discovery admits any JSON with a top-level type key as an Agent Spec, so that file becomes a phantom
   agent (Discovery admits any JSON with a top-level type key as an Agent Spec, so an ordinary {"type":"module"} package.json becomes a phantom agent st2#247).
3. discovered.specs is therefore non-empty.
4. The catalog-less flat-bus fallback is skipped for every seat, because it is gated on that unrelated catalog
   state (The catalog-less flat-bus fallback is gated on unrelated catalog state: one stray package.json (or an empty .st2) un-addresses every seat in a compact-team eval st2#248).
5. ding and send resolve nothing; delegated messages sit unread.
6. The run burns its full 900s max-timeout.

Self-amplifying

st2 ls, st2 agents, st2 validate, and st2 doctor each create .st2, which permanently breaks send.
The agents ran exactly those commands to diagnose the broken bus — so the act of investigating the failure made it
unrecoverable.

Verified isolation breach

A stray send wrote repo/resources/inbox/ inside the audited fixture, which the isolation judge grades as
byte-identical to the frozen baseline. That judge can pass while the fixture is mutated. This is verified, not
theoretical, and it is the most serious finding here: an isolation judge that passes on a mutated fixture is not an
isolation judge.

What the run did prove works

Fixture copy, team boot, kickoff delivery, bus addressing, judge execution, the isolation judge's mechanics, the
signal judge's metric sink, and SPEC_DIR resolution. The coordinator persona worked: it delegated real scoped
slices to the right peers without doing the work itself, which is the behaviour the managed arms depend on.

The cell design is largely vindicated; the runner underneath it is not.

Disclosed defects

Three known problems, stated rather than hidden:

1. delegate_labels metric is poisoned. judges/observe.sh globs findings/*.md, which includes
   CONTRACT.md. The contract documents the attribution format with a placeholder
   (delegate: <the identity of the delegate that wrote this file>), so the metric reads the literal string
   <the in every arm. Gating is unaffected — grade.sh reads only slice-*.md — but the metric is unusable
   as shipped.
2. The design's max-timeout premise is false for multi-seat cells. The design argues st2 eval's wall clock is
   useless because cells always burn their max-timeout. That is wrong: the runner does exit early on a
   sup→requester confirmation post-dating a worker report. Once DING works, runner duration becomes a usable
   measure alongside bus latency.
3. Inherited ST_AGENT breakage (Corpus migration: 40 cells declare env { ST_AGENT "<team>.<seat>" }, which st2 now rejects at boot (the runner owns ST_AGENT) #75). The tournament copied
   the established seat pattern, which declares env { ST_AGENT "<team>.<seat>" }; st2 now owns ST_AGENT and
   rejects that at boot. This is an intentional upstream ownership change, not a bug, but it broke 40 of 59 cells
   corpus-wide (94 seat declarations) and these 12 cells inherit it. It should be fixed as one corpus-wide
   migration, not patched per-cell here — a partial fix would reintroduce exactly the harness confound the 4-arm
   design exists to remove.

Note also that bin/check-delegation-parity.sh prints 12 delegation-parity cells match within each task, but its
matching check deliberately does not compare the team { } seat blocks across arms, since those differ by design.
An asymmetric seat-env divergence between claude-st2 and codex-st2 would pass the gate. Read that PASS line as
"tasks, graders, fixtures, personas, timeout, and kickoff match", not "the cells are identical".

Structural limitation — what this eval cannot conclude

The native arms cannot run managed. A folder eval launches exec claude / exec codex directly, so
AGENT_LAUNCH_HOSTED is unset and the managed refusal never fires.

So this tournament compares managed delegation against UNMANAGED native delegation. It is not a clean
managed-vs-native contrast holding hosting constant, and it cannot answer whether the managed wrapper is what
costs anything. It bounds the claim to: does delegating over the st2 bus produce worse outcomes or materially worse
latency than a harness delegating to its own sub-agents, when the native side runs unmanaged.

What's in this branch

• 12 cells under cells/delegation-*/
• bin/check-delegation-parity.sh — a free, model-free gate proving arm matching, task premises, and grader
  discrimination against 20 planted failures plus tolerance positives
• evidence/delegation-parity-design-20260812.md — the preregistered design
• README tournament section, a docs/vrs/spec.md paragraph, regenerated CATALOG.md, and a .gitattributes rule
  so the reviewed proposal patch's blank context lines don't trip the corpus whitespace gate

Gate output

PASS: 12 delegation-parity cells match within each task and hold their graders out
PASS: the baseline suite is green, the reviewed proposal applies and stays green, and the frozen pre-change implementation matches
PASS: the sweep oracle accepts a complete answer in both bus shapes and two markdown styles, and rejects 8 planted failures
PASS: the review oracle accepts a complete review in both bus shapes and rejects 4 planted failures
PASS: the implementation oracle accepts a complete change in both bus shapes and rejects 8 planted failures
PASS: the observation recorder derives a bus latency, writes a durable row, and never gates
PASS: delegation-parity matching and oracle validity hold

Review asks

1. Is the pass criterion the right one, and is 2× the right latency multiple? Now is the time to argue — it is
   preregistered precisely so it can't be moved later.
2. Does the isolation breach warrant hardening the isolation judge in this PR, or is it purely an st2 fix?
3. Should the ST_AGENT migration land first as one corpus-wide change, with this branch rebasing onto it?