Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
3 changes: 3 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# A unified diff legitimately carries a bare space on blank context lines; the corpus whitespace gate must
# not read the reviewed proposal fixture as an authoring defect.
cells/**/review/*.patch -whitespace
16 changes: 14 additions & 2 deletions CATALOG.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
<!-- Generated by bin/generate-catalog.sh from cell KDL and evidence/*.tsv. Do not edit. -->
# Current eval catalog and evidence

This is the sole current corpus surface: **59 included cells** (19 model-free,
40 model-backed) and **4 retired exclusions**. Inventory fields are derived from the active
This is the sole current corpus surface: **71 included cells** (19 model-free,
52 model-backed) and **4 retired exclusions**. Inventory fields are derived from the active
KDL and executable judge scripts. Latest accepted PASS evidence remains distinct from the append-only run
history: a failed last run is visible without being advertised as accepted evidence, and cells with no
structured run row say so explicitly.
Expand All @@ -29,6 +29,18 @@ harness-native loader plus canonical hook file.
| `assignment-contract-hot-resources` | Codex | `gpt-5.6-sol` / medium | 2 | medium | `1200s` | 4 | — | **NO STRUCTURED RUN** |
| `context-resource-continuity` | model-free | — | 0 | none | `90s` | 4 | — | **NO STRUCTURED RUN** |
| `crash-ding` | mixed | `claude-sonnet-5+gpt-5.6-sol` / medium | 4 | high | `180s` | 5 | — | **NO STRUCTURED RUN** |
| `delegation-implement-claude-native` | Claude | `claude-sonnet-5` / medium | 1 | low | `900s` | 7 | — | **NO STRUCTURED RUN** |
| `delegation-implement-claude-st2` | Claude | `claude-sonnet-5` / medium | 2 | medium | `900s` | 7 | — | **NO STRUCTURED RUN** |
| `delegation-implement-codex-native` | Codex | `gpt-5.6-sol` / medium | 1 | low | `900s` | 7 | — | **NO STRUCTURED RUN** |
| `delegation-implement-codex-st2` | Codex | `gpt-5.6-sol` / medium | 2 | medium | `900s` | 7 | — | **NO STRUCTURED RUN** |
| `delegation-review-claude-native` | Claude | `claude-sonnet-5` / medium | 1 | low | `900s` | 5 | — | **NO STRUCTURED RUN** |
| `delegation-review-claude-st2` | Claude | `claude-sonnet-5` / medium | 3 | high | `900s` | 5 | — | **NO STRUCTURED RUN** |
| `delegation-review-codex-native` | Codex | `gpt-5.6-sol` / medium | 1 | low | `900s` | 5 | — | **NO STRUCTURED RUN** |
| `delegation-review-codex-st2` | Codex | `gpt-5.6-sol` / medium | 3 | high | `900s` | 5 | — | **NO STRUCTURED RUN** |
| `delegation-sweep-claude-native` | Claude | `claude-sonnet-5` / medium | 1 | low | `900s` | 5 | — | **NO STRUCTURED RUN** |
| `delegation-sweep-claude-st2` | Claude | `claude-sonnet-5` / medium | 3 | high | `900s` | 5 | — | **NO STRUCTURED RUN** |
| `delegation-sweep-codex-native` | Codex | `gpt-5.6-sol` / medium | 1 | low | `900s` | 5 | — | **NO STRUCTURED RUN** |
| `delegation-sweep-codex-st2` | Codex | `gpt-5.6-sol` / medium | 3 | high | `900s` | 5 | — | **NO STRUCTURED RUN** |
| `ding-mode` | Claude | `claude-sonnet-5` / medium | 2 | medium | `1200s` | 4 | — | **NO STRUCTURED RUN** |
| `ding-reply` | Claude | `claude-sonnet-5` / medium | 1 | low | `420s` | 2 | — | **NO STRUCTURED RUN** |
| `docs` | Claude | `claude-sonnet-5` / medium | 3 | high | `1200s` | 5 | — | **NO STRUCTURED RUN** |
Expand Down
35 changes: 35 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,41 @@ and include model-free regression tests for those oracle boundaries. These pre-p
evidence, not accepted corpus receipts; `CATALOG.md` remains authoritative and will show no accepted PASS until
a committed cell is rerun.

## Delegation-parity tournament

The twelve `delegation-*` cells form one matched tournament asking whether **st2-managed sub-agents are worse
than harness-native sub-agents** on real delegation work. It is a regression check on an existing fleet
decision, not a benchmark: the preregistered question, metrics, replication plan, falsification rule, and
limitations live in
[`evidence/delegation-parity-design-20260812.md`](evidence/delegation-parity-design-20260812.md).

| Task | `claude-st2` / `codex-st2` | `claude-native` | `codex-native` |
| --- | --- | --- | --- |
| Broad multi-file search (`sweep`, fan-out 2) | peer seats over the native bus | one seat, `Agent` sub-agents | one seat, `spawn_agent` sub-agents |
| Independent review of a diff (`review`, fan-out 2) | peer seats over the native bus | one seat, `Agent` sub-agents | one seat, `spawn_agent` sub-agents |
| Scoped implementation (`implement`, fan-out 1) | peer seats over the native bus | one seat, `Agent` sub-agents | one seat, `spawn_agent` sub-agents |

Within one task the four arms are byte-identical in task, held-out graders, frozen product repository,
deliverable contract, and timeout; only the delegation layer differs, and the outcome graders never learn the
arm. Each native arm has a same-harness managed counterpart, because the decision-relevant comparison is
within a harness family. Deliverables land in a shared `findings/` directory with per-delegate attribution, so
one mechanical grader reads a bus delegation and a native fan-out identically.

Two honest boundaries: the native arms necessarily run **unmanaged** — a folder eval launches the harness
directly, so the fleet's native-spawn refusal never fires — and native fan-out is **self-attested**. The only
mechanical guarantee there is negative: a native cell declares exactly one bus seat and its delegation judge
fails if a second mailbox ever appears. Timing is measured as bus latency between the kickoff receipt and the
coordinator's confirmation, because a single-seat folder eval never signals completion and therefore always
consumes its whole `max-timeout`.

Matching, grader discrimination, and the tasks' premises are proven for free, without starting a model:

```sh
bin/check-delegation-parity.sh
```

No arm has been run. `CATALOG.md` will show no accepted PASS for these cells until one is.

## Add or change a cell

1. Create exactly one `cells/<name>/<name>.kdl`.
Expand Down
1 change: 1 addition & 0 deletions bin/check-corpus.sh
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ bin/check-fixture-reset-terminal.sh
bin/check-harness-contract.sh
bin/check-vrs-scope-drift.sh
bin/check-vrs-variations.sh
bin/check-delegation-parity.sh
bin/check-weird-git-setup.sh
bin/check-preflight-closed-set-mutations.sh
bin/check-overnight-policy.sh
Expand Down
Loading