eval: prove one-turn inbox delivery

[schickling]

Summary

• add one focused mixed Claude/Codex cell for st2 #238
• run the identical canonical Agent Spec fixture at exact metadata-only #237 and bounded-body #239 heads
• measure provider delivery separately from model-originated discovery/action commands
• prove exact threaded replies, archives, burst handling, post-batch durability, versions, hashes, timing, usage, and cleanup
• keep evals Model-free A/B: hook-plus-idle DING versus aggressive delivery #57's generic PTY prompt/draft collision matrix out of this body-availability experiment

Treatment and driver boundary

The st2 binary is the only A/B variable.

arm	st2 head	Nix-built binary SHA256
baseline	1d06c4b263a7c5a2a6b8eec1f2e8c4fbea5e2edc	06997b2c63ddf58678c7a4d024de62d957faf6c2bdc56766c15501e7946c1338
candidate	c1a0f90dd4814ec3ce8067219530d7bd8723e191	6795c8539ca308fc49b439a3b6495d2bcd2ad4a5ede4cfe02dd53be2f2c67d81

Codex uses structured argv with deliver "app-server"; Claude uses the maintained SessionStart hook. Both exact heads consume the same canonical cell, keeping the acceptance surface on the Agent Spec/driver-facing declaration boundary ahead of st2 #162.

The cell invents no settle, claim, lease, cursor, provider opt-in, or eval-only delivery switch. It covers a two-message cold backlog for both providers plus a typed three-message Codex burst and one post-batch arrival.

Real-provider A/B

Tracked receipt: evidence/inbox-one-turn-provider-ab-runs-20260811.json

metric	baseline	candidate	change
held-out functional score	FAIL, 3/4	PASS, 4/4	exact-body outcome fixed
Codex total tokens	535,286	227,871	-57.43%
Codex recorded model-response events	17	7	-58.82%
Codex tool-call boundaries	11	4	-63.64%
Codex wall time	115.367 s	45.676 s	-60.41%
Claude token volume	367,011	319,857	-12.85%
Claude session duration	114.706 s	45.315 s	-60.49%

The baseline was body-blind: Claude explicitly discovered bodies, while Codex archived all six messages but replied with subject-derived summaries instead of the hidden exact tokens. The candidate replied on and archived every exact body, including the bounded burst and post-batch arrival.

The original provider executions completed, but the judge launcher used a cell-relative path and exited 127. The receipt records this rather than calling the invocation green. Both preserved catalogs were deterministically regraded with corrected graders at 3de9f7e2cffbe4d24da95d4422ed5f687c4b30f9: baseline returns the semantic FAIL and candidate returns PASS. Claude API-call count and total serialized prompt bytes were not persisted and remain null.

Free verification

At exact evals head 0e6b64ebdc652687926d43f4a41a955831ddd846:

• held-out grader synthetic control and preserved-catalog RED/GREEN regrade: PASS
• 115 KDL files parse with pinned kdl-rs 6.7.1: PASS
• candidate st2 semantically loads 60 cells and validates 12 nested catalogs: PASS
• model policy, event-first delivery, harness, fixture reset, run-history, generated catalog, privacy/PII, and diff gates: PASS
• bin/check-corpus.sh: no verdict at its first provenance gate because the corpus expects published runner 0fed14b; all subsequent free gates were run directly with the exact candidate binary

Related to #57. Companion proof for compoundingtech/st2#238 and draft st2 PR #239.