eval: prove one-turn inbox delivery

AGENT-SPEC.md

@@ -352,6 +352,32 @@ consecutive liveness misses once the target was observed alive.
 
 Settable presence is `offline`, `available`, `busy`, `away`, or `dnd`; `unknown` is derived and cannot be set.
 
+<a id="agent-spec-bounded-provider-inbox-delivery"></a>
+
+### Bounded maintained-provider inbox delivery
+
+At a maintained provider's native turn boundary, unread messages are offered as the largest complete FIFO
+prefix within fixed message-count and byte bounds. The provider envelope carries exact filenames, metadata,
+and complete bodies from the existing `message ls <identity> --json --include-body` result. The read-only
+`message delivery` surface exposes that provider-neutral rendering to thin adapters. Bodies are never
+truncated. Overflow remains durably unread for a later delivery; an oversized FIFO head falls back to its
+metadata and the existing explicit read path without reordering later messages.
+
+The ordinary path must let one inference issue one shell tool call containing the existing per-message
+`message reply` and `message archive` commands. This is an efficiency guarantee, not a new settlement
+protocol: there is no batch claim, lease, cursor, or implicit archive authority. New arrivals remain outside
+the rendered prefix, and archive receipts keep their existing exact-filename idempotency.
+
+Maintained Codex uses its structured app-server delivery channel. Maintained Claude injects the same semantic
+envelope through its native prompt hook. Unknown or custom harnesses retain the bounded metadata-only generic
+DING. Provider adapters may move behind the driver boundary without changing selection, payload, overflow,
+or archive semantics.
+
+Executable evidence: [`inbox-one-turn-provider-ab`](cells/inbox-one-turn-provider-ab/) is the matched
+Claude/Codex acceptance scaffold for cold backlog, during-turn arrival, bounded burst, and post-batch arrival.
+It is run at exact baseline and candidate st2 heads; provider transcripts and tracked receipts supply model,
+token, tool-boundary, wall-time, provenance, archive-outcome, and cleanup evidence.
+
 <a id="agent-spec-claude-declaration"></a>
 
 ## Claude declaration
@@ -503,8 +529,10 @@ workspace:
 
 `bin/check-harness-contract.sh` enumerates every model agent from every maintained root KDL,
 materializes only the known offline fixture builders, and compares each hook file byte-for-byte with
-`harness/`. `bin/check-event-first.sh` separately requires one cold-start inbox drain followed by
-native DING wakeups. Structured exceptions are in `evidence/harness-exclusions.tsv`.
+`harness/`. The bounded-delivery acceptance cell has one mechanically checked candidate-only Claude
+`UserPromptSubmit` hook in addition to that canonical baseline. `bin/check-event-first.sh` separately requires
+one cold-start inbox drain followed by native event delivery. Structured exceptions are in
+`evidence/harness-exclusions.tsv`.
 
 <a id="agent-spec-minimum-exhaustive-authoring-examples"></a>
 

CATALOG.md

@@ -1,8 +1,8 @@
 <!-- Generated by bin/generate-catalog.sh from cell KDL and evidence/*.tsv. Do not edit. -->
 # Current eval catalog and evidence
 
-This is the sole current corpus surface: **59 included cells** (19 model-free,
-40 model-backed) and **4 retired exclusions**. Inventory fields are derived from the active
+This is the sole current corpus surface: **60 included cells** (19 model-free,
+41 model-backed) and **4 retired exclusions**. Inventory fields are derived from the active
 KDL and executable judge scripts. Latest accepted PASS evidence remains distinct from the append-only run
 history: a failed last run is visible without being advertised as accepted evidence, and cells with no
 structured run row say so explicitly.
@@ -41,6 +41,7 @@ harness-native loader plus canonical hook file.
 | `hook-integrity` | model-free | — | 0 | none | `90s` | 4 | — | **NO STRUCTURED RUN** |
 | `host-lock-health-negatives` | model-free | — | 0 | none | `30s` | 5 | — | **NO STRUCTURED RUN** |
 | `inbox-hygiene` | Claude | `claude-sonnet-5` / medium | 1 | low | `900s` | 4 | — | **NO STRUCTURED RUN** |
+| `inbox-one-turn-provider-ab` | mixed | `claude-sonnet-5+gpt-5.6-sol` / medium | 2 | medium | `900s` | 2 | **PASS** 2026-08-11T15:43:02Z, 4/4, source [`117edf4`](https://github.com/compoundingtech/evals/commit/117edf4b6872c34a75921ccdbf95e0709541fc96), [receipt](evidence/inbox-one-turn-provider-ab-runs-20260811.json) | **PASS** 2026-08-11T15:43:02Z, 4/4, source [`117edf4`](https://github.com/compoundingtech/evals/commit/117edf4b6872c34a75921ccdbf95e0709541fc96), [receipt](evidence/inbox-one-turn-provider-ab-runs-20260811.json) |
 | `incident-response` | Claude | `claude-sonnet-5` / medium | 2 | medium | `1200s` | 5 | — | **NO STRUCTURED RUN** |
 | `license-mit` | mixed | `claude-sonnet-5+gpt-5.6-sol` / medium | 3 | high | `1200s` | 6 | — | **FAIL** 2026-07-28T13:27:18Z, 5/6, source [`b3cd5fb`](https://github.com/compoundingtech/evals/commit/b3cd5fbd98c11179a4555d0f9bbccfe98351a734), [receipt](evidence/stable-main-phase4-runs-20260728.json) |
 | `license-mit-codex` | Codex | `gpt-5.6-sol` / medium | 3 | high | `1200s` | 6 | **PASS** 2026-07-26, st2 `9d26245`, 1m39s, [`40ec23d`](https://github.com/compoundingtech/evals/commit/40ec23d3e068e5bd70b6d4b9a92a92d291868dce) | **NO STRUCTURED RUN** |

bin/check-event-first.sh

@@ -1,5 +1,5 @@
 #!/usr/bin/env bash
-# Enforce one cold-start drain followed by native DING wakeups for every maintained model seat.
+# Enforce one cold-start drain followed by native event delivery for every maintained model seat.
 set -euo pipefail
 
 repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
@@ -55,7 +55,7 @@ if rg -n --pcre2 \
 fi
 
 if [ "$failed" -eq 0 ]; then
-  printf 'PASS: %s model seats use one cold-start drain and event-first native DING wakeups\n' \
+  printf 'PASS: %s model seats use one cold-start drain and event-first native delivery\n' \
     "$(wc -l < "$inventory" | tr -d ' ')"
 fi
 exit "$failed"

bin/check-harness-contract.sh

@@ -110,8 +110,18 @@ while IFS=$'\t' read -r cell agent harness workspace st_agent command_line; do
       { echo "FAIL: $cell/$agent CLAUDE.md does not load @PERSONA.md" >&2; failed=1; }
     [ -s "$target/PERSONA.md" ] ||
       { echo "FAIL: $cell/$agent has no non-empty PERSONA.md" >&2; failed=1; }
-    cmp -s harness/claude-settings.local.json "$target/.claude/settings.local.json" ||
-      { echo "FAIL: $cell/$agent does not materialize the canonical Claude hooks" >&2; failed=1; }
+    if [ "$cell" = "inbox-one-turn-provider-ab" ]; then
+      stripped="$scratch/$cell-$agent-hooks.json"
+      jq 'del(.hooks.UserPromptSubmit)' "$target/.claude/settings.local.json" >"$stripped"
+      cmp -s harness/claude-settings.local.json "$stripped" ||
+        { echo "FAIL: $cell/$agent differs from canonical Claude hooks beyond UserPromptSubmit" >&2; failed=1; }
+      jq -e '.hooks.UserPromptSubmit == [{"hooks":[{"type":"command","command":"test ! -x \"$ST_HOOKS/claude-user-prompt-submit.sh\" || \"$ST_HOOKS/claude-user-prompt-submit.sh\""}]}]' \
+        "$target/.claude/settings.local.json" >/dev/null ||
+        { echo "FAIL: $cell/$agent does not declare the exact guarded candidate UserPromptSubmit hook" >&2; failed=1; }
+    else
+      cmp -s harness/claude-settings.local.json "$target/.claude/settings.local.json" ||
+        { echo "FAIL: $cell/$agent does not materialize the canonical Claude hooks" >&2; failed=1; }
+    fi
     grep -Fq 'Read CLAUDE.md.' <<< "$command_text" ||
       { echo "FAIL: $cell/$agent launch does not use its Claude loader" >&2; failed=1; }
     [ ! -e "$target/.codex/hooks.json" ] ||

bin/check-model-policy-mutations.sh

@@ -58,4 +58,29 @@ grep -Fq 'selects Opus' "$opus_output" || {
   exit 1
 }
 
-echo "PASS: nested provider launches are counted independently; unpinned and Opus mutations fail"
+mkdir -p "$scratch/structured-valid" "$scratch/structured-unpinned"
+printf '%s\n' \
+  'agent "worker" {' \
+  '  argv "codex" "--dangerously-bypass-hook-trust" "--model" "gpt-5.6-sol" "-c" "model_reasoning_effort=\"medium\"" "task"' \
+  '}' >"$scratch/structured-valid/agent.kdl"
+printf '%s\n' \
+  'agent "worker" {' \
+  '  argv "codex" "--dangerously-bypass-hook-trust" "-c" "model_reasoning_effort=\"medium\"" "task"' \
+  '}' >"$scratch/structured-unpinned/agent.kdl"
+
+bash "$checker" "$scratch/structured-valid" >/dev/null || {
+  echo "FAIL: model policy rejected a fully pinned structured Codex argv" >&2
+  exit 1
+}
+structured_output="$scratch/structured-unpinned.out"
+if bash "$checker" "$scratch/structured-unpinned" >"$structured_output" 2>&1; then
+  echo "FAIL: model policy accepted a structured Codex argv without its model pin" >&2
+  exit 1
+fi
+grep -Fq 'launches Codex without --model gpt-5.6-sol' "$structured_output" || {
+  echo "FAIL: structured Codex mutation failed without the expected model diagnostic" >&2
+  cat "$structured_output" >&2
+  exit 1
+}
+
+echo "PASS: nested and structured provider launches are counted; unpinned and Opus mutations fail"

bin/check-model-policy.sh

@@ -30,11 +30,14 @@ while IFS=: read -r file line text; do
     code="${code%%//*}"
   fi
 
+  code="${code//\\\"/__EVAL_ESCAPED_QUOTE__}"
+  code="${code//\"/}"
+  code="${code//__EVAL_ESCAPED_QUOTE__/\"}"
   remaining="$code"
-  provider_regex='(^|[^[:alnum:]_-])(exec[[:space:]]+)?(claude|codex)[[:space:]]+-'
+  provider_regex='(^|[^[:alnum:]_-])((exec|argv)[[:space:]]+)?(claude|codex)[[:space:]]+-'
   while [[ "$remaining" =~ $provider_regex ]]; do
     match="${BASH_REMATCH[0]}"
-    provider="${BASH_REMATCH[3]}"
+    provider="${BASH_REMATCH[4]}"
     after="${remaining#*"$match"}"
     invocation="$match$after"
     if [[ "$after" =~ $provider_regex ]]; then
@@ -57,7 +60,7 @@ while IFS=: read -r file line text; do
       ((codex_launches += 1))
       [[ "$invocation" == *"--model gpt-5.6-sol"* ]] ||
         fail "$file:$line launches Codex without --model gpt-5.6-sol"
-      [[ "$invocation" == *"model_reasoning_effort=\"medium\""* ]] ||
+      [[ "$invocation" == *"model_reasoning_effort=medium"* || "$invocation" == *"model_reasoning_effort=\"medium\""* ]] ||
         fail "$file:$line launches Codex without explicit medium reasoning effort"
       [[ "$invocation" == *"--dangerously-bypass-hook-trust"* ]] ||
         fail "$file:$line launches Codex without trusting the canonical workspace hooks"
@@ -67,7 +70,7 @@ while IFS=: read -r file line text; do
   done
 done < <(
   rg --no-ignore -n --no-heading \
-    'exec[[:space:]]+(claude|codex)|(^|[^[:alnum:]_-])(claude|codex)[[:space:]]+-' \
+    '(exec[[:space:]]+|argv[[:space:]]+")?(claude|codex)"?[[:space:]]+"?-' \
     "$scan_root" -g '*.kdl' -g '*.sh' -g '!**/_git/**' || true
 )
 

bin/corpus-inventory.sh

@@ -56,16 +56,19 @@ for cell in "${cells[@]}"; do
     if [[ "$file" == *.kdl ]]; then
       code="${code%%//*}"
     fi
+    code="${code//\"/}"
     if [[ "$code" =~ exec[[:space:]]+claude([[:space:]]|$) ]] ||
+      [[ "$code" =~ argv[[:space:]]+claude([[:space:]]|$) ]] ||
       [[ "$code" =~ (^|[^[:alnum:]_-])claude[[:space:]]+- ]]; then
       ((claude += 1))
     elif [[ "$code" =~ exec[[:space:]]+codex([[:space:]]|$) ]] ||
+      [[ "$code" =~ argv[[:space:]]+codex([[:space:]]|$) ]] ||
       [[ "$code" =~ (^|[^[:alnum:]_-])codex[[:space:]]+- ]]; then
       ((codex += 1))
     fi
   done < <(
     rg -n --no-heading \
-      'exec[[:space:]]+(claude|codex)|(^|[^[:alnum:]_-])(claude|codex)[[:space:]]+-' \
+      '(exec[[:space:]]+|argv[[:space:]]+")?(claude|codex)"?[[:space:]]+"?-' \
       "$cell_dir" -g '*.kdl' -g '*.sh' -g '!**/_git/**' || true
   )
 

bin/model-seat-inventory.sh

@@ -44,11 +44,12 @@ while IFS= read -r cell; do
         printf "FAIL: %s agent %s has a model command but no direct ST_AGENT\n", cell, agent > "/dev/stderr"
         bad = 1
       }
-      if (ding != 1) {
-        printf "FAIL: %s agent %s has a model command but %d direct bare ding declarations\n", cell, agent, ding > "/dev/stderr"
+      native_delivery = (harness == "Codex" && delivery == "app-server" && ding == 0) || (delivery == "" && ding == 1)
+      if (!native_delivery) {
+        printf "FAIL: %s agent %s has invalid native delivery (harness=%s delivery=%s ding=%d)\n", cell, agent, harness, delivery, ding > "/dev/stderr"
         bad = 1
       }
-      if (workspace != "" && st_agent != "" && ding == 1) {
+      if (workspace != "" && st_agent != "" && native_delivery) {
         printf "%s\t%s\t%s\t%s\t%s\t%d\n", cell, agent, harness, workspace, st_agent, command_line
       }
     }
@@ -68,6 +69,7 @@ while IFS= read -r cell; do
       harness = ""
       command_line = 0
       ding = 0
+      delivery = ""
       next
     }
     in_agent {
@@ -93,15 +95,20 @@ while IFS= read -r cell; do
         sub(/.*ST_AGENT[[:space:]]+"/, "", st_agent)
         sub(/".*/, "", st_agent)
       }
-      if ($0 ~ /^[[:space:]]*command[[:space:]]+/) {
+      if ($0 ~ /^[[:space:]]*(command|argv)[[:space:]]+/) {
         if ($0 ~ /exec claude[[:space:]]/) {
           harness = "Claude"
           command_line = NR
-        } else if ($0 ~ /exec codex[[:space:]]/) {
+        } else if ($0 ~ /(exec codex[[:space:]]|argv[[:space:]]+"codex")/) {
           harness = "Codex"
           command_line = NR
         }
       }
+      if ($0 ~ /^[[:space:]]*deliver[[:space:]]+"/) {
+        delivery = $0
+        sub(/^[[:space:]]*deliver[[:space:]]+"/, "", delivery)
+        sub(/".*/, "", delivery)
+      }
       if (trimmed == child_indent "ding") {
         ding += 1
       }
@@ -118,7 +125,7 @@ while IFS= read -r cell; do
 done < <(find cells -mindepth 1 -maxdepth 1 -type d -printf '%f\n' | LC_ALL=C sort)
 
 launches="$(
-  rg -n --no-heading '^[[:space:]]*command[[:space:]]+.*exec (claude|codex)[[:space:]]' \
+  rg -n --no-heading '^[[:space:]]*(command[[:space:]]+.*exec |argv[[:space:]]+")(claude|codex)' \
     cells/*/*.kdl | wc -l | tr -d ' '
 )"
 rows="$(wc -l < "$inventory" | tr -d ' ')"

cells/inbox-one-turn-provider-ab/README.md

@@ -0,0 +1,97 @@
+# inbox-one-turn-provider-ab
+
+Focused real-provider A/B scaffold for [st2 #238](https://github.com/compoundingtech/st2/issues/238). The
+same mixed Claude/Codex cell is run at two exact immutable st2 heads:
+
+| Arm | Delivery | Expected inference/tool shape |
+|---|---|---|
+| baseline | ordered bounded metadata without bodies | discovery tool call, another inference, then action |
+| candidate | the same ordered slice with bounded bodies | one inference can issue one shell tool call containing the existing reply/archive commands |
+
+There is deliberately no `settle`, claim, lease, cursor, provider opt-in, or eval-only delivery switch. The
+candidate's read-only `st2 message delivery` seam renders the existing
+`st2 message ls <identity> --json --include-body` result into the maintained provider's delivery envelope.
+The exact st2 binary is the only A/B variable. This makes the cell transplantable when provider adapters move
+behind the driver boundary tracked by st2 #162.
+
+## Scenario and evidence
+
+Both maintained providers receive two cold-backlog messages before boot. That matched batch measures whether
+their first inference must discover bodies or can act immediately. Codex additionally receives the evented
+sequence through its typed app-server transport:
+
+1. a bounded three-message burst injected after its first bus CLI process begins;
+2. one post-batch message injected only after its first five messages are durably archived.
+
+The held-out outcome judge requires an exact threaded acknowledgement and one archive copy for both Claude
+cold tokens and all six Codex tokens, empty subject inboxes, and Codex delivery and handling of the final
+post-batch arrival. The
+measurement judge emits JSON containing every wrapped bus CLI argv, provider-delivery/discovery/mutation
+counts, first subject inbox-operation classification,
+source payload bytes, first/last CLI timestamps, scenario timestamps, exact archive filenames, `st2
+--version`, and the runner binary SHA256. The wrapper forwards to the real installed `st2`; it does not model
+delivery or settlement.
+
+Provider/API call counts, provider-reported input/output/cache tokens, tool-call boundaries, total prompt
+bytes, wall time, exact evals source commit, and exact st2 source commit belong in the normal tracked run
+receipt and append-only `evidence/run-history.tsv` row. CLI process counts are a robust fixture-local metric.
+The Claude hook's read-only `message delivery` process is reported separately and is not mislabeled as a
+model tool call. The candidate is accepted only when the provider transcript
+also proves one inference before one shell action call; a green archive outcome alone is insufficient.
+
+Codex uses a canonical Agent Spec with structured `argv` plus `deliver "app-server"`, so both arms exercise
+the native #237 app-server seam rather than generic PTY DING. Canonical Agent Specs also keep the A/B on the
+driver-facing declaration boundary: both exact heads consume the same cell even though the baseline compact
+eval grammar cannot express structured argv. The setup uses st2's existing batch `pretrust` utility before
+boot, matching the standard compact-eval lifecycle without adding a provider-specific trust mechanism.
+The immutable baseline is st2 #237 head
+`1d06c4b263a7c5a2a6b8eec1f2e8c4fbea5e2edc`; the candidate is st2 #239 head
+`c1a0f90dd4814ec3ce8067219530d7bd8723e191`. Claude proves the maintained SessionStart hook path against the
+cold batch. Codex proves the maintained typed DING path against both active-turn and post-batch arrivals.
+Generic Claude PTY DING is deliberately left to evals #57: its prompt/draft collision policy is a transport
+safety axis and would confound this cell's body-availability comparison.
+
+Exact Nix-built runner binaries used for the matched run:
+
+| arm | st2 version | binary SHA256 |
+|---|---|---|
+| baseline | `st2 0.1.0+1d06c4b` | `06997b2c63ddf58678c7a4d024de62d957faf6c2bdc56766c15501e7946c1338` |
+| candidate | `st2 0.1.0+c1a0f90` | `6795c8539ca308fc49b439a3b6495d2bcd2ad4a5ede4cfe02dd53be2f2c67d81` |
+
+The matched real-provider result is tracked in
+[`evidence/inbox-one-turn-provider-ab-runs-20260811.json`](../../evidence/inbox-one-turn-provider-ab-runs-20260811.json).
+The baseline failed the exact-body outcome: Claude succeeded, but Codex archived all six messages while
+acknowledging subject-derived summaries. The candidate passed all exact-body, threaded-reply, archive,
+bounded-burst, and post-batch checks.
+
+| metric | baseline | candidate | change |
+|---|---:|---:|---:|
+| functional score | FAIL, 3/4 | PASS, 4/4 | exact-body outcome fixed |
+| Codex total tokens | 535,286 | 227,871 | -57.43% |
+| Codex recorded model-response events | 17 | 7 | -58.82% |
+| Codex tool-call boundaries | 11 | 4 | -63.64% |
+| Codex wall time | 115.367 s | 45.676 s | -60.41% |
+| Claude token volume | 367,011 | 319,857 | -12.85% |
+| Claude session duration | 114.706 s | 45.315 s | -60.49% |
+
+The original judge launcher used a cell-relative path and exited 127 after the provider executions completed.
+The receipt records this explicitly: both preserved catalogs were deterministically regraded with the corrected
+tracked judges at `3de9f7e2cffbe4d24da95d4422ed5f687c4b30f9`. The baseline returns the expected semantic failure and the
+candidate returns success. Claude API-call count and total serialized prompt bytes were not persisted and remain
+unknown rather than inferred.
+
+## Scope boundary
+
+This is related to evals #57, but does not duplicate its broader PTY matrix. It excludes idle/activity
+classification, partial human drafts, input collision, DND, compaction/crash routing, guarded writes, and
+generic/custom adapters. Those are transport-safety axes, not #238's body-availability efficiency axis.
+
+Free fixture/grader self-test and corpus gate:
+
+```sh
+bash cells/inbox-one-turn-provider-ab/fixture/self-test.sh
+bin/generate-catalog.sh --write
+bin/check-corpus.sh
+```
+
+Any further paid run requires fresh explicit approval and exact baseline/candidate st2 binary provenance.