diff --git a/AGENT-SPEC.md b/AGENT-SPEC.md index 4fdb959c..26a859fc 100644 --- a/AGENT-SPEC.md +++ b/AGENT-SPEC.md @@ -352,6 +352,32 @@ consecutive liveness misses once the target was observed alive. Settable presence is `offline`, `available`, `busy`, `away`, or `dnd`; `unknown` is derived and cannot be set. + + +### Bounded maintained-provider inbox delivery + +At a maintained provider's native turn boundary, unread messages are offered as the largest complete FIFO +prefix within fixed message-count and byte bounds. The provider envelope carries exact filenames, metadata, +and complete bodies from the existing `message ls --json --include-body` result. The read-only +`message delivery` surface exposes that provider-neutral rendering to thin adapters. Bodies are never +truncated. Overflow remains durably unread for a later delivery; an oversized FIFO head falls back to its +metadata and the existing explicit read path without reordering later messages. + +The ordinary path must let one inference issue one shell tool call containing the existing per-message +`message reply` and `message archive` commands. This is an efficiency guarantee, not a new settlement +protocol: there is no batch claim, lease, cursor, or implicit archive authority. New arrivals remain outside +the rendered prefix, and archive receipts keep their existing exact-filename idempotency. + +Maintained Codex uses its structured app-server delivery channel. Maintained Claude injects the same semantic +envelope through its native prompt hook. Unknown or custom harnesses retain the bounded metadata-only generic +DING. Provider adapters may move behind the driver boundary without changing selection, payload, overflow, +or archive semantics. + +Executable evidence: [`inbox-one-turn-provider-ab`](cells/inbox-one-turn-provider-ab/) is the matched +Claude/Codex acceptance scaffold for cold backlog, during-turn arrival, bounded burst, and post-batch arrival. +It is run at exact baseline and candidate st2 heads; provider transcripts and tracked receipts supply model, +token, tool-boundary, wall-time, provenance, archive-outcome, and cleanup evidence. + ## Claude declaration @@ -503,8 +529,10 @@ workspace: `bin/check-harness-contract.sh` enumerates every model agent from every maintained root KDL, materializes only the known offline fixture builders, and compares each hook file byte-for-byte with -`harness/`. `bin/check-event-first.sh` separately requires one cold-start inbox drain followed by -native DING wakeups. Structured exceptions are in `evidence/harness-exclusions.tsv`. +`harness/`. The bounded-delivery acceptance cell has one mechanically checked candidate-only Claude +`UserPromptSubmit` hook in addition to that canonical baseline. `bin/check-event-first.sh` separately requires +one cold-start inbox drain followed by native event delivery. Structured exceptions are in +`evidence/harness-exclusions.tsv`. diff --git a/CATALOG.md b/CATALOG.md index a4f12707..0bc1a0ec 100644 --- a/CATALOG.md +++ b/CATALOG.md @@ -1,8 +1,8 @@ # Current eval catalog and evidence -This is the sole current corpus surface: **59 included cells** (19 model-free, -40 model-backed) and **4 retired exclusions**. Inventory fields are derived from the active +This is the sole current corpus surface: **60 included cells** (19 model-free, +41 model-backed) and **4 retired exclusions**. Inventory fields are derived from the active KDL and executable judge scripts. Latest accepted PASS evidence remains distinct from the append-only run history: a failed last run is visible without being advertised as accepted evidence, and cells with no structured run row say so explicitly. @@ -41,6 +41,7 @@ harness-native loader plus canonical hook file. | `hook-integrity` | model-free | — | 0 | none | `90s` | 4 | — | **NO STRUCTURED RUN** | | `host-lock-health-negatives` | model-free | — | 0 | none | `30s` | 5 | — | **NO STRUCTURED RUN** | | `inbox-hygiene` | Claude | `claude-sonnet-5` / medium | 1 | low | `900s` | 4 | — | **NO STRUCTURED RUN** | +| `inbox-one-turn-provider-ab` | mixed | `claude-sonnet-5+gpt-5.6-sol` / medium | 2 | medium | `900s` | 2 | **PASS** 2026-08-11T15:43:02Z, 4/4, source [`117edf4`](https://github.com/compoundingtech/evals/commit/117edf4b6872c34a75921ccdbf95e0709541fc96), [receipt](evidence/inbox-one-turn-provider-ab-runs-20260811.json) | **PASS** 2026-08-11T15:43:02Z, 4/4, source [`117edf4`](https://github.com/compoundingtech/evals/commit/117edf4b6872c34a75921ccdbf95e0709541fc96), [receipt](evidence/inbox-one-turn-provider-ab-runs-20260811.json) | | `incident-response` | Claude | `claude-sonnet-5` / medium | 2 | medium | `1200s` | 5 | — | **NO STRUCTURED RUN** | | `license-mit` | mixed | `claude-sonnet-5+gpt-5.6-sol` / medium | 3 | high | `1200s` | 6 | — | **FAIL** 2026-07-28T13:27:18Z, 5/6, source [`b3cd5fb`](https://github.com/compoundingtech/evals/commit/b3cd5fbd98c11179a4555d0f9bbccfe98351a734), [receipt](evidence/stable-main-phase4-runs-20260728.json) | | `license-mit-codex` | Codex | `gpt-5.6-sol` / medium | 3 | high | `1200s` | 6 | **PASS** 2026-07-26, st2 `9d26245`, 1m39s, [`40ec23d`](https://github.com/compoundingtech/evals/commit/40ec23d3e068e5bd70b6d4b9a92a92d291868dce) | **NO STRUCTURED RUN** | diff --git a/bin/check-event-first.sh b/bin/check-event-first.sh index f47d96d4..f94eeb10 100755 --- a/bin/check-event-first.sh +++ b/bin/check-event-first.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash -# Enforce one cold-start drain followed by native DING wakeups for every maintained model seat. +# Enforce one cold-start drain followed by native event delivery for every maintained model seat. set -euo pipefail repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" @@ -55,7 +55,7 @@ if rg -n --pcre2 \ fi if [ "$failed" -eq 0 ]; then - printf 'PASS: %s model seats use one cold-start drain and event-first native DING wakeups\n' \ + printf 'PASS: %s model seats use one cold-start drain and event-first native delivery\n' \ "$(wc -l < "$inventory" | tr -d ' ')" fi exit "$failed" diff --git a/bin/check-harness-contract.sh b/bin/check-harness-contract.sh index e03e1a68..b03d8c17 100755 --- a/bin/check-harness-contract.sh +++ b/bin/check-harness-contract.sh @@ -110,8 +110,18 @@ while IFS=$'\t' read -r cell agent harness workspace st_agent command_line; do { echo "FAIL: $cell/$agent CLAUDE.md does not load @PERSONA.md" >&2; failed=1; } [ -s "$target/PERSONA.md" ] || { echo "FAIL: $cell/$agent has no non-empty PERSONA.md" >&2; failed=1; } - cmp -s harness/claude-settings.local.json "$target/.claude/settings.local.json" || - { echo "FAIL: $cell/$agent does not materialize the canonical Claude hooks" >&2; failed=1; } + if [ "$cell" = "inbox-one-turn-provider-ab" ]; then + stripped="$scratch/$cell-$agent-hooks.json" + jq 'del(.hooks.UserPromptSubmit)' "$target/.claude/settings.local.json" >"$stripped" + cmp -s harness/claude-settings.local.json "$stripped" || + { echo "FAIL: $cell/$agent differs from canonical Claude hooks beyond UserPromptSubmit" >&2; failed=1; } + jq -e '.hooks.UserPromptSubmit == [{"hooks":[{"type":"command","command":"test ! -x \"$ST_HOOKS/claude-user-prompt-submit.sh\" || \"$ST_HOOKS/claude-user-prompt-submit.sh\""}]}]' \ + "$target/.claude/settings.local.json" >/dev/null || + { echo "FAIL: $cell/$agent does not declare the exact guarded candidate UserPromptSubmit hook" >&2; failed=1; } + else + cmp -s harness/claude-settings.local.json "$target/.claude/settings.local.json" || + { echo "FAIL: $cell/$agent does not materialize the canonical Claude hooks" >&2; failed=1; } + fi grep -Fq 'Read CLAUDE.md.' <<< "$command_text" || { echo "FAIL: $cell/$agent launch does not use its Claude loader" >&2; failed=1; } [ ! -e "$target/.codex/hooks.json" ] || diff --git a/bin/check-model-policy-mutations.sh b/bin/check-model-policy-mutations.sh index 3ff9a7da..2fd4d05e 100755 --- a/bin/check-model-policy-mutations.sh +++ b/bin/check-model-policy-mutations.sh @@ -58,4 +58,29 @@ grep -Fq 'selects Opus' "$opus_output" || { exit 1 } -echo "PASS: nested provider launches are counted independently; unpinned and Opus mutations fail" +mkdir -p "$scratch/structured-valid" "$scratch/structured-unpinned" +printf '%s\n' \ + 'agent "worker" {' \ + ' argv "codex" "--dangerously-bypass-hook-trust" "--model" "gpt-5.6-sol" "-c" "model_reasoning_effort=\"medium\"" "task"' \ + '}' >"$scratch/structured-valid/agent.kdl" +printf '%s\n' \ + 'agent "worker" {' \ + ' argv "codex" "--dangerously-bypass-hook-trust" "-c" "model_reasoning_effort=\"medium\"" "task"' \ + '}' >"$scratch/structured-unpinned/agent.kdl" + +bash "$checker" "$scratch/structured-valid" >/dev/null || { + echo "FAIL: model policy rejected a fully pinned structured Codex argv" >&2 + exit 1 +} +structured_output="$scratch/structured-unpinned.out" +if bash "$checker" "$scratch/structured-unpinned" >"$structured_output" 2>&1; then + echo "FAIL: model policy accepted a structured Codex argv without its model pin" >&2 + exit 1 +fi +grep -Fq 'launches Codex without --model gpt-5.6-sol' "$structured_output" || { + echo "FAIL: structured Codex mutation failed without the expected model diagnostic" >&2 + cat "$structured_output" >&2 + exit 1 +} + +echo "PASS: nested and structured provider launches are counted; unpinned and Opus mutations fail" diff --git a/bin/check-model-policy.sh b/bin/check-model-policy.sh index 52133e6e..306aca96 100755 --- a/bin/check-model-policy.sh +++ b/bin/check-model-policy.sh @@ -30,11 +30,14 @@ while IFS=: read -r file line text; do code="${code%%//*}" fi + code="${code//\\\"/__EVAL_ESCAPED_QUOTE__}" + code="${code//\"/}" + code="${code//__EVAL_ESCAPED_QUOTE__/\"}" remaining="$code" - provider_regex='(^|[^[:alnum:]_-])(exec[[:space:]]+)?(claude|codex)[[:space:]]+-' + provider_regex='(^|[^[:alnum:]_-])((exec|argv)[[:space:]]+)?(claude|codex)[[:space:]]+-' while [[ "$remaining" =~ $provider_regex ]]; do match="${BASH_REMATCH[0]}" - provider="${BASH_REMATCH[3]}" + provider="${BASH_REMATCH[4]}" after="${remaining#*"$match"}" invocation="$match$after" if [[ "$after" =~ $provider_regex ]]; then @@ -57,7 +60,7 @@ while IFS=: read -r file line text; do ((codex_launches += 1)) [[ "$invocation" == *"--model gpt-5.6-sol"* ]] || fail "$file:$line launches Codex without --model gpt-5.6-sol" - [[ "$invocation" == *"model_reasoning_effort=\"medium\""* ]] || + [[ "$invocation" == *"model_reasoning_effort=medium"* || "$invocation" == *"model_reasoning_effort=\"medium\""* ]] || fail "$file:$line launches Codex without explicit medium reasoning effort" [[ "$invocation" == *"--dangerously-bypass-hook-trust"* ]] || fail "$file:$line launches Codex without trusting the canonical workspace hooks" @@ -67,7 +70,7 @@ while IFS=: read -r file line text; do done done < <( rg --no-ignore -n --no-heading \ - 'exec[[:space:]]+(claude|codex)|(^|[^[:alnum:]_-])(claude|codex)[[:space:]]+-' \ + '(exec[[:space:]]+|argv[[:space:]]+")?(claude|codex)"?[[:space:]]+"?-' \ "$scan_root" -g '*.kdl' -g '*.sh' -g '!**/_git/**' || true ) diff --git a/bin/corpus-inventory.sh b/bin/corpus-inventory.sh index ed35fed9..5b743b52 100755 --- a/bin/corpus-inventory.sh +++ b/bin/corpus-inventory.sh @@ -56,16 +56,19 @@ for cell in "${cells[@]}"; do if [[ "$file" == *.kdl ]]; then code="${code%%//*}" fi + code="${code//\"/}" if [[ "$code" =~ exec[[:space:]]+claude([[:space:]]|$) ]] || + [[ "$code" =~ argv[[:space:]]+claude([[:space:]]|$) ]] || [[ "$code" =~ (^|[^[:alnum:]_-])claude[[:space:]]+- ]]; then ((claude += 1)) elif [[ "$code" =~ exec[[:space:]]+codex([[:space:]]|$) ]] || + [[ "$code" =~ argv[[:space:]]+codex([[:space:]]|$) ]] || [[ "$code" =~ (^|[^[:alnum:]_-])codex[[:space:]]+- ]]; then ((codex += 1)) fi done < <( rg -n --no-heading \ - 'exec[[:space:]]+(claude|codex)|(^|[^[:alnum:]_-])(claude|codex)[[:space:]]+-' \ + '(exec[[:space:]]+|argv[[:space:]]+")?(claude|codex)"?[[:space:]]+"?-' \ "$cell_dir" -g '*.kdl' -g '*.sh' -g '!**/_git/**' || true ) diff --git a/bin/model-seat-inventory.sh b/bin/model-seat-inventory.sh index e429ff1d..2279a936 100755 --- a/bin/model-seat-inventory.sh +++ b/bin/model-seat-inventory.sh @@ -44,11 +44,12 @@ while IFS= read -r cell; do printf "FAIL: %s agent %s has a model command but no direct ST_AGENT\n", cell, agent > "/dev/stderr" bad = 1 } - if (ding != 1) { - printf "FAIL: %s agent %s has a model command but %d direct bare ding declarations\n", cell, agent, ding > "/dev/stderr" + native_delivery = (harness == "Codex" && delivery == "app-server" && ding == 0) || (delivery == "" && ding == 1) + if (!native_delivery) { + printf "FAIL: %s agent %s has invalid native delivery (harness=%s delivery=%s ding=%d)\n", cell, agent, harness, delivery, ding > "/dev/stderr" bad = 1 } - if (workspace != "" && st_agent != "" && ding == 1) { + if (workspace != "" && st_agent != "" && native_delivery) { printf "%s\t%s\t%s\t%s\t%s\t%d\n", cell, agent, harness, workspace, st_agent, command_line } } @@ -68,6 +69,7 @@ while IFS= read -r cell; do harness = "" command_line = 0 ding = 0 + delivery = "" next } in_agent { @@ -93,15 +95,20 @@ while IFS= read -r cell; do sub(/.*ST_AGENT[[:space:]]+"/, "", st_agent) sub(/".*/, "", st_agent) } - if ($0 ~ /^[[:space:]]*command[[:space:]]+/) { + if ($0 ~ /^[[:space:]]*(command|argv)[[:space:]]+/) { if ($0 ~ /exec claude[[:space:]]/) { harness = "Claude" command_line = NR - } else if ($0 ~ /exec codex[[:space:]]/) { + } else if ($0 ~ /(exec codex[[:space:]]|argv[[:space:]]+"codex")/) { harness = "Codex" command_line = NR } } + if ($0 ~ /^[[:space:]]*deliver[[:space:]]+"/) { + delivery = $0 + sub(/^[[:space:]]*deliver[[:space:]]+"/, "", delivery) + sub(/".*/, "", delivery) + } if (trimmed == child_indent "ding") { ding += 1 } @@ -118,7 +125,7 @@ while IFS= read -r cell; do done < <(find cells -mindepth 1 -maxdepth 1 -type d -printf '%f\n' | LC_ALL=C sort) launches="$( - rg -n --no-heading '^[[:space:]]*command[[:space:]]+.*exec (claude|codex)[[:space:]]' \ + rg -n --no-heading '^[[:space:]]*(command[[:space:]]+.*exec |argv[[:space:]]+")(claude|codex)' \ cells/*/*.kdl | wc -l | tr -d ' ' )" rows="$(wc -l < "$inventory" | tr -d ' ')" diff --git a/cells/inbox-one-turn-provider-ab/README.md b/cells/inbox-one-turn-provider-ab/README.md new file mode 100644 index 00000000..10b3db62 --- /dev/null +++ b/cells/inbox-one-turn-provider-ab/README.md @@ -0,0 +1,97 @@ +# inbox-one-turn-provider-ab + +Focused real-provider A/B scaffold for [st2 #238](https://github.com/compoundingtech/st2/issues/238). The +same mixed Claude/Codex cell is run at two exact immutable st2 heads: + +| Arm | Delivery | Expected inference/tool shape | +|---|---|---| +| baseline | ordered bounded metadata without bodies | discovery tool call, another inference, then action | +| candidate | the same ordered slice with bounded bodies | one inference can issue one shell tool call containing the existing reply/archive commands | + +There is deliberately no `settle`, claim, lease, cursor, provider opt-in, or eval-only delivery switch. The +candidate's read-only `st2 message delivery` seam renders the existing +`st2 message ls --json --include-body` result into the maintained provider's delivery envelope. +The exact st2 binary is the only A/B variable. This makes the cell transplantable when provider adapters move +behind the driver boundary tracked by st2 #162. + +## Scenario and evidence + +Both maintained providers receive two cold-backlog messages before boot. That matched batch measures whether +their first inference must discover bodies or can act immediately. Codex additionally receives the evented +sequence through its typed app-server transport: + +1. a bounded three-message burst injected after its first bus CLI process begins; +2. one post-batch message injected only after its first five messages are durably archived. + +The held-out outcome judge requires an exact threaded acknowledgement and one archive copy for both Claude +cold tokens and all six Codex tokens, empty subject inboxes, and Codex delivery and handling of the final +post-batch arrival. The +measurement judge emits JSON containing every wrapped bus CLI argv, provider-delivery/discovery/mutation +counts, first subject inbox-operation classification, +source payload bytes, first/last CLI timestamps, scenario timestamps, exact archive filenames, `st2 +--version`, and the runner binary SHA256. The wrapper forwards to the real installed `st2`; it does not model +delivery or settlement. + +Provider/API call counts, provider-reported input/output/cache tokens, tool-call boundaries, total prompt +bytes, wall time, exact evals source commit, and exact st2 source commit belong in the normal tracked run +receipt and append-only `evidence/run-history.tsv` row. CLI process counts are a robust fixture-local metric. +The Claude hook's read-only `message delivery` process is reported separately and is not mislabeled as a +model tool call. The candidate is accepted only when the provider transcript +also proves one inference before one shell action call; a green archive outcome alone is insufficient. + +Codex uses a canonical Agent Spec with structured `argv` plus `deliver "app-server"`, so both arms exercise +the native #237 app-server seam rather than generic PTY DING. Canonical Agent Specs also keep the A/B on the +driver-facing declaration boundary: both exact heads consume the same cell even though the baseline compact +eval grammar cannot express structured argv. The setup uses st2's existing batch `pretrust` utility before +boot, matching the standard compact-eval lifecycle without adding a provider-specific trust mechanism. +The immutable baseline is st2 #237 head +`1d06c4b263a7c5a2a6b8eec1f2e8c4fbea5e2edc`; the candidate is st2 #239 head +`c1a0f90dd4814ec3ce8067219530d7bd8723e191`. Claude proves the maintained SessionStart hook path against the +cold batch. Codex proves the maintained typed DING path against both active-turn and post-batch arrivals. +Generic Claude PTY DING is deliberately left to evals #57: its prompt/draft collision policy is a transport +safety axis and would confound this cell's body-availability comparison. + +Exact Nix-built runner binaries used for the matched run: + +| arm | st2 version | binary SHA256 | +|---|---|---| +| baseline | `st2 0.1.0+1d06c4b` | `06997b2c63ddf58678c7a4d024de62d957faf6c2bdc56766c15501e7946c1338` | +| candidate | `st2 0.1.0+c1a0f90` | `6795c8539ca308fc49b439a3b6495d2bcd2ad4a5ede4cfe02dd53be2f2c67d81` | + +The matched real-provider result is tracked in +[`evidence/inbox-one-turn-provider-ab-runs-20260811.json`](../../evidence/inbox-one-turn-provider-ab-runs-20260811.json). +The baseline failed the exact-body outcome: Claude succeeded, but Codex archived all six messages while +acknowledging subject-derived summaries. The candidate passed all exact-body, threaded-reply, archive, +bounded-burst, and post-batch checks. + +| metric | baseline | candidate | change | +|---|---:|---:|---:| +| functional score | FAIL, 3/4 | PASS, 4/4 | exact-body outcome fixed | +| Codex total tokens | 535,286 | 227,871 | -57.43% | +| Codex recorded model-response events | 17 | 7 | -58.82% | +| Codex tool-call boundaries | 11 | 4 | -63.64% | +| Codex wall time | 115.367 s | 45.676 s | -60.41% | +| Claude token volume | 367,011 | 319,857 | -12.85% | +| Claude session duration | 114.706 s | 45.315 s | -60.49% | + +The original judge launcher used a cell-relative path and exited 127 after the provider executions completed. +The receipt records this explicitly: both preserved catalogs were deterministically regraded with the corrected +tracked judges at `3de9f7e2cffbe4d24da95d4422ed5f687c4b30f9`. The baseline returns the expected semantic failure and the +candidate returns success. Claude API-call count and total serialized prompt bytes were not persisted and remain +unknown rather than inferred. + +## Scope boundary + +This is related to evals #57, but does not duplicate its broader PTY matrix. It excludes idle/activity +classification, partial human drafts, input collision, DND, compaction/crash routing, guarded writes, and +generic/custom adapters. Those are transport-safety axes, not #238's body-availability efficiency axis. + +Free fixture/grader self-test and corpus gate: + +```sh +bash cells/inbox-one-turn-provider-ab/fixture/self-test.sh +bin/generate-catalog.sh --write +bin/check-corpus.sh +``` + +Any further paid run requires fresh explicit approval and exact baseline/candidate st2 binary provenance. diff --git a/cells/inbox-one-turn-provider-ab/fixture/agents/iot/claude/agent.kdl b/cells/inbox-one-turn-provider-ab/fixture/agents/iot/claude/agent.kdl new file mode 100644 index 00000000..493b1a8a --- /dev/null +++ b/cells/inbox-one-turn-provider-ab/fixture/agents/iot/claude/agent.kdl @@ -0,0 +1,9 @@ +agent "claude" { + host "iot" + workspace "$CATALOG/claude" + env { + ST_AGENT "iot.claude" + PATH "$CATALOG/bin:$PATH" + } + command #"exec claude --model claude-sonnet-5 --effort medium --permission-mode auto 'You just cold-started in a hermetic st2 eval. Read CLAUDE.md. Drain the inbox once with st2 message using $ST_ROOT and $ST_AGENT: an injected bounded body envelope is that drain; otherwise use one st2 message ls $ST_AGENT --json --include-body call. Then set status available; presence is best-effort if this flat eval seat cannot resolve. If the drain contains no work, stand by for DING. After any DING, drain the inbox once again by the same rule, act on every delivered message, archive each handled item, and report completion or blockers over the st2 bus.'"# +} diff --git a/cells/inbox-one-turn-provider-ab/fixture/agents/iot/codex/agent.kdl b/cells/inbox-one-turn-provider-ab/fixture/agents/iot/codex/agent.kdl new file mode 100644 index 00000000..a6b1945b --- /dev/null +++ b/cells/inbox-one-turn-provider-ab/fixture/agents/iot/codex/agent.kdl @@ -0,0 +1,10 @@ +agent "codex" { + host "iot" + workspace "$CATALOG/codex" + env { + ST_AGENT "iot.codex" + PATH "$CATALOG/bin:$PATH" + } + deliver "app-server" + argv "codex" "--dangerously-bypass-approvals-and-sandbox" "--dangerously-bypass-hook-trust" "--model" "gpt-5.6-sol" "-c" "model_reasoning_effort=\"medium\"" "You just cold-started in a hermetic st2 eval. Read AGENTS.md. Drain the inbox once with st2 message using $ST_ROOT and $ST_AGENT: an injected bounded body envelope is that drain; otherwise use one st2 message ls $ST_AGENT --json --include-body call. Then set status available; presence is best-effort if this flat eval seat cannot resolve. If the drain contains no work, stand by for DING. After any DING, drain the inbox once again by the same rule, act on every delivered message, archive each handled item, and report completion or blockers over the st2 bus." +} diff --git a/cells/inbox-one-turn-provider-ab/fixture/agents/iot/injector/agent.kdl b/cells/inbox-one-turn-provider-ab/fixture/agents/iot/injector/agent.kdl new file mode 100644 index 00000000..b1299ff1 --- /dev/null +++ b/cells/inbox-one-turn-provider-ab/fixture/agents/iot/injector/agent.kdl @@ -0,0 +1,9 @@ +agent "injector" { + host "iot" + workspace "$CATALOG/injector" + env { + ST_AGENT "iot.injector" + PATH "$CATALOG/bin:$PATH" + } + command "exec bash ./inject.sh" +} diff --git a/cells/inbox-one-turn-provider-ab/fixture/bin/st2 b/cells/inbox-one-turn-provider-ab/fixture/bin/st2 new file mode 100755 index 00000000..9c965d61 --- /dev/null +++ b/cells/inbox-one-turn-provider-ab/fixture/bin/st2 @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +set -euo pipefail + +self="$(readlink -f "$0")" +real="${ST2_REAL_BIN:-}" +if [ -z "$real" ]; then + while IFS= read -r candidate; do + if [ "$(readlink -f "$candidate")" != "$self" ]; then + real="$candidate" + break + fi + done < <(type -a -P st2) +fi +[ -n "$real" ] || { + echo "instrumentation wrapper could not resolve the real st2 binary" >&2 + exit 127 +} + +log_root="${CATALOG:?CATALOG must be set}/metrics" +mkdir -p "$log_root" +log="$log_root/${ST_AGENT:?ST_AGENT must be set}.jsonl" +record="$(jq -cn --arg at_ns "$(date +%s%N)" --arg cwd "$PWD" --args \ + '$ARGS.named + {argv: $ARGS.positional}' -- "$@")" +{ + flock 9 + printf '%s\n' "$record" >&9 +} 9>>"$log" + +exec "$real" "$@" diff --git a/cells/inbox-one-turn-provider-ab/fixture/claude/.claude/settings.local.json b/cells/inbox-one-turn-provider-ab/fixture/claude/.claude/settings.local.json new file mode 100644 index 00000000..37a07959 --- /dev/null +++ b/cells/inbox-one-turn-provider-ab/fixture/claude/.claude/settings.local.json @@ -0,0 +1,47 @@ +{ + "$schema": "https://json.schemastore.org/claude-code-settings.json", + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "async": true, + "asyncRewake": true, + "command": "$ST_HOOKS/claude-session-start.sh" + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "test ! -x \"$ST_HOOKS/claude-user-prompt-submit.sh\" || \"$ST_HOOKS/claude-user-prompt-submit.sh\"" + } + ] + } + ], + "PreCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "$ST_HOOKS/claude-pre-compact.sh" + } + ] + } + ], + "StopFailure": [ + { + "hooks": [ + { + "type": "command", + "command": "$ST_HOOKS/claude-stop-failure.sh" + } + ] + } + ] + } +} diff --git a/cells/inbox-one-turn-provider-ab/fixture/claude/CLAUDE.md b/cells/inbox-one-turn-provider-ab/fixture/claude/CLAUDE.md new file mode 100644 index 00000000..d3a79ea4 --- /dev/null +++ b/cells/inbox-one-turn-provider-ab/fixture/claude/CLAUDE.md @@ -0,0 +1 @@ +@PERSONA.md diff --git a/cells/inbox-one-turn-provider-ab/fixture/claude/PERSONA.md b/cells/inbox-one-turn-provider-ab/fixture/claude/PERSONA.md new file mode 100644 index 00000000..9bbfc1b6 --- /dev/null +++ b/cells/inbox-one-turn-provider-ab/fixture/claude/PERSONA.md @@ -0,0 +1,12 @@ +# One-turn inbox subject + +You are the Claude subject in a synthetic, hermetic message-bus efficiency eval. + +- Treat a bounded body-bearing provider delivery envelope as the inbox drain result. Do not list or read + those messages again. +- If delivery contains metadata only, retrieve the ordered bodies once with + `st2 message ls "$ST_AGENT" --json --include-body`. +- For every body, reply on its exact thread with the requested exact `ACK ...` text and archive it only after + replying. Put all reply/archive commands for the delivered slice into one shell tool call when possible. +- New arrivals are a later slice. Process them without losing or reprocessing earlier messages. +- Do not inspect the fixture, metrics, scenario, requester inbox, or held-out judges. diff --git a/cells/inbox-one-turn-provider-ab/fixture/codex/.codex/hooks.json b/cells/inbox-one-turn-provider-ab/fixture/codex/.codex/hooks.json new file mode 100644 index 00000000..76f8a9d8 --- /dev/null +++ b/cells/inbox-one-turn-provider-ab/fixture/codex/.codex/hooks.json @@ -0,0 +1,39 @@ +{ + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "$ST_HOOKS/codex-session-start.sh", + "timeout": 5, + "statusMessage": "Restoring st2 context" + } + ] + } + ], + "PreCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "$ST_HOOKS/codex-pre-compact.sh", + "timeout": 5, + "statusMessage": "Checkpointing st2 context" + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "command": "$ST_HOOKS/codex-stop.sh", + "timeout": 5 + } + ] + } + ] + } +} diff --git a/cells/inbox-one-turn-provider-ab/fixture/codex/AGENTS.md b/cells/inbox-one-turn-provider-ab/fixture/codex/AGENTS.md new file mode 100644 index 00000000..fa785d90 --- /dev/null +++ b/cells/inbox-one-turn-provider-ab/fixture/codex/AGENTS.md @@ -0,0 +1,12 @@ +# One-turn inbox subject + +You are the Codex subject in a synthetic, hermetic message-bus efficiency eval. + +- Treat a bounded body-bearing provider delivery envelope as the inbox drain result. Do not list or read + those messages again. +- If delivery contains metadata only, retrieve the ordered bodies once with + `st2 message ls "$ST_AGENT" --json --include-body`. +- For every body, reply on its exact thread with the requested exact `ACK ...` text and archive it only after + replying. Put all reply/archive commands for the delivered slice into one shell tool call when possible. +- New arrivals are a later slice. Process them without losing or reprocessing earlier messages. +- Do not inspect the fixture, metrics, scenario, requester inbox, or held-out judges. diff --git a/cells/inbox-one-turn-provider-ab/fixture/injector/inject.sh b/cells/inbox-one-turn-provider-ab/fixture/injector/inject.sh new file mode 100755 index 00000000..2f79f645 --- /dev/null +++ b/cells/inbox-one-turn-provider-ab/fixture/injector/inject.sh @@ -0,0 +1,77 @@ +#!/usr/bin/env bash +set -euo pipefail + +root="${CATALOG:?CATALOG must be set}" +bus="${ST_ROOT:-$root}" +state="$root/scenario" +mkdir -p "$state" + +wait_for_subject_cli() { + local agent="$1" file="$root/metrics/$1.jsonl" + for _ in $(seq 1 1200); do + if [ -s "$file" ] && jq -e -s ' + any(.[]; .argv[0] == "message" and .argv[1] != "delivery") + ' "$file" >/dev/null 2>&1; then + return 0 + fi + sleep 0.25 + done + echo "timed out waiting for $agent first subject inbox CLI process" >&2 + return 1 +} + +wait_for_archives() { + local agent="$1" expected="$2" + for _ in $(seq 1 1200); do + if [ "$(archive_token_count "$agent")" -eq "$expected" ]; then + return 0 + fi + sleep 0.25 + done + echo "timed out waiting for $agent archive count $expected" >&2 + return 1 +} + +mailbox_dir() { + printf '%s\n' "$bus/agents/iot/${1#iot.}/resources" +} + +archive_token_count() { + local agent="$1" + grep -lRE 'IOT-(COLD-[AB]|BURST-[123])-' "$(mailbox_dir "$agent")/archive" 2>/dev/null | wc -l | tr -d ' ' +} + +send() { + local target="$1" subject="$2" token="$3" + st2 message send "$target" --as iot.injector --subject "$subject" \ + -m "Reply on this thread with exactly \`ACK $token\`, then archive this message." >/dev/null +} + +for agent in iot.claude iot.codex; do + wait_for_subject_cli "$agent" + printf '%s\tfirst-subject-cli\t%s\n' "$(date +%s%N)" "$agent" >>"$state/events.tsv" +done + +send iot.codex "bounded burst 1" IOT-BURST-1-31ef +send iot.codex "bounded burst 2" IOT-BURST-2-82bc +send iot.codex "bounded burst 3" IOT-BURST-3-c540 +printf '%s\tburst-sent\tiot.codex\n' "$(date +%s%N)" >>"$state/events.tsv" + +wait_for_archives iot.codex 5 +printf '%s\tfirst-batch-archived\tiot.codex\n' "$(date +%s%N)" >>"$state/events.tsv" +send iot.codex "post-batch durability" IOT-POST-BATCH-f92a +printf '%s\tpost-batch-sent\tiot.codex\n' "$(date +%s%N)" >>"$state/events.tsv" + +for pair in iot.claude:2 iot.codex:6; do + agent="${pair%:*}" + expected="${pair#*:}" + for _ in $(seq 1 1200); do + if [ "$(find "$(mailbox_dir "$agent")/archive" -maxdepth 1 -type f | wc -l | tr -d ' ')" -eq "$expected" ]; then + break + fi + sleep 0.25 + done + [ "$(find "$(mailbox_dir "$agent")/archive" -maxdepth 1 -type f | wc -l | tr -d ' ')" -eq "$expected" ] +done + +st2 message send requester --as iot.injector -m "bounded inbox scenario complete" >/dev/null diff --git a/cells/inbox-one-turn-provider-ab/fixture/injector/seed.sh b/cells/inbox-one-turn-provider-ab/fixture/injector/seed.sh new file mode 100644 index 00000000..4b90811c --- /dev/null +++ b/cells/inbox-one-turn-provider-ab/fixture/injector/seed.sh @@ -0,0 +1,12 @@ +#!/usr/bin/env bash +set -euo pipefail + +export ST_AGENT=iot.injector +export ST_ROOT="$CATALOG" +st2 pretrust "$CATALOG/claude" "$CATALOG/codex" >/dev/null +for target in iot.claude iot.codex; do + st2 message send "$target" --subject "cold backlog A" \ + -m 'Reply on this thread with exactly `ACK IOT-COLD-A-4d91`, then archive this message.' >/dev/null + st2 message send "$target" --subject "cold backlog B" \ + -m 'Reply on this thread with exactly `ACK IOT-COLD-B-7a2c`, then archive this message.' >/dev/null +done diff --git a/cells/inbox-one-turn-provider-ab/fixture/judges/measurements.sh b/cells/inbox-one-turn-provider-ab/fixture/judges/measurements.sh new file mode 100755 index 00000000..bb05ea5e --- /dev/null +++ b/cells/inbox-one-turn-provider-ab/fixture/judges/measurements.sh @@ -0,0 +1,68 @@ +#!/usr/bin/env bash +set -euo pipefail + +root="${CATALOG:?CATALOG must be set}" +bus="${ST_ROOT:-$root}" +wrapper="$(readlink -f "$root/bin/st2")" +real="${ST2_REAL_BIN:-}" +if [ -z "$real" ]; then + while IFS= read -r candidate; do + if [ "$(readlink -f "$candidate")" != "$wrapper" ]; then + real="$candidate" + break + fi + done < <(PATH="$root/bin:$PATH" type -a -P st2) +fi +[ -n "$real" ] + +mailbox_dir() { + printf '%s\n' "$bus/agents/iot/${1#iot.}/resources" +} + +subjects='[]' +for agent in iot.claude iot.codex; do + log="$root/metrics/$agent.jsonl" + [ -s "$log" ] + jq -s -e 'all(.[]; (.at_ns | test("^[0-9]+$")) and (.argv | type == "array"))' "$log" >/dev/null + archive_names="$(find "$(mailbox_dir "$agent")/archive" -maxdepth 1 -type f -printf '%f\n' | LC_ALL=C sort | jq -Rsc 'split("\n")[:-1]')" + subject="$(jq -sc --arg agent "$agent" --argjson archives "$archive_names" ' + def is_delivery: .argv[0:2] == ["message", "delivery"]; + def is_discovery: .argv[0:2] == ["message", "ls"] or .argv[0:2] == ["message", "read"]; + def is_mutation: .argv[0:2] == ["message", "reply"] or .argv[0:2] == ["message", "archive"]; + { + agent: $agent, + cli_processes: length, + provider_delivery_cli_processes: ([.[] | select(is_delivery)] | length), + subject_inbox_cli_processes: ([.[] | select((.argv[0] == "message") and (is_delivery | not))] | length), + discovery_cli_processes: ([.[] | select(is_discovery)] | length), + mutation_cli_processes: ([.[] | select(is_mutation)] | length), + first_subject_inbox_operation: ([.[] | select((.argv[0] == "message") and (is_delivery | not))][0].argv[0:2] | join(" ")), + argv_bytes: ([.[] | (.argv | join(" ") | length)] | add), + first_cli_at_ns: .[0].at_ns, + last_cli_at_ns: .[-1].at_ns, + archive_filenames: $archives, + cli: . + } + ' "$log")" + subjects="$(jq -cn --argjson all "$subjects" --argjson one "$subject" '$all + [$one]')" +done + +source_payload_bytes="$(find "$(mailbox_dir iot.claude)/archive" -maxdepth 1 -type f -print0 | xargs -0 wc -c | awk 'END {print $1}')" +jq -cn \ + --arg runner_version "$("$real" --version)" \ + --arg runner_binary_sha256 "$(sha256sum "$real" | awk '{print $1}')" \ + --argjson source_payload_bytes "$source_payload_bytes" \ + --argjson subjects "$subjects" \ + --rawfile scenario "$root/scenario/events.tsv" \ + '{ + schema: "inbox-one-turn-provider-ab/v1", + runner: {version: $runner_version, binary_sha256: $runner_binary_sha256}, + source_payload_bytes: $source_payload_bytes, + subjects: $subjects, + scenario_events_tsv: $scenario, + provider_metrics_required_in_run_receipt: [ + "model_api_calls", "input_tokens", "output_tokens", "cache_tokens", + "tool_call_boundaries", "total_prompt_bytes", "wall_time_seconds", + "evals_source_commit", "st2_source_commit", "cleanup" + ] + }' diff --git a/cells/inbox-one-turn-provider-ab/fixture/judges/outcomes.sh b/cells/inbox-one-turn-provider-ab/fixture/judges/outcomes.sh new file mode 100755 index 00000000..47c68854 --- /dev/null +++ b/cells/inbox-one-turn-provider-ab/fixture/judges/outcomes.sh @@ -0,0 +1,54 @@ +#!/usr/bin/env bash +set -euo pipefail + +root="${CATALOG:?CATALOG must be set}" +bus="${ST_ROOT:-$root}" +codex_tokens=( + IOT-COLD-A-4d91 + IOT-COLD-B-7a2c + IOT-BURST-1-31ef + IOT-BURST-2-82bc + IOT-BURST-3-c540 + IOT-POST-BATCH-f92a +) +claude_tokens=(IOT-COLD-A-4d91 IOT-COLD-B-7a2c) + +mailbox_dir() { + printf '%s\n' "$bus/agents/iot/${1#iot.}/resources" +} + +for agent in iot.claude iot.codex; do + if [ "$agent" = iot.claude ]; then + tokens=("${claude_tokens[@]}") + else + tokens=("${codex_tokens[@]}") + fi + agent_dir="$(mailbox_dir "$agent")" + [ -d "$agent_dir/archive" ] + [ "$(find "$agent_dir/inbox" -maxdepth 1 -type f | wc -l | tr -d ' ')" -eq 0 ] + for token in "${tokens[@]}"; do + [ "$(grep -lRF "$token" "$agent_dir/archive" 2>/dev/null | wc -l | tr -d ' ')" -eq 1 ] + reply="" + mapfile -t candidates < <( + grep -lRF "ACK $token" "$(mailbox_dir iot.injector)/inbox" \ + "$(mailbox_dir iot.injector)/archive" 2>/dev/null || true + ) + for candidate in "${candidates[@]}"; do + if grep -Eq "^from:[[:space:]]*$agent([[:space:]]|$)" "$candidate"; then + reply="$candidate" + break + fi + done + [ -n "$reply" ] + grep -Fq 'in-reply-to:' "$reply" + grep -Eq "^from:[[:space:]]*$agent([[:space:]]|$)" "$reply" + done + echo "PASS: $agent replied on and archived all ${#tokens[@]} exact messages" +done + +awk -F '\t' ' + $2 == "first-batch-archived" && $3 == "iot.codex" { archived = $1 } + $2 == "post-batch-sent" && $3 == "iot.codex" { sent = $1 } + END { exit !(archived > 0 && sent > archived) } + ' "$root/scenario/events.tsv" +echo "PASS: Codex post-batch arrival was created only after its bounded burst was durably archived" diff --git a/cells/inbox-one-turn-provider-ab/fixture/self-test.sh b/cells/inbox-one-turn-provider-ab/fixture/self-test.sh new file mode 100755 index 00000000..a2a71fb9 --- /dev/null +++ b/cells/inbox-one-turn-provider-ab/fixture/self-test.sh @@ -0,0 +1,60 @@ +#!/usr/bin/env bash +set -euo pipefail + +fixture="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +scratch="$(mktemp -d)" +cleanup() { rm -rf -- "$scratch"; } +trap cleanup EXIT + +mkdir -p "$scratch"/{bin,metrics,scenario,requester/inbox,requester/archive} +mkdir -p "$scratch/agents/iot/injector/resources"/{inbox,archive} +cp "$fixture/bin/st2" "$scratch/bin/st2" +for agent in iot.claude iot.codex; do + agent_dir="$scratch/agents/iot/${agent#iot.}/resources" + mkdir -p "$agent_dir"/{inbox,archive} + : >"$scratch/metrics/$agent.jsonl" + index=0 + tokens=(IOT-COLD-A-4d91 IOT-COLD-B-7a2c) + if [ "$agent" = iot.codex ]; then + tokens+=(IOT-BURST-1-31ef IOT-BURST-2-82bc IOT-BURST-3-c540 IOT-POST-BATCH-f92a) + fi + for token in "${tokens[@]}"; do + index=$((index + 1)) + name="$(printf '%03d-%s.md' "$index" "$agent")" + printf '%s\n' "$token" >"$agent_dir/archive/$name" + { + printf 'from: %s\n' "$agent" + printf 'in-reply-to: %s\n\n' "$name" + printf 'ACK %s\n' "$token" + } >"$scratch/agents/iot/injector/resources/inbox/reply-$name" + done + if [ "$agent" = "iot.claude" ]; then + jq -cn --arg at_ns "0" --argjson argv '["message","delivery","iot.claude"]' \ + '{at_ns:$at_ns,cwd:"/fixture",argv:$argv}' >>"$scratch/metrics/$agent.jsonl" + fi + jq -cn --arg at_ns "1" --argjson argv '["message","ls","'$agent'","--json","--include-body"]' \ + '{at_ns:$at_ns,cwd:"/fixture",argv:$argv}' >>"$scratch/metrics/$agent.jsonl" + jq -cn --arg at_ns "2" --argjson argv '["message","reply","001.md","-m","ACK"]' \ + '{at_ns:$at_ns,cwd:"/fixture",argv:$argv}' >>"$scratch/metrics/$agent.jsonl" + jq -cn --arg at_ns "3" --argjson argv '["message","archive","001.md"]' \ + '{at_ns:$at_ns,cwd:"/fixture",argv:$argv}' >>"$scratch/metrics/$agent.jsonl" + if [ "$agent" = iot.codex ]; then + printf '1\tfirst-batch-archived\t%s\n2\tpost-batch-sent\t%s\n' "$agent" "$agent" >>"$scratch/scenario/events.tsv" + fi +done + +CATALOG="$scratch" ST_ROOT="$scratch" bash "$fixture/judges/outcomes.sh" >/dev/null +CATALOG="$scratch" ST_ROOT="$scratch" ST2_REAL_BIN=/run/current-system/sw/bin/true PATH="$scratch/bin:$PATH" \ + bash "$fixture/judges/measurements.sh" | jq -e ' + .schema == "inbox-one-turn-provider-ab/v1" and + (.subjects | length) == 2 and + ([.subjects[] | select(.agent == "iot.claude")][0] | + (.archive_filenames | length) == 2 and + .cli_processes == 4 and .provider_delivery_cli_processes == 1 and + .first_subject_inbox_operation == "message ls") and + ([.subjects[] | select(.agent == "iot.codex")][0] | + (.archive_filenames | length) == 6 and + .cli_processes == 3 and .provider_delivery_cli_processes == 0 and + .first_subject_inbox_operation == "message ls") + ' >/dev/null +echo "PASS: one-turn provider A/B held-out graders accept the complete synthetic control" diff --git a/cells/inbox-one-turn-provider-ab/inbox-one-turn-provider-ab.kdl b/cells/inbox-one-turn-provider-ab/inbox-one-turn-provider-ab.kdl new file mode 100644 index 00000000..357fec58 --- /dev/null +++ b/cells/inbox-one-turn-provider-ab/inbox-one-turn-provider-ab.kdl @@ -0,0 +1,33 @@ +// Focused #238 provider A/B. Canonical Agent Specs are intentional: both exact st2 heads understand +// the driver-facing argv/deliver declaration, while the older compact eval grammar does not. The +// exact st2 binary remains the only treatment variable. + +host "iot" + +eval { + copy "./fixture" + canonical-agents + + // Arrange the cold backlog before the maintained provider seats launch. + run "seed-cold-backlog" { + command "bash ./injector/seed.sh" + } + + // The injector is the eval supervisor. Subject replies are worker reports; its final requester + // message closes the ordinary canonical-eval done loop without a special completion channel. + message { + from "requester" + to "iot.injector" + content "Run the bounded inbox scenario." + } + max-timeout "900s" + + judges { + judge "OUTCOMES - both providers handle the cold batch and Codex subsequently handles active and post-batch arrivals" { + exec "bash \"$CATALOG/judges/outcomes.sh\"" + } + judge "MEASUREMENTS - emit exact bus CLI sequence, payload bytes, timing, archive names, and runner version hashes" { + exec "bash \"$CATALOG/judges/measurements.sh\"" + } + } +} diff --git a/evidence/inbox-one-turn-provider-ab-runs-20260811.json b/evidence/inbox-one-turn-provider-ab-runs-20260811.json new file mode 100644 index 00000000..40fe5d49 --- /dev/null +++ b/evidence/inbox-one-turn-provider-ab-runs-20260811.json @@ -0,0 +1,183 @@ +{ + "schema": "inbox-one-turn-provider-ab-runs/v1", + "scope": "Matched real-provider A/B for body availability at the st2 delivery boundary. The st2 binary is the treatment; the canonical Agent Spec, fixtures, models, effort, and held-out criteria are fixed.", + "grader_source_commit": "3de9f7e2cffbe4d24da95d4422ed5f687c4b30f9", + "grading_note": "The provider executions completed, but the original judge launcher resolved ./judges from the cell directory and exited 127. The outcomes and measurements below are deterministic regrades of the preserved catalogs with the corrected tracked graders. This receipt does not describe the original judge invocation as green.", + "runs": [ + { + "run_id": "20260811T153932Z-inbox-one-turn-provider-ab-8f042d7", + "cell": "inbox-one-turn-provider-ab", + "completed_at_utc": "2026-08-11T15:39:32Z", + "source_commit": "8f042d797bc2bfd3ad48f2e711d2a2f239fff5da", + "st2_commit": "1d06c4b263a7c5a2a6b8eec1f2e8c4fbea5e2edc", + "st2_version": "st2 0.1.0+1d06c4b", + "st2_binary_sha256": "06997b2c63ddf58678c7a4d024de62d957faf6c2bdc56766c15501e7946c1338", + "harness": "mixed", + "model": "claude-sonnet-5+gpt-5.6-sol", + "effort": "medium", + "duration_seconds": 116, + "result": "FAIL", + "score": "3/4", + "grading": { + "mode": "manual deterministic regrade of preserved catalog", + "catalog_id": "st2e-3538942", + "setup": "PASS", + "provider_completion": "PASS", + "outcomes": "FAIL", + "measurements": "PASS", + "failure": "Codex archived all six messages but acknowledged subject-derived summaries instead of the exact body tokens; Claude passed both exact-body checks." + }, + "measurements": { + "source_payload_bytes": 270, + "claude": { + "cli_processes": 13, + "provider_delivery_cli_processes": 0, + "subject_inbox_cli_processes": 9, + "discovery_cli_processes": 1, + "mutation_cli_processes": 7, + "first_subject_inbox_operation": "message ls", + "cli_span_seconds": 35.746004224, + "archive_count": 2 + }, + "codex": { + "cli_processes": 18, + "provider_delivery_cli_processes": 0, + "subject_inbox_cli_processes": 16, + "discovery_cli_processes": 0, + "mutation_cli_processes": 15, + "first_subject_inbox_operation": "message --help", + "first_subject_cli_to_first_batch_archived_seconds": 78.75810432, + "archive_count": 6 + } + }, + "usage": { + "declared_seats": 2, + "claude": { + "session_id": "f63213bb-d597-4a6c-8858-89935852eed5", + "model_api_calls": null, + "input_tokens": 1836, + "cache_creation_input_tokens": 31048, + "cache_read_input_tokens": 332968, + "output_tokens": 1159, + "token_volume": 367011, + "provider_reported_cost_usd": 0.3060854, + "session_duration_seconds": 114.706 + }, + "codex": { + "session_id": "019ff178-dbdb-7882-be25-b64f760ca56f", + "recorded_model_response_events": 17, + "task_turns": 6, + "tool_call_boundaries": 11, + "input_tokens": 532370, + "cached_input_tokens": 500480, + "output_tokens": 2916, + "reasoning_output_tokens": 944, + "total_tokens": 535286, + "raw_user_message_bytes": 865, + "total_serialized_prompt_bytes": null, + "wall_time_seconds": 115.367 + }, + "api_equivalent_usd": 0.3060854 + }, + "cleanup": { + "status": "green", + "live_process_residue": false, + "production_st2_service_started": false, + "catalog_preserved_for_regrade": true + } + }, + { + "run_id": "20260811T154302Z-inbox-one-turn-provider-ab-117edf4", + "cell": "inbox-one-turn-provider-ab", + "completed_at_utc": "2026-08-11T15:43:02Z", + "source_commit": "117edf4b6872c34a75921ccdbf95e0709541fc96", + "st2_commit": "c1a0f90dd4814ec3ce8067219530d7bd8723e191", + "st2_version": "st2 0.1.0+c1a0f90", + "st2_binary_sha256": "6795c8539ca308fc49b439a3b6495d2bcd2ad4a5ede4cfe02dd53be2f2c67d81", + "harness": "mixed", + "model": "claude-sonnet-5+gpt-5.6-sol", + "effort": "medium", + "duration_seconds": 47, + "result": "PASS", + "score": "4/4", + "grading": { + "mode": "manual deterministic regrade of preserved catalog", + "catalog_id": "st2e-3830609", + "setup": "PASS", + "provider_completion": "PASS", + "outcomes": "PASS", + "measurements": "PASS", + "outcome": "Claude replied on and archived both exact cold bodies. Codex replied on and archived all six exact bodies, including the bounded burst and the post-batch arrival." + }, + "measurements": { + "source_payload_bytes": 270, + "claude": { + "cli_processes": 13, + "provider_delivery_cli_processes": 2, + "subject_inbox_cli_processes": 6, + "discovery_cli_processes": 0, + "mutation_cli_processes": 6, + "first_subject_inbox_operation": "message reply", + "cli_span_seconds": 23.496061696, + "archive_count": 2 + }, + "codex": { + "cli_processes": 14, + "provider_delivery_cli_processes": 0, + "subject_inbox_cli_processes": 13, + "discovery_cli_processes": 0, + "mutation_cli_processes": 13, + "first_subject_inbox_operation": "message reply", + "first_subject_cli_to_first_batch_archived_seconds": 28.934609664, + "archive_count": 6 + } + }, + "usage": { + "declared_seats": 2, + "claude": { + "session_id": "dfee460d-0146-46cf-984e-21b17b20a92c", + "model_api_calls": null, + "input_tokens": 1834, + "cache_creation_input_tokens": 29862, + "cache_read_input_tokens": 287080, + "output_tokens": 1081, + "token_volume": 319857, + "provider_reported_cost_usd": 0.284027, + "session_duration_seconds": 45.315 + }, + "codex": { + "session_id": "019ff17d-1eca-7740-b020-1eb15f445455", + "recorded_model_response_events": 7, + "task_turns": 3, + "tool_call_boundaries": 4, + "input_tokens": 226271, + "cached_input_tokens": 196864, + "output_tokens": 1600, + "reasoning_output_tokens": 662, + "total_tokens": 227871, + "raw_user_message_bytes": 3116, + "total_serialized_prompt_bytes": null, + "wall_time_seconds": 45.676 + }, + "api_equivalent_usd": 0.284027 + }, + "cleanup": { + "status": "green", + "live_process_residue": false, + "production_st2_service_started": false, + "catalog_preserved_for_regrade": true + } + } + ], + "comparison": { + "functional_outcome": "FAIL to PASS", + "codex_total_tokens_reduction_percent": 57.43, + "codex_recorded_model_response_events_reduction_percent": 58.82, + "codex_tool_call_boundaries_reduction_percent": 63.64, + "codex_wall_time_reduction_percent": 60.41, + "claude_token_volume_reduction_percent": 12.85, + "claude_provider_reported_cost_reduction_percent": 7.21, + "claude_session_duration_reduction_percent": 60.49, + "interpretation": "Body-bearing bounded delivery increased useful Codex user-message bytes while eliminating repeated inference context and discovery turns." + } +} diff --git a/evidence/run-history.tsv b/evidence/run-history.tsv index cbf4bb46..64ddc0c5 100644 --- a/evidence/run-history.tsv +++ b/evidence/run-history.tsv @@ -3,3 +3,5 @@ run_id cell completed_at_utc source_commit st2_commit harness model effort durat 20260728T155735Z-weird-git-setup-a49fb0f weird-git-setup 2026-07-28T15:57:35Z a49fb0ffa00c525e361b3178fb7c35b93538501f 9887b2842222def0838c2cd82e6c24c218f7efa6 Claude claude-sonnet-5 medium 604 FAIL 5/6 {"declared_seats":1,"calls":19,"input_tokens":70,"cache_creation_input_tokens":121662,"cache_read_input_tokens":1508376,"output_tokens":6040,"api_equivalent_usd":1.27329480} green evidence/stable-main-phase4-runs-20260728.json 20260728T195359Z-pty-send-peek-b7da882 pty-send-peek 2026-07-28T19:53:59Z b7da8826b503e0408e309af8702fb684d28e3a19 9887b2842222def0838c2cd82e6c24c218f7efa6 model-free - - 6 PASS 13/13 {"declared_seats":0,"calls":0,"input_tokens":0,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"output_tokens":0,"api_equivalent_usd":0} green evidence/stable-main-phase4-runs-20260728.json 20260728T201020Z-weird-git-setup-28d7fb6 weird-git-setup 2026-07-28T20:10:20Z 28d7fb6e1235dc3680081bb3ee4e67109dc10c89 9887b2842222def0838c2cd82e6c24c218f7efa6 Claude claude-sonnet-5 medium 604 PASS 6/6 {"declared_seats":1,"calls":19,"input_tokens":38,"cache_creation_input_tokens":60136,"cache_read_input_tokens":811031,"output_tokens":3004,"api_equivalent_usd":0.51399330} green evidence/stable-main-phase4-runs-20260728.json +20260811T153932Z-inbox-one-turn-provider-ab-8f042d7 inbox-one-turn-provider-ab 2026-08-11T15:39:32Z 8f042d797bc2bfd3ad48f2e711d2a2f239fff5da 1d06c4b263a7c5a2a6b8eec1f2e8c4fbea5e2edc mixed claude-sonnet-5+gpt-5.6-sol medium 116 FAIL 3/4 {"declared_seats":2,"claude":{"input_tokens":1836,"cache_creation_input_tokens":31048,"cache_read_input_tokens":332968,"output_tokens":1159,"token_volume":367011,"provider_reported_cost_usd":0.3060854},"codex":{"recorded_model_response_events":17,"tool_call_boundaries":11,"input_tokens":532370,"cached_input_tokens":500480,"output_tokens":2916,"reasoning_output_tokens":944,"total_tokens":535286},"api_equivalent_usd":0.3060854} green evidence/inbox-one-turn-provider-ab-runs-20260811.json +20260811T154302Z-inbox-one-turn-provider-ab-117edf4 inbox-one-turn-provider-ab 2026-08-11T15:43:02Z 117edf4b6872c34a75921ccdbf95e0709541fc96 c1a0f90dd4814ec3ce8067219530d7bd8723e191 mixed claude-sonnet-5+gpt-5.6-sol medium 47 PASS 4/4 {"declared_seats":2,"claude":{"input_tokens":1834,"cache_creation_input_tokens":29862,"cache_read_input_tokens":287080,"output_tokens":1081,"token_volume":319857,"provider_reported_cost_usd":0.284027},"codex":{"recorded_model_response_events":7,"tool_call_boundaries":4,"input_tokens":226271,"cached_input_tokens":196864,"output_tokens":1600,"reasoning_output_tokens":662,"total_tokens":227871},"api_equivalent_usd":0.284027} green evidence/inbox-one-turn-provider-ab-runs-20260811.json