diff --git a/AGENT-SPEC.md b/AGENT-SPEC.md index f1e46481..07ea1b86 100644 --- a/AGENT-SPEC.md +++ b/AGENT-SPEC.md @@ -1,8 +1,8 @@ # Canonical st2 agent specification This is the sole agent-authoring specification for this repository. It is pinned to st2 -[`9887b2842222def0838c2cd82e6c24c218f7efa6`](https://github.com/compoundingtech/st2/commit/9887b2842222def0838c2cd82e6c24c218f7efa6) -(`0.1.0`, source `9887b28`). It documents the hand-authored KDL accepted at that commit. Do not infer +[`c6846f6239329f0803142afc06c15a07b93937c1`](https://github.com/compoundingtech/st2/commit/c6846f6239329f0803142afc06c15a07b93937c1) +(`0.1.0`, source `c6846f6`). It documents the hand-authored KDL accepted at that commit. Do not infer additional fields or commands from older corpus fixtures. st2 runs long-lived `service` agents made of interactive `pty` tasks and terminal-free `exec` tasks. @@ -248,6 +248,20 @@ idempotently clears a restored inbox copy. New post-start unread work sends: Subject and sender are normalized to one bounded printable line. Delivery is one bracketed-paste PTY send, 500 ms, then Return. DING does not inspect terminal pixels or classify modal state. +The issue #57 experimental matrix separately pins corrected st2 PR #123 at +`d7500b0fcad8bb268da9da96c0226d9caddbe305`. In that opt-in path, a compact +`ding { adapter { argv ... } }` lowers to a generated exec sidecar whose argv is +expanded only after the complete managed task environment is known. Core +consumes a provider-neutral activity/input-buffer lease, requires exact PTY +generation/activity sequence plus I/O revision, and performs one atomic +guarded send only for fresh `idle` + `empty`. A generic `ding-control +hook-owned` receipt removes exact unread filenames from PTY work when an +already-occurring post-turn hook owns their next-context delivery. The +integrated model-free A/B covers 11 durable messages: the aggressive arm makes +six non-idle/colliding PTY writes and the configured hook-plus-idle arm makes +zero. Real-provider fixtures, live-pane acceptance, merge, and release remain +outside that evidence. + Arrivals remain FIFO. `busy` and `away` do not suppress delivery; only fresh `dnd` does. Non-DND presence is refreshed every five minutes, while abandoned DND ages to derived `unknown` after 15 minutes and delivery resumes. Failed PTY sends retain the queue head. Startup backlog produces one generic unread-work DING rather @@ -432,8 +446,7 @@ Inspect the declaration, every referenced template, and every workspace destinat materialization command. Materialization is byte-idempotent and does not imply hook installation. Starting the network is a separate, explicitly authorized action. -For source `9887b28`, the accepted Linux executable has SHA256 -`d49d44fd4f3f6f655455c212353a469fefa956082bedf22163deb767d8a36a0d`; its published archive has SHA256 -`32ee103bd17ccb3e155ac63d816a3906c2470a3c98e3cc04b56e5a67138b9927`. `bin/check-corpus.sh` verifies -the variable-age version contract, exact installed binary, embedded full source commit, strict semantic +For source `c6846f6239329f0803142afc06c15a07b93937c1`, the accepted local-source Linux executable has +SHA256 `2bba8d58be24250bc262f75f835ce2d780369add275774f3f2135c623d23d29c`. `bin/check-corpus.sh` verifies +the variable-age version contract, exact installed binary, shared source-pin consistency, strict semantic validation, fixture resets, and the rest of the model-free corpus gate before an eval may run. diff --git a/CATALOG.md b/CATALOG.md index 58cd2eb1..50314943 100644 --- a/CATALOG.md +++ b/CATALOG.md @@ -1,7 +1,7 @@ # Current eval catalog and evidence -This is the sole current corpus surface: **45 included cells** (14 model-free, +This is the sole current corpus surface: **51 included cells** (20 model-free, 31 model-backed) and **4 retired exclusions**. Inventory fields are derived from the active KDL and executable judge scripts. Latest accepted PASS evidence remains distinct from the append-only run history: a failed last run is visible without being advertised as accepted evidence, and cells with no @@ -18,7 +18,13 @@ harness-native loader plus canonical hook file. |---|---|---|---:|---|---|---:|---|---| | `context-resource-continuity` | model-free | — | 0 | none | `90s` | 4 | — | **NO STRUCTURED RUN** | | `crash-ding` | mixed | `claude-sonnet-5+gpt-5.6-sol` / medium | 4 | high | `180s` | 5 | — | **NO STRUCTURED RUN** | +| `ding-adapter-fixture-contract` | model-free | — | 0 | none | `30s` | 7 | — | **NO STRUCTURED RUN** | +| `ding-delivery-a-aggressive-control` | model-free | — | 0 | none | `45s` | 7 | — | **NO STRUCTURED RUN** | +| `ding-delivery-pty-ab` | model-free | — | 0 | none | `120s` | 8 | — | **NO STRUCTURED RUN** | +| `ding-delivery-st2-rich-ab` | model-free | — | 0 | none | `120s` | 10 | — | **NO STRUCTURED RUN** | | `ding-mode` | Claude | `claude-sonnet-5` / medium | 2 | medium | `1200s` | 4 | — | **NO STRUCTURED RUN** | +| `ding-pty-activity-lease` | model-free | — | 0 | none | `60s` | 8 | — | **NO STRUCTURED RUN** | +| `ding-pty-guarded-send` | model-free | — | 0 | none | `90s` | 8 | — | **NO STRUCTURED RUN** | | `ding-reply` | Claude | `claude-sonnet-5` / medium | 1 | low | `420s` | 2 | — | **NO STRUCTURED RUN** | | `docs` | Claude | `claude-sonnet-5` / medium | 3 | high | `1200s` | 5 | — | **NO STRUCTURED RUN** | | `exec-lifecycle-logging` | model-free | — | 0 | none | `90s` | 5 | — | **NO STRUCTURED RUN** | @@ -82,6 +88,12 @@ while the last-run column makes a recorded failure distinct from a cell with no ## Harness hook exclusions - `context-resource-continuity` / `cell` — deterministic native context/resource restart probe; no harness seat +- `ding-adapter-fixture-contract` / `cell` — deterministic fixture provenance and blocker contract; no bus-connected agent +- `ding-delivery-a-aggressive-control` / `cell` — deterministic unconfigured DING ordinary-input control; no harness or model agent +- `ding-delivery-st2-rich-ab` / `cell` — deterministic exact-source configured generic-adapter versus aggressive DING A/B; no harness or model agent +- `ding-delivery-pty-ab` / `cell` — deterministic PTY-boundary aggressive-versus-guarded delivery A/B; no harness or model agent +- `ding-pty-activity-lease` / `cell` — deterministic exact-source PTY activity-lease probe; no harness or model agent +- `ding-pty-guarded-send` / `cell` — deterministic exact-source PTY compare-and-send race probe; no harness or model agent - `docs` / `judge:cold-reader` — one-shot offline Claude print grader; no bus identity, DING, or hook surface - `exec-lifecycle-logging` / `cell` — deterministic native exec lifecycle probe; no harness seat - `hook-integrity` / `cell` — deterministic native hook-installation probe; no live harness seat @@ -97,7 +109,7 @@ while the last-run column makes a recorded failure distinct from a cell with no - `targeted-reconcile-isolation` / `cell` — deterministic selected reconcile isolation probe; no bus-connected agent - `two-networks-coexist` / `cell` — deterministic catalog-isolation probe; no harness seat -These **15** rows are generated from structured exclusions. A cell-level row is required for +These **21** rows are generated from structured exclusions. A cell-level row is required for every derived model-free cell. The one-shot docs cold reader is a model-backed grader, but not a bus-connected agent: it has no st2 identity, DING sidecar, or lifecycle hooks. diff --git a/README.md b/README.md index 37808f25..0466ccfd 100644 --- a/README.md +++ b/README.md @@ -96,10 +96,26 @@ VERDICT: PASS ``` Requirements are `st2 0.1.0` from source -[`9887b28`](https://github.com/compoundingtech/st2/commit/9887b2842222def0838c2cd82e6c24c218f7efa6), +[`c6846f6`](https://github.com/compoundingtech/st2/commit/c6846f6239329f0803142afc06c15a07b93937c1), +using the accepted local-source Linux binary SHA256 +`2bba8d58be24250bc262f75f835ce2d780369add275774f3f2135c623d23d29c`; `pty`, Bash, Git, `jq`, Rust/Cargo for the pinned KDL parser gate, and Node for JavaScript fixtures. A paid cell also needs every harness named by its dry-run row. +The issue #57 PTY draft experiments are separately pinned to activity PR #131 +`46c71d31c0d6daee43adf568061b2b84a65ae8c0` and stacked guarded-send PR #133 +`743ceb796a41a3282e31382575bff0d0e3826d59`. Run their model-free cells with +`EVALS_PTY_PR133_ROOT` pointing to a clean exact-head checkout built by +`npm ci && npm run build`; the cells reject any other Git head or package lock. + +The integrated configured-DING A/B additionally pins corrected st2 PR #123 +`d7500b0fcad8bb268da9da96c0226d9caddbe305` and its accepted local-source Linux +release binary SHA256 +`705ad3ebd0bce497a4117c7c7993505c0579fc1f7df2421e0525a22208f6949f`. +Set `EVALS_ST2_PR123_ROOT` to a clean exact-head checkout built with +`cargo build --release --locked`; the cell launches the generated configured +sidecar from that artifact and rejects a different source, lockfile, or binary. + ## Cell layout ```text diff --git a/bin/check-corpus.sh b/bin/check-corpus.sh index fc85e69d..5bde6c79 100755 --- a/bin/check-corpus.sh +++ b/bin/check-corpus.sh @@ -5,26 +5,20 @@ set -euo pipefail repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" cd "$repo_root" -expected_source="9887b28" -expected_source_full="9887b2842222def0838c2cd82e6c24c218f7efa6" -expected_binary_sha256="d49d44fd4f3f6f655455c212353a469fefa956082bedf22163deb767d8a36a0d" -expected_version_regex='^st2 0\.1\.0 — running from local source \(9887b28, .+ ago\)$' +source bin/st2-pin.sh +bin/check-st2-pin-consistency.sh st2_path="$(command -v st2)" actual_version="$(st2 --version)" -[[ "$actual_version" =~ $expected_version_regex ]] || { - echo "FAIL: expected st2 0.1.0 from pinned source $expected_source, found $actual_version" >&2 +[[ "$actual_version" =~ $ST2_VERSION_REGEX ]] || { + echo "FAIL: expected st2 0.1.0 from pinned source $ST2_SOURCE_SHORT, found $actual_version" >&2 exit 1 } actual_binary_sha256="$(sha256sum "$st2_path" | awk '{ print $1 }')" -[ "$actual_binary_sha256" = "$expected_binary_sha256" ] || { - echo "FAIL: expected st2 binary sha256 $expected_binary_sha256, found $actual_binary_sha256 at $st2_path" >&2 +[ "$actual_binary_sha256" = "$ST2_BINARY_SHA256" ] || { + echo "FAIL: expected st2 binary sha256 $ST2_BINARY_SHA256, found $actual_binary_sha256 at $st2_path" >&2 exit 1 } -LC_ALL=C grep -aFq "$expected_source_full" "$st2_path" || { - echo "FAIL: st2 binary at $st2_path does not embed full pinned source $expected_source_full" >&2 - exit 1 -} -echo "PASS: pinned runner source $expected_source ($actual_version; sha256 $actual_binary_sha256)" +echo "PASS: pinned runner source $ST2_SOURCE_FULL ($actual_version; sha256 $actual_binary_sha256)" mapfile -d '' shell_files < <( find bin cells -type f -name '*.sh' -not -path '*/_git/*' -print0 | sort -z diff --git a/bin/check-preflight-safety.sh b/bin/check-preflight-safety.sh index 4af8f41e..978b114f 100755 --- a/bin/check-preflight-safety.sh +++ b/bin/check-preflight-safety.sh @@ -20,6 +20,7 @@ reachable=( bin/check-preflight-safety.sh bin/check-run-history.sh bin/check-retired-surfaces.sh + bin/check-st2-pin-consistency.sh bin/check-st2-semantic.sh bin/check-vrs-scope-drift.sh bin/check-vrs-variations.sh @@ -27,6 +28,7 @@ reachable=( bin/corpus-inventory.sh bin/generate-catalog.sh bin/model-seat-inventory.sh + bin/st2-pin.sh ) materializers=( cells/signal-rename/fixture/materialize.sh @@ -68,6 +70,7 @@ expected_direct=( bin/check-preflight-safety.sh bin/check-retired-surfaces.sh bin/check-run-history.sh + bin/check-st2-pin-consistency.sh bin/check-st2-semantic.sh bin/check-vrs-scope-drift.sh bin/check-vrs-variations.sh diff --git a/bin/check-st2-pin-consistency.sh b/bin/check-st2-pin-consistency.sh new file mode 100755 index 00000000..13b370ab --- /dev/null +++ b/bin/check-st2-pin-consistency.sh @@ -0,0 +1,21 @@ +#!/usr/bin/env bash +set -euo pipefail + +repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +cd "$repo_root" +source bin/st2-pin.sh + +grep -Fq "$ST2_SOURCE_FULL" AGENT-SPEC.md +grep -Fq "$ST2_SOURCE_FULL" README.md +grep -Fq "$ST2_BINARY_SHA256" AGENT-SPEC.md +grep -Fq "$ST2_BINARY_SHA256" README.md +grep -Fq 'source bin/st2-pin.sh' bin/check-corpus.sh + +if rg -n \ + '9887b28|d49d44fd4f3f6f655455c212353a469fefa956082bedf22163deb767d8a36a0d' \ + AGENT-SPEC.md README.md bin/check-corpus.sh bin/st2-pin.sh; then + echo "FAIL: active st2 pin surfaces retain the superseded source or binary hash" >&2 + exit 1 +fi + +echo "PASS: active spec, README, and executable preflight share st2 $ST2_SOURCE_FULL / $ST2_BINARY_SHA256" diff --git a/bin/st2-pin.sh b/bin/st2-pin.sh new file mode 100755 index 00000000..f784ad21 --- /dev/null +++ b/bin/st2-pin.sh @@ -0,0 +1,6 @@ +#!/usr/bin/env bash + +ST2_SOURCE_SHORT="c6846f6" +ST2_SOURCE_FULL="c6846f6239329f0803142afc06c15a07b93937c1" +ST2_BINARY_SHA256="2bba8d58be24250bc262f75f835ce2d780369add275774f3f2135c623d23d29c" +ST2_VERSION_REGEX='^st2 0\.1\.0 — running from local source \(c6846f6, .+ ago\)$' diff --git a/cells/ding-adapter-fixture-contract/README.md b/cells/ding-adapter-fixture-contract/README.md new file mode 100644 index 00000000..50e9e12d --- /dev/null +++ b/cells/ding-adapter-fixture-contract/README.md @@ -0,0 +1,59 @@ +# ding-adapter-fixture-contract + +Hermetic, model-free contract inventory for the optional DING activity-lease +matrix tracked by [evals#57](https://github.com/compoundingtech/evals/issues/57). + +This cell validates only inputs that are safe to establish before the product +seams exist: + +- nine shared, synthetic ASCII screen diagnostics; +- exact content hashes and provenance metadata; +- symmetric, explicitly blocked activity-event and post-turn-hook fixture + slots for the two maintained provider adapters; +- matched A/B outcome and generic conditional-send race inventories; and +- the external dependency/evidence ledger, including exact runnable-draft PTY + heads and the still-blocked adapter/core seams. + +The screen text and requested terminal diagnostics are never activity +authority. No provider event payload or adapter interface is guessed here. +The exact PTY activity and guarded-send product surfaces are now exercised by +`ding-pty-activity-lease` and `ding-pty-guarded-send` against stacked draft head +`743ceb796a41a3282e31382575bff0d0e3826d59`, which contains activity head +`46c71d31c0d6daee43adf568061b2b84a65ae8c0`. Those runs are local exact-source +experiment evidence, not release artifacts. Corrected st2 PR #123 at +`d7500b0fcad8bb268da9da96c0226d9caddbe305` now supplies the generic configured +sidecar, managed-task-env argv expansion, guarded-send integration, DND/error +holds, typed ownership receipts, and generic hook-control ingress. The +`ding-delivery-st2-rich-ab` cell exercises that draft end to end with a +synthetic adapter; merge/release and real-provider acceptance remain blocked. + +The intended opt-in shape is an external adapter command selected by DING +through a generic protocol. Provider commands may differ, but core st2 and the +shared eval judge must not branch on provider identity. The generic +selection/envelope and hook ownership contract are pinned by st2 PR #123. +Provider-specific activity translation, composer evidence, and hook payload +fixtures remain blocked on immutable owner artifacts. + +The A/B inventory freezes the intended comparison without pretending it can +execute unfinished behavior. The conditional-send inventory names races and +outcomes only: a provider adapter must establish fresh idle plus an empty +composer, while PTY must atomically reject a DING candidate if its live +generation or generic I/O/snapshot revision changed. PTY does not classify +submit, newline, or human intent. + +## Validation boundary + +The scaffold cell, parser, semantic loader, fixture-reset, harness-exclusion, +preflight-reachability, generated-catalog, and full corpus gates are +model-free. The accepted successor runner is exact source +`c6846f6239329f0803142afc06c15a07b93937c1` with local-source Linux binary +SHA256 `2bba8d58be24250bc262f75f835ce2d780369add275774f3f2135c623d23d29c`. +Do not rebuild or substitute a different binary to claim the full corpus gate. + +The PTY cells additionally require `EVALS_PTY_PR133_ROOT` to point to a +clean exact-head checkout after `npm ci && npm run build`. They verify the Git +head, activity-base ancestry, package-lock hash, and runtime artifact hashes in +their receipts. The integrated cell also requires `EVALS_ST2_PR123_ROOT` at +exact corrected head `d7500b0fcad8bb268da9da96c0226d9caddbe305` after +`cargo build --release --locked` and pins binary SHA256 +`705ad3ebd0bce497a4117c7c7993505c0579fc1f7df2421e0525a22208f6949f`. diff --git a/cells/ding-adapter-fixture-contract/ding-adapter-fixture-contract.kdl b/cells/ding-adapter-fixture-contract/ding-adapter-fixture-contract.kdl new file mode 100644 index 00000000..4c3f3426 --- /dev/null +++ b/cells/ding-adapter-fixture-contract/ding-adapter-fixture-contract.kdl @@ -0,0 +1,33 @@ +// Seam-independent fixture and provenance contract for the optional DING activity lease. +eval { + copy "./fixture" + max-timeout "30s" + + run "fixture-contract" { + command "bash ./check-fixtures.sh" + } + + judges { + judge "DIAGNOSTICS — nine shared ASCII screens are exact, hashed, and non-authoritative" { + exec "grep -Fqx DIAGNOSTIC-FIXTURES-GREEN-57d1 $RUNS_DIR/fixture-contract.out" + } + judge "PARITY SLOTS — every declared provider reserves the same activity and post-turn hook cases" { + exec "grep -Fqx PROVIDER-SLOTS-GREEN-57d1 $RUNS_DIR/fixture-contract.out" + } + judge "A/B CONTRACT — matched workloads preserve durability while B eliminates active-turn PTY input" { + exec "grep -Fqx AB-INVENTORY-GREEN-57d1 $RUNS_DIR/fixture-contract.out" + } + judge "CONDITIONAL SEND — generic I/O races reject the DING candidate without terminal heuristics" { + exec "grep -Fqx CONDITIONAL-SEND-INVENTORY-GREEN-57d1 $RUNS_DIR/fixture-contract.out" + } + judge "PROVENANCE — every ready asset has reviewed metadata and every provider asset is explicitly blocked" { + exec "grep -Fqx PROVENANCE-INVENTORY-GREEN-57d1 $RUNS_DIR/fixture-contract.out" + } + judge "SEAMS — exact draft PTY/st2 surfaces and remaining provider blockers are classified without invention" { + exec "grep -Fqx PRODUCT-BLOCKERS-GREEN-57d1 $RUNS_DIR/fixture-contract.out && grep -Fqx PTY-DRAFT-SEAMS-GREEN-57d1 $RUNS_DIR/fixture-contract.out" + } + judge "MODEL FREE — the scaffold contains no provider payload, provider launch, network, or model call" { + exec "grep -Fqx HERMETIC-SKELETON-GREEN-57d1 $RUNS_DIR/fixture-contract.out" + } + } +} diff --git a/cells/ding-adapter-fixture-contract/fixture/ab-cases.tsv b/cells/ding-adapter-fixture-contract/fixture/ab-cases.tsv new file mode 100644 index 00000000..c0657d08 --- /dev/null +++ b/cells/ding-adapter-fixture-contract/fixture/ab-cases.tsv @@ -0,0 +1,10 @@ +case_id a_delivery b_delivery durable_expectation fifo_expectation model_calls b_pty_bytes b_collision_goal +idle aggressive-pty bounded-idle-wake eventual-exactly-once preserve 0 exactly-one zero +active-turn aggressive-pty next-hook-context eventual-exactly-once preserve 0 zero zero +long-child aggressive-pty hold-until-authorized eventual-exactly-once preserve 0 zero zero +dnd dnd-hold-then-deliver dnd-hold-then-authorize eventual-exactly-once preserve 0 zero-while-held zero +stale-unknown aggressive-pty hold-until-authorized eventual-exactly-once preserve 0 zero zero +hook-failure aggressive-pty durable-hold-until-authorized eventual-exactly-once preserve 0 zero zero +crash-restart restart-backlog durable-hold-until-authorized eventual-exactly-once preserve 0 zero-before-authorization zero +compaction-clear aggressive-pty hold-until-authorized eventual-exactly-once preserve 0 zero zero +fifo-burst aggressive-fifo authorized-fifo eventual-exactly-once preserve 0 bounded-by-messages zero diff --git a/cells/ding-adapter-fixture-contract/fixture/blockers.tsv b/cells/ding-adapter-fixture-contract/fixture/blockers.tsv new file mode 100644 index 00000000..cddd8af1 --- /dev/null +++ b/cells/ding-adapter-fixture-contract/fixture/blockers.tsv @@ -0,0 +1,8 @@ +blocker_id owner dependency required_evidence status +pty-activity pty-owner https://github.com/compoundingtech/pty/pull/131 exact head 46c71d31c0d6daee43adf568061b2b84a65ae8c0 is contained by stacked 743ceb796a41a3282e31382575bff0d0e3826d59; public activity lease and STATUS are covered by ding-pty-activity-lease; merge and release artifact remain present-draft +pty-conditional-send pty-owner https://github.com/compoundingtech/pty/pull/133 exact head 743ceb796a41a3282e31382575bff0d0e3826d59 supplies generation, ioRevision, compareAndSend, and combined fixtures; covered by ding-pty-guarded-send; merge and release artifact remain present-draft +pty-diagnostics pty-owner https://github.com/compoundingtech/pty/pull/131 alternateScreen is present as a non-authoritative diagnostic; composer-empty and input-origin semantics deliberately remain outside PTY partial-draft +st2-adapter st2-owner https://github.com/compoundingtech/st2/pull/123 exact corrected head d7500b0fcad8bb268da9da96c0226d9caddbe305 and binary 705ad3ebd0bce497a4117c7c7993505c0579fc1f7df2421e0525a22208f6949f provide generic selection/envelope, final-task-env argv, guarded send, DND/error holds, typed receipts, and hook ownership; integrated synthetic matrix is green; merge and release remain present-draft +provider-a-fixtures provider-adapter-owner immutable first-provider adapter fixtures verbatim activity and post-turn hook schema provenance, redaction review, content hashes, and owner approval pending +provider-b-fixtures provider-adapter-owner immutable second-provider adapter fixtures verbatim activity and post-turn hook schema provenance, redaction review, content hashes, and owner approval pending +real-pane-acceptance evals-owner https://github.com/compoundingtech/st2/issues/111 exact merged seams, immutable adapters, independent readiness review, and separate live-execution authorization blocked diff --git a/cells/ding-adapter-fixture-contract/fixture/cases.tsv b/cells/ding-adapter-fixture-contract/fixture/cases.tsv new file mode 100644 index 00000000..f97240d1 --- /dev/null +++ b/cells/ding-adapter-fixture-contract/fixture/cases.tsv @@ -0,0 +1,10 @@ +case_id screen_fixture expected_activity diagnostic_case authority +active-turn screens/active-turn.txt active none live-activity-only +idle-prompt screens/idle-prompt.txt idle none live-activity-only +long-child screens/long-child.txt child_command none live-activity-only +alternate-screen screens/alternate-screen.txt unknown alternate-screen live-activity-only +raw-mode screens/raw-mode.txt unknown raw-input live-activity-only +restore screens/restore.txt idle terminal-restore live-activity-only +compaction screens/compaction.txt unknown compaction live-activity-only +clear screens/clear.txt unknown clear live-activity-only +crash screens/crash.txt unknown crash live-activity-only diff --git a/cells/ding-adapter-fixture-contract/fixture/check-fixtures.sh b/cells/ding-adapter-fixture-contract/fixture/check-fixtures.sh new file mode 100755 index 00000000..47d7ec8b --- /dev/null +++ b/cells/ding-adapter-fixture-contract/fixture/check-fixtures.sh @@ -0,0 +1,268 @@ +#!/usr/bin/env bash +set -euo pipefail + +fixture_root="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +cases="$fixture_root/cases.tsv" +hook_cases="$fixture_root/hook-cases.tsv" +providers="$fixture_root/providers.tsv" +ab_cases="$fixture_root/ab-cases.tsv" +input_race_cases="$fixture_root/input-race-cases.tsv" +provenance="$fixture_root/provenance.tsv" +blockers="$fixture_root/blockers.tsv" + +fail() { + printf 'FAIL: %s\n' "$*" >&2 + exit 1 +} + +header="$(head -n 1 "$cases")" +[ "$header" = $'case_id\tscreen_fixture\texpected_activity\tdiagnostic_case\tauthority' ] || + fail "cases.tsv header changed" + +expected_cases=( + active-turn + alternate-screen + clear + compaction + crash + idle-prompt + long-child + raw-mode + restore +) +mapfile -t actual_cases < <(tail -n +2 "$cases" | cut -f1 | LC_ALL=C sort) +[ "${actual_cases[*]}" = "${expected_cases[*]}" ] || + fail "case IDs differ from the reviewed nine-case matrix" +[ "$(printf '%s\n' "${actual_cases[@]}" | uniq -d | wc -l)" -eq 0 ] || + fail "duplicate case ID" + +provenance_header="$(head -n 1 "$provenance")" +[ "$provenance_header" = $'asset_id\tkind\tcase_id\tproduct\tsource\tschema_version\tderivation\tredaction_review\tsha256\towner\tstatus\texpected_result' ] || + fail "provenance.tsv header changed" +[ -z "$(tail -n +2 "$provenance" | cut -f1 | LC_ALL=C sort | uniq -d)" ] || + fail "duplicate provenance asset ID" + +provider_header="$(head -n 1 "$providers")" +[ "$provider_header" = $'adapter_id\tfixture_owner\tactivity_status\thook_status' ] || + fail "providers.tsv header changed" +mapfile -t adapter_ids < <(tail -n +2 "$providers" | cut -f1 | LC_ALL=C sort) +[ "${#adapter_ids[@]}" -eq 2 ] || fail "expected two maintained provider fixture inventories" +[ -z "$(printf '%s\n' "${adapter_ids[@]}" | uniq -d)" ] || + fail "duplicate adapter fixture ID" +while IFS=$'\t' read -r adapter_id fixture_owner activity_status hook_status; do + [ "$adapter_id" != "adapter_id" ] || continue + [ -n "$adapter_id" ] && [ -n "$fixture_owner" ] || + fail "provider fixture inventory omits identity or owner" + [ "$activity_status" = "blocked" ] && [ "$hook_status" = "blocked" ] || + fail "$adapter_id claims fixtures before reviewed artifacts exist" +done <"$providers" + +screen_count=0 +provider_count=0 +while IFS=$'\t' read -r case_id screen expected_activity diagnostic authority; do + [ "$case_id" != "case_id" ] || continue + case "$expected_activity" in + unknown|active|child_command|idle) ;; + *) fail "$case_id has invalid generic expected activity $expected_activity" ;; + esac + [ "$authority" = "live-activity-only" ] || + fail "$case_id makes a diagnostic fixture authoritative" + [ -n "$diagnostic" ] || fail "$case_id omits its diagnostic classification" + + screen_path="$fixture_root/$screen" + [ -f "$screen_path" ] || fail "$case_id screen is missing: $screen" + if ! LC_ALL=C tr -d '\11\12\15\40-\176' <"$screen_path" | cmp -s - /dev/null; then + fail "$case_id screen is not stable ASCII" + fi + row="$(awk -F '\t' -v id="screen/$case_id" '$1 == id { print }' "$provenance")" + [ -n "$row" ] || fail "$case_id screen has no provenance row" + IFS=$'\t' read -r asset_id kind row_case product source schema derivation redaction expected_hash owner status row_activity <<<"$row" + [ "$asset_id" = "screen/$case_id" ] && + [ "$kind" = "ascii-screen" ] && + [ "$row_case" = "$case_id" ] && + [ "$product" = "generic" ] && + [ "$source" = "evals#57" ] && + [ "$schema" = "diagnostic-ascii-v1" ] && + [ "$derivation" = "synthetic" ] && + [ "$redaction" = "not-required-synthetic" ] && + [ "$owner" = "evals" ] && + [ "$status" = "ready" ] && + [ "$row_activity" = "$expected_activity" ] || + fail "$case_id screen provenance is incomplete" + [ "$(sha256sum "$screen_path" | cut -d' ' -f1)" = "$expected_hash" ] || + fail "$case_id screen hash differs from provenance" + ((screen_count += 1)) + + for provider in "${adapter_ids[@]}"; do + provider_row="$(awk -F '\t' -v id="$provider/$case_id" '$1 == id { print }' "$provenance")" + [ -n "$provider_row" ] || fail "$provider/$case_id slot is missing" + IFS=$'\t' read -r provider_asset provider_kind provider_case provider_name provider_source provider_schema provider_derivation provider_redaction provider_hash provider_owner provider_status provider_activity <<<"$provider_row" + [ "$provider_asset" = "$provider/$case_id" ] && + [ "$provider_kind" = "adapter-event" ] && + [ "$provider_case" = "$case_id" ] && + [ "$provider_name" = "$provider" ] && + [ "$provider_source" = "pending-owner-artifact" ] && + [ "$provider_schema" = "pending" ] && + [ "$provider_derivation" = "none" ] && + [ "$provider_redaction" = "pending" ] && + [ "$provider_hash" = "-" ] && + [ "$provider_owner" = "pending" ] && + [ "$provider_status" = "blocked" ] && + [ "$provider_activity" = "$expected_activity" ] || + fail "$provider/$case_id slot invents or omits provenance" + ((provider_count += 1)) + done +done <"$cases" + +[ "$screen_count" -eq 9 ] || fail "expected nine screen fixtures" +[ "$provider_count" -eq $((screen_count * ${#adapter_ids[@]})) ] || + fail "expected symmetric activity fixture slots for every provider" + +hook_header="$(head -n 1 "$hook_cases")" +[ "$hook_header" = $'case_id\texpected_outcome\tauthority' ] || + fail "hook-cases.tsv header changed" +expected_hook_cases=( + hook-failure + post-turn-empty + post-turn-multiple + post-turn-unread +) +mapfile -t actual_hook_cases < <(tail -n +2 "$hook_cases" | cut -f1 | LC_ALL=C sort) +[ "${actual_hook_cases[*]}" = "${expected_hook_cases[*]}" ] || + fail "post-turn hook case IDs differ from the reviewed matrix" + +hook_slot_count=0 +while IFS=$'\t' read -r hook_case expected_outcome authority; do + [ "$hook_case" != "case_id" ] || continue + [ "$authority" = "durable-inbox-only" ] || + fail "$hook_case does not preserve durable-inbox authority" + for provider in "${adapter_ids[@]}"; do + hook_row="$(awk -F '\t' -v id="$provider/hook/$hook_case" '$1 == id { print }' "$provenance")" + [ -n "$hook_row" ] || fail "$provider/hook/$hook_case slot is missing" + IFS=$'\t' read -r hook_asset hook_kind row_case row_product hook_source hook_schema hook_derivation hook_redaction hook_hash hook_owner hook_status hook_outcome <<<"$hook_row" + [ "$hook_asset" = "$provider/hook/$hook_case" ] && + [ "$hook_kind" = "post-turn-hook" ] && + [ "$row_case" = "$hook_case" ] && + [ "$row_product" = "$provider" ] && + [ "$hook_source" = "pending-owner-artifact" ] && + [ "$hook_schema" = "pending" ] && + [ "$hook_derivation" = "none" ] && + [ "$hook_redaction" = "pending" ] && + [ "$hook_hash" = "-" ] && + [ "$hook_owner" = "pending" ] && + [ "$hook_status" = "blocked" ] && + [ "$hook_outcome" = "$expected_outcome" ] || + fail "$provider/hook/$hook_case slot invents or omits provenance" + ((hook_slot_count += 1)) + done +done <"$hook_cases" +[ "$hook_slot_count" -eq $((${#actual_hook_cases[@]} * ${#adapter_ids[@]})) ] || + fail "expected symmetric post-turn hook slots for every provider" + +ab_header="$(head -n 1 "$ab_cases")" +[ "$ab_header" = $'case_id\ta_delivery\tb_delivery\tdurable_expectation\tfifo_expectation\tmodel_calls\tb_pty_bytes\tb_collision_goal' ] || + fail "ab-cases.tsv header changed" +expected_ab_cases=( + active-turn + compaction-clear + crash-restart + dnd + fifo-burst + hook-failure + idle + long-child + stale-unknown +) +mapfile -t actual_ab_cases < <(tail -n +2 "$ab_cases" | cut -f1 | LC_ALL=C sort) +[ "${actual_ab_cases[*]}" = "${expected_ab_cases[*]}" ] || + fail "A/B case IDs differ from the authorized matrix" +while IFS=$'\t' read -r ab_case a_delivery b_delivery durable fifo model_calls b_bytes collision_goal; do + [ "$ab_case" != "case_id" ] || continue + [ -n "$a_delivery" ] && [ -n "$b_delivery" ] || + fail "$ab_case omits an A or B delivery expectation" + [ "$durable" = "eventual-exactly-once" ] && + [ "$fifo" = "preserve" ] && + [ "$model_calls" = "0" ] && + [ "$collision_goal" = "zero" ] || + fail "$ab_case biases or weakens the shared durability/cost/collision contract" +done <"$ab_cases" +active_b="$(awk -F '\t' '$1 == "active-turn" { print $3 "\t" $7 }' "$ab_cases")" +[ "$active_b" = $'next-hook-context\tzero' ] || + fail "active-turn B must write zero PTY bytes and deliver through the next hook context" +idle_b="$(awk -F '\t' '$1 == "idle" { print $3 "\t" $7 }' "$ab_cases")" +[ "$idle_b" = $'bounded-idle-wake\texactly-one' ] || + fail "idle B must permit exactly one bounded wake" + +input_header="$(head -n 1 "$input_race_cases")" +[ "$input_header" = $'case_id\tadapter_precondition\tconcurrent_change\texpected_ding_bytes\texpected_candidate_result\tauthority_dependency' ] || + fail "input-race-cases.tsv header changed" +expected_input_cases=( + attached-viewer-no-io + escape-race + newline-race + output-race + partial-command + paste-race + quiet-no-viewer + recent-key + restart-race + supervisor-input-race +) +mapfile -t actual_input_cases < <(tail -n +2 "$input_race_cases" | cut -f1 | LC_ALL=C sort) +[ "${actual_input_cases[*]}" = "${expected_input_cases[*]}" ] || + fail "conditional-send cases differ from the authorized generic race matrix" +while IFS=$'\t' read -r input_case adapter_precondition concurrent_change expected_bytes candidate_result dependency; do + [ "$input_case" != "case_id" ] || continue + [ -n "$adapter_precondition" ] && [ -n "$concurrent_change" ] && [ -n "$dependency" ] || + fail "$input_case omits generic conditional-send evidence" + case "$candidate_result" in + accept) + [ "$concurrent_change" = "none" ] && [ "$expected_bytes" = "exactly-one" ] || + fail "$input_case accepts after an I/O race" + ;; + reject|reject-before-candidate|reject-ding-candidate) + [ "$expected_bytes" = "zero" ] || + fail "$input_case rejection still permits DING bytes" + ;; + *) fail "$input_case has unsupported candidate outcome $candidate_result" ;; + esac +done <"$input_race_cases" + +[ ! -e "$fixture_root/provider-payloads" ] || + fail "provider payload exists before reviewed fixture delivery" + +blocker_header="$(head -n 1 "$blockers")" +[ "$blocker_header" = $'blocker_id\towner\tdependency\trequired_evidence\tstatus' ] || + fail "blockers.tsv header changed" +blocker_count=0 +present_draft_count=0 +partial_draft_count=0 +unresolved_count=0 +while IFS=$'\t' read -r blocker_id owner dependency evidence status; do + [ "$blocker_id" != "blocker_id" ] || continue + [ -n "$blocker_id" ] && [ -n "$owner" ] && [ -n "$dependency" ] && [ -n "$evidence" ] || + fail "incomplete blocker row" + case "$status" in + present-draft) ((present_draft_count += 1)) ;; + partial-draft) ((partial_draft_count += 1)) ;; + pending|blocked) ((unresolved_count += 1)) ;; + *) fail "$blocker_id has unsupported blocker status $status" ;; + esac + ((blocker_count += 1)) +done <"$blockers" +[ "$blocker_count" -eq 7 ] || fail "expected seven explicit product/fixture blockers" +[ "$present_draft_count" -eq 3 ] || + fail "expected exact draft activity, guarded-send, and configured st2 surfaces" +[ "$partial_draft_count" -eq 1 ] || + fail "expected one explicitly partial PTY diagnostic boundary" +[ "$unresolved_count" -eq 3 ] || + fail "expected three unresolved provider/live seams" + +echo "DIAGNOSTIC-FIXTURES-GREEN-57d1" +echo "PROVIDER-SLOTS-GREEN-57d1" +echo "AB-INVENTORY-GREEN-57d1" +echo "CONDITIONAL-SEND-INVENTORY-GREEN-57d1" +echo "PROVENANCE-INVENTORY-GREEN-57d1" +echo "PRODUCT-BLOCKERS-GREEN-57d1" +echo "PTY-DRAFT-SEAMS-GREEN-57d1" +echo "HERMETIC-SKELETON-GREEN-57d1" diff --git a/cells/ding-adapter-fixture-contract/fixture/hook-cases.tsv b/cells/ding-adapter-fixture-contract/fixture/hook-cases.tsv new file mode 100644 index 00000000..944ed686 --- /dev/null +++ b/cells/ding-adapter-fixture-contract/fixture/hook-cases.tsv @@ -0,0 +1,5 @@ +case_id expected_outcome authority +post-turn-unread next-context-injection durable-inbox-only +post-turn-multiple next-context-fifo durable-inbox-only +post-turn-empty no-injection durable-inbox-only +hook-failure hold-unread durable-inbox-only diff --git a/cells/ding-adapter-fixture-contract/fixture/input-race-cases.tsv b/cells/ding-adapter-fixture-contract/fixture/input-race-cases.tsv new file mode 100644 index 00000000..f79ac77b --- /dev/null +++ b/cells/ding-adapter-fixture-contract/fixture/input-race-cases.tsv @@ -0,0 +1,11 @@ +case_id adapter_precondition concurrent_change expected_ding_bytes expected_candidate_result authority_dependency +quiet-no-viewer fresh-idle-empty-composer none exactly-one accept pending-generic-conditional-send +attached-viewer-no-io fresh-idle-empty-composer none exactly-one accept pending-generic-conditional-send +partial-command fresh-idle-nonempty-composer none zero reject-before-candidate pending-adapter-composer-evidence +recent-key fresh-idle-empty-composer key-input zero reject pending-generic-conditional-send +newline-race fresh-idle-empty-composer newline-input zero reject pending-generic-conditional-send +paste-race fresh-idle-empty-composer paste-input zero reject pending-generic-conditional-send +escape-race fresh-idle-empty-composer escape-input zero reject pending-generic-conditional-send +output-race fresh-idle-empty-composer pty-output zero reject pending-generic-conditional-send +restart-race fresh-idle-empty-composer generation-change zero reject pending-generic-conditional-send +supervisor-input-race fresh-idle-empty-composer supervisor-input zero reject-ding-candidate pending-generic-input-origin diff --git a/cells/ding-adapter-fixture-contract/fixture/provenance.tsv b/cells/ding-adapter-fixture-contract/fixture/provenance.tsv new file mode 100644 index 00000000..974bb135 --- /dev/null +++ b/cells/ding-adapter-fixture-contract/fixture/provenance.tsv @@ -0,0 +1,36 @@ +asset_id kind case_id product source schema_version derivation redaction_review sha256 owner status expected_result +screen/active-turn ascii-screen active-turn generic evals#57 diagnostic-ascii-v1 synthetic not-required-synthetic d26061bbe4e95439923f2c8f9dd51a18b3f6fe0197efe7506187648f2a28649d evals ready active +screen/idle-prompt ascii-screen idle-prompt generic evals#57 diagnostic-ascii-v1 synthetic not-required-synthetic 6c177fb679d7aa84ecb0df76ec4fb084265a900bf8591a8227a9cd8040b60590 evals ready idle +screen/long-child ascii-screen long-child generic evals#57 diagnostic-ascii-v1 synthetic not-required-synthetic afdaa19fc18fadf821b752ed9d906b0f7d9a96fb69697f702070d68a4a2cad2f evals ready child_command +screen/alternate-screen ascii-screen alternate-screen generic evals#57 diagnostic-ascii-v1 synthetic not-required-synthetic d3a12cb62d70f5bf54e1bde9ad857c195e1b570a31dc7dc66414c65ad567b7a1 evals ready unknown +screen/raw-mode ascii-screen raw-mode generic evals#57 diagnostic-ascii-v1 synthetic not-required-synthetic 502f627f6c6cc8f2287f4d6e51ce0e60f257da398a4ac5b22440e40e7c43b75a evals ready unknown +screen/restore ascii-screen restore generic evals#57 diagnostic-ascii-v1 synthetic not-required-synthetic e520b13e37f5a3b03b5995a36d49e03ee9e52318f9b8df7d6dc803f0692f447a evals ready idle +screen/compaction ascii-screen compaction generic evals#57 diagnostic-ascii-v1 synthetic not-required-synthetic f4bf99a5e7d32e53b42d9a13cf785a3cd2c6399055fa1cb836d2d45101c01a91 evals ready unknown +screen/clear ascii-screen clear generic evals#57 diagnostic-ascii-v1 synthetic not-required-synthetic caf48276ce1523bd8b9fba2c5446d9eebec1ef2aadbfc50ec4b252209eb9f5d6 evals ready unknown +screen/crash ascii-screen crash generic evals#57 diagnostic-ascii-v1 synthetic not-required-synthetic 983ee3465b93323279717a6c2419f16582a2369b556b18eef9d2c66a323060f8 evals ready unknown +codex/active-turn adapter-event active-turn codex pending-owner-artifact pending none pending - pending blocked active +codex/idle-prompt adapter-event idle-prompt codex pending-owner-artifact pending none pending - pending blocked idle +codex/long-child adapter-event long-child codex pending-owner-artifact pending none pending - pending blocked child_command +codex/alternate-screen adapter-event alternate-screen codex pending-owner-artifact pending none pending - pending blocked unknown +codex/raw-mode adapter-event raw-mode codex pending-owner-artifact pending none pending - pending blocked unknown +codex/restore adapter-event restore codex pending-owner-artifact pending none pending - pending blocked idle +codex/compaction adapter-event compaction codex pending-owner-artifact pending none pending - pending blocked unknown +codex/clear adapter-event clear codex pending-owner-artifact pending none pending - pending blocked unknown +codex/crash adapter-event crash codex pending-owner-artifact pending none pending - pending blocked unknown +claude/active-turn adapter-event active-turn claude pending-owner-artifact pending none pending - pending blocked active +claude/idle-prompt adapter-event idle-prompt claude pending-owner-artifact pending none pending - pending blocked idle +claude/long-child adapter-event long-child claude pending-owner-artifact pending none pending - pending blocked child_command +claude/alternate-screen adapter-event alternate-screen claude pending-owner-artifact pending none pending - pending blocked unknown +claude/raw-mode adapter-event raw-mode claude pending-owner-artifact pending none pending - pending blocked unknown +claude/restore adapter-event restore claude pending-owner-artifact pending none pending - pending blocked idle +claude/compaction adapter-event compaction claude pending-owner-artifact pending none pending - pending blocked unknown +claude/clear adapter-event clear claude pending-owner-artifact pending none pending - pending blocked unknown +claude/crash adapter-event crash claude pending-owner-artifact pending none pending - pending blocked unknown +codex/hook/post-turn-unread post-turn-hook post-turn-unread codex pending-owner-artifact pending none pending - pending blocked next-context-injection +codex/hook/post-turn-multiple post-turn-hook post-turn-multiple codex pending-owner-artifact pending none pending - pending blocked next-context-fifo +codex/hook/post-turn-empty post-turn-hook post-turn-empty codex pending-owner-artifact pending none pending - pending blocked no-injection +codex/hook/hook-failure post-turn-hook hook-failure codex pending-owner-artifact pending none pending - pending blocked hold-unread +claude/hook/post-turn-unread post-turn-hook post-turn-unread claude pending-owner-artifact pending none pending - pending blocked next-context-injection +claude/hook/post-turn-multiple post-turn-hook post-turn-multiple claude pending-owner-artifact pending none pending - pending blocked next-context-fifo +claude/hook/post-turn-empty post-turn-hook post-turn-empty claude pending-owner-artifact pending none pending - pending blocked no-injection +claude/hook/hook-failure post-turn-hook hook-failure claude pending-owner-artifact pending none pending - pending blocked hold-unread diff --git a/cells/ding-adapter-fixture-contract/fixture/providers.tsv b/cells/ding-adapter-fixture-contract/fixture/providers.tsv new file mode 100644 index 00000000..1db9cb2f --- /dev/null +++ b/cells/ding-adapter-fixture-contract/fixture/providers.tsv @@ -0,0 +1,3 @@ +adapter_id fixture_owner activity_status hook_status +codex codex-adapter-owner blocked blocked +claude claude-adapter-owner blocked blocked diff --git a/cells/ding-adapter-fixture-contract/fixture/screens/active-turn.txt b/cells/ding-adapter-fixture-contract/fixture/screens/active-turn.txt new file mode 100644 index 00000000..24f5e19d --- /dev/null +++ b/cells/ding-adapter-fixture-contract/fixture/screens/active-turn.txt @@ -0,0 +1,3 @@ +ACTIVITY DIAGNOSTIC: active turn +work is in progress +input readiness is not derived from this screen diff --git a/cells/ding-adapter-fixture-contract/fixture/screens/alternate-screen.txt b/cells/ding-adapter-fixture-contract/fixture/screens/alternate-screen.txt new file mode 100644 index 00000000..e4906016 --- /dev/null +++ b/cells/ding-adapter-fixture-contract/fixture/screens/alternate-screen.txt @@ -0,0 +1,3 @@ +TERMINAL DIAGNOSTIC: alternate screen +application buffer remains active +screen mode is not idle authority diff --git a/cells/ding-adapter-fixture-contract/fixture/screens/clear.txt b/cells/ding-adapter-fixture-contract/fixture/screens/clear.txt new file mode 100644 index 00000000..d2061d08 --- /dev/null +++ b/cells/ding-adapter-fixture-contract/fixture/screens/clear.txt @@ -0,0 +1,3 @@ +ACTIVITY DIAGNOSTIC: clear +the visible screen has been reset +an empty screen is not idle authority diff --git a/cells/ding-adapter-fixture-contract/fixture/screens/compaction.txt b/cells/ding-adapter-fixture-contract/fixture/screens/compaction.txt new file mode 100644 index 00000000..4c998ee2 --- /dev/null +++ b/cells/ding-adapter-fixture-contract/fixture/screens/compaction.txt @@ -0,0 +1,3 @@ +ACTIVITY DIAGNOSTIC: compaction +session history is being rewritten +no prompt readiness is inferred diff --git a/cells/ding-adapter-fixture-contract/fixture/screens/crash.txt b/cells/ding-adapter-fixture-contract/fixture/screens/crash.txt new file mode 100644 index 00000000..57139cb6 --- /dev/null +++ b/cells/ding-adapter-fixture-contract/fixture/screens/crash.txt @@ -0,0 +1,3 @@ +ACTIVITY DIAGNOSTIC: crash +the previous process incarnation ended +old activity cannot authorize new input diff --git a/cells/ding-adapter-fixture-contract/fixture/screens/idle-prompt.txt b/cells/ding-adapter-fixture-contract/fixture/screens/idle-prompt.txt new file mode 100644 index 00000000..95da67c1 --- /dev/null +++ b/cells/ding-adapter-fixture-contract/fixture/screens/idle-prompt.txt @@ -0,0 +1,3 @@ +ACTIVITY DIAGNOSTIC: idle prompt +explicit prompt-ready event is required +> diff --git a/cells/ding-adapter-fixture-contract/fixture/screens/long-child.txt b/cells/ding-adapter-fixture-contract/fixture/screens/long-child.txt new file mode 100644 index 00000000..c8fd6d2b --- /dev/null +++ b/cells/ding-adapter-fixture-contract/fixture/screens/long-child.txt @@ -0,0 +1,3 @@ +ACTIVITY DIAGNOSTIC: long-running child command +the screen may remain unchanged +silence is not idle authority diff --git a/cells/ding-adapter-fixture-contract/fixture/screens/raw-mode.txt b/cells/ding-adapter-fixture-contract/fixture/screens/raw-mode.txt new file mode 100644 index 00000000..6ca54eb9 --- /dev/null +++ b/cells/ding-adapter-fixture-contract/fixture/screens/raw-mode.txt @@ -0,0 +1,3 @@ +TERMINAL DIAGNOSTIC: raw input mode +application input remains noncanonical +input mode is not idle authority diff --git a/cells/ding-adapter-fixture-contract/fixture/screens/restore.txt b/cells/ding-adapter-fixture-contract/fixture/screens/restore.txt new file mode 100644 index 00000000..ea6489e5 --- /dev/null +++ b/cells/ding-adapter-fixture-contract/fixture/screens/restore.txt @@ -0,0 +1,3 @@ +TERMINAL DIAGNOSTIC: restored terminal +primary buffer and canonical input are requested +restoration is not idle authority diff --git a/cells/ding-delivery-a-aggressive-control/README.md b/cells/ding-delivery-a-aggressive-control/README.md new file mode 100644 index 00000000..7c099b2e --- /dev/null +++ b/cells/ding-delivery-a-aggressive-control/README.md @@ -0,0 +1,16 @@ +# ding-delivery-a-aggressive-control + +Current-product, model-free control for unconfigured DING on st2 +`c6846f6239329f0803142afc06c15a07b93937c1` with PTY activity base +`46c71d31c0d6daee43adf568061b2b84a65ae8c0` and stacked guarded-send head +`743ceb796a41a3282e31382575bff0d0e3826d59`. + +The target has no activity publisher, so live activity is exactly `unknown`. +With no rich adapter configuration, st2 still invokes ordinary `pty send`; the +synthetic raw composer records those bytes. Because the synthetic pane does not +emit a provider-specific positive receipt, st2 correctly retains staged +ownership and leaves the durable inbox message unread. + +This isolates the aggressive baseline fact needed by issue #57 without +pretending the older multi-message `presence-ding-matrix` is compatible with +the positive-receipt semantics added by st2 `c6846f6`. diff --git a/cells/ding-delivery-a-aggressive-control/ding-delivery-a-aggressive-control.kdl b/cells/ding-delivery-a-aggressive-control/ding-delivery-a-aggressive-control.kdl new file mode 100644 index 00000000..4a7ea20f --- /dev/null +++ b/cells/ding-delivery-a-aggressive-control/ding-delivery-a-aggressive-control.kdl @@ -0,0 +1,33 @@ +// Model-free current-product control: unconfigured DING sends while PTY activity is unknown. +eval { + copy "./fixture" + max-timeout "45s" + + run "aggressive-control" { + command "bash ./aggressive-control.sh" + } + + judges { + judge "EXACT SOURCE — current st2 and stacked PTY heads are exact" { + exec "grep -Fqx AGGRESSIVE-EXACT-HEADS-GREEN-57c1 $RUNS_DIR/aggressive-control.out" + } + judge "UNCONFIGURED — no rich adapter participates in the DING sidecar" { + exec "grep -Fqx AGGRESSIVE-UNCONFIGURED-GREEN-57c1 $RUNS_DIR/aggressive-control.out" + } + judge "UNKNOWN — PTY has no activity publisher and reports exact unknown" { + exec "grep -Fqx AGGRESSIVE-UNKNOWN-ACTIVITY-GREEN-57c1 $RUNS_DIR/aggressive-control.out" + } + judge "ORDINARY INPUT — st2 invokes the ordinary PTY send path despite unknown activity" { + exec "grep -Fqx AGGRESSIVE-PTY-SEND-GREEN-57c1 $RUNS_DIR/aggressive-control.out" + } + judge "PANE BYTES — the raw synthetic composer receives the DING notice" { + exec "grep -Fqx AGGRESSIVE-PANE-BYTES-GREEN-57c1 $RUNS_DIR/aggressive-control.out" + } + judge "DURABLE OWNERSHIP — without a positive receipt the message remains unread" { + exec "grep -Fqx AGGRESSIVE-DURABLE-UNREAD-GREEN-57c1 $RUNS_DIR/aggressive-control.out" + } + judge "CLEANUP/MODEL FREE — sidecar and PTY are gone with zero model calls" { + exec "grep -Fqx AGGRESSIVE-CLEANUP-MODEL-FREE-GREEN-57c1 $RUNS_DIR/aggressive-control.out" + } + } +} diff --git a/cells/ding-delivery-a-aggressive-control/fixture/aggressive-control.sh b/cells/ding-delivery-a-aggressive-control/fixture/aggressive-control.sh new file mode 100644 index 00000000..462bacc1 --- /dev/null +++ b/cells/ding-delivery-a-aggressive-control/fixture/aggressive-control.sh @@ -0,0 +1,109 @@ +#!/usr/bin/env bash +set -euo pipefail + +root="${CATALOG:?CATALOG must be set}" +package_root="${EVALS_PTY_PR133_ROOT:?EVALS_PTY_PR133_ROOT must point to exact PTY head}" +expected_st2="c6846f6239329f0803142afc06c15a07b93937c1" +expected_st2_sha256="2bba8d58be24250bc262f75f835ce2d780369add275774f3f2135c623d23d29c" +expected_activity="46c71d31c0d6daee43adf568061b2b84a65ae8c0" +expected_guarded="743ceb796a41a3282e31382575bff0d0e3826d59" +net="$root/net" +pty_root="$net/pty" +fake_bin="$root/fake-bin" +send_log="$root/pty-send.log" +deliveries="$net/deliveries.log" +ding_log="$root/ding.log" +real_pty="$(command -v pty)" +ding_pid="" + +cleanup() { + if test -n "$ding_pid"; then + kill "$ding_pid" 2>/dev/null || true + wait "$ding_pid" 2>/dev/null || true + fi + PTY_ROOT="$pty_root" "$real_pty" kill control.target >/dev/null 2>&1 || true + for _ in $(seq 1 80); do + if PTY_ROOT="$pty_root" "$real_pty" rm control.target >/dev/null 2>&1; then + break + fi + if ! PTY_ROOT="$pty_root" "$real_pty" list --json | + jq -e '.[] | select(.name == "control.target")' >/dev/null; then + break + fi + sleep 0.025 + done +} +trap cleanup EXIT + +wait_for_text() { + marker="$1" + file="$2" + for _ in $(seq 1 400); do + if grep -Fq "$marker" "$file" 2>/dev/null; then + return 0 + fi + sleep 0.025 + done + printf 'text %s did not reach %s\n' "$marker" "$file" >&2 + return 1 +} + +test "$(git -C "$package_root" rev-parse HEAD)" = "$expected_guarded" +git -C "$package_root" merge-base --is-ancestor "$expected_activity" "$expected_guarded" +test -z "$(git -C "$package_root" status --porcelain)" +test "$(readlink -f "$real_pty")" = "$(readlink -f "$package_root/bin/pty")" +case "$(st2 --version)" in + "st2 0.1.0 — running from local source (${expected_st2:0:7}, "*")") ;; + *) echo "unexpected st2 version" >&2; exit 1 ;; +esac +test "$(sha256sum "$(command -v st2)" | awk '{ print $1 }')" = "$expected_st2_sha256" + +mkdir -p "$fake_bin" +cp "$root/pty-wrapper" "$fake_bin/pty" +chmod +x "$fake_bin/pty" +: >"$deliveries" +: >"$send_log" + +PTY_ROOT="$pty_root" "$real_pty" run -d --id control.target --no-display-name \ + --env CATALOG="$net" -- perl "$net/fake-tui.pl" +for _ in $(seq 1 200); do + if PTY_ROOT="$pty_root" "$real_pty" stats control.target --json >"$root/stats.json" 2>/dev/null; then + break + fi + sleep 0.025 +done +jq -e \ + '.activity.state == "unknown" and + .activity.producerEpoch == null and + .activity.sequence == 0 and + .activity.generation == .generation' \ + "$root/stats.json" >/dev/null + +st2 status control.target --set busy --catalog "$net" --host control --as control.target >/dev/null +CATALOG="$net" REAL_PTY="$real_pty" PTY_ROOT="$pty_root" PTY_SEND_LOG="$send_log" \ + PATH="$fake_bin:$PATH" \ + st2 ping control.target --identity control.target --catalog "$net" --host control --interval 50 \ + >"$ding_log" 2>&1 & +ding_pid="$!" +wait_for_text 'ready — found 0 existing unread message(s)' "$ding_log" + +st2 message send control.target --catalog "$net" --host control --as control.sender \ + --subject aggressive-unknown-57 -m "durable aggressive control" >/dev/null +wait_for_text $'send\tsend\tcontrol.target' "$send_log" +wait_for_text aggressive-unknown-57 "$deliveries" +test "$(st2 message ls control.target --catalog "$net" --count)" -eq 1 + +kill "$ding_pid" +wait "$ding_pid" 2>/dev/null || true +ding_pid="" +cleanup +trap - EXIT +test "$(PTY_ROOT="$pty_root" "$real_pty" list --json | jq 'length')" -eq 0 + +echo "AGGRESSIVE-EXACT-HEADS-GREEN-57c1" +echo "AGGRESSIVE-UNCONFIGURED-GREEN-57c1" +echo "AGGRESSIVE-UNKNOWN-ACTIVITY-GREEN-57c1" +echo "AGGRESSIVE-PTY-SEND-GREEN-57c1" +echo "AGGRESSIVE-PANE-BYTES-GREEN-57c1" +echo "AGGRESSIVE-DURABLE-UNREAD-GREEN-57c1" +echo "AGGRESSIVE-CLEANUP-MODEL-FREE-GREEN-57c1" diff --git a/cells/ding-delivery-a-aggressive-control/fixture/net/agents/control/sender/agent.kdl b/cells/ding-delivery-a-aggressive-control/fixture/net/agents/control/sender/agent.kdl new file mode 100644 index 00000000..b4c7b83c --- /dev/null +++ b/cells/ding-delivery-a-aggressive-control/fixture/net/agents/control/sender/agent.kdl @@ -0,0 +1,7 @@ +agent "sender" { + identity "sender" + host "control" + role "worker" + retired #true + command "exec sleep 100000" +} diff --git a/cells/ding-delivery-a-aggressive-control/fixture/net/agents/control/target/agent.kdl b/cells/ding-delivery-a-aggressive-control/fixture/net/agents/control/target/agent.kdl new file mode 100644 index 00000000..b95c5b72 --- /dev/null +++ b/cells/ding-delivery-a-aggressive-control/fixture/net/agents/control/target/agent.kdl @@ -0,0 +1,7 @@ +agent "target" { + identity "target" + host "control" + role "worker" + retired #true + command "exec sleep 100000" +} diff --git a/cells/ding-delivery-a-aggressive-control/fixture/net/fake-tui.pl b/cells/ding-delivery-a-aggressive-control/fixture/net/fake-tui.pl new file mode 100644 index 00000000..70e60f38 --- /dev/null +++ b/cells/ding-delivery-a-aggressive-control/fixture/net/fake-tui.pl @@ -0,0 +1,27 @@ +#!/usr/bin/env perl +use strict; +use warnings; + +$| = 1; +system("stty raw -echo"); +open(my $log, ">>", "$ENV{CATALOG}/deliveries.log") or die "open deliveries: $!"; +select((select($log), $| = 1)[0]); +print "\e[2J\e[Hsynthetic aggressive composer ready\r\n"; + +my $buffer = ""; +my $payload; +while (1) { + my $chunk = ""; + my $read = sysread(STDIN, $chunk, 4096); + last if !defined($read) || $read == 0; + $buffer .= $chunk; + + if (!defined($payload) && $buffer =~ s/.*?\e\[200~(.*?)\e\[201~//s) { + $payload = $1; + } + if (defined($payload) && $buffer =~ s/^[^\r\n]*[\r\n]//s) { + $payload =~ s/[\r\n]+/ /g; + print {$log} "$payload\n"; + undef($payload); + } +} diff --git a/cells/ding-delivery-a-aggressive-control/fixture/pty-wrapper b/cells/ding-delivery-a-aggressive-control/fixture/pty-wrapper new file mode 100644 index 00000000..acef74cd --- /dev/null +++ b/cells/ding-delivery-a-aggressive-control/fixture/pty-wrapper @@ -0,0 +1,11 @@ +#!/usr/bin/env bash +set -euo pipefail + +if [ "${1:-}" = "send" ]; then + { + printf 'send' + printf '\t%q' "$@" + printf '\n' + } >>"${PTY_SEND_LOG:?PTY_SEND_LOG must be set}" +fi +exec "${REAL_PTY:?REAL_PTY must be set}" "$@" diff --git a/cells/ding-delivery-pty-ab/README.md b/cells/ding-delivery-pty-ab/README.md new file mode 100644 index 00000000..558e23d6 --- /dev/null +++ b/cells/ding-delivery-pty-ab/README.md @@ -0,0 +1,32 @@ +# ding-delivery-pty-ab + +Matched, model-free PTY-boundary A/B for evals issue #57. + +- **A** uses ordinary unguarded PTY input, matching the current aggressive + unconfigured DING behavior. +- **B** is an external reference adapter: it attempts PTY input only when the + exact live activity state is `idle`, its fixture-owned composer fact is + empty, and `compareAndSend()` accepts the same generation and I/O revision. + Otherwise the durable message remains queued for the deterministic post-turn + hook. + +Both arms receive identical durable bodies and preserve SHA256, exactly-once +delivery, FIFO, DND, hook retry, and restart recovery with zero model calls. +The active-turn case places the same partial human draft in both PTYs: A +physically appends its DING marker to that input stream, while B writes zero +DING bytes and the later hook delivers the message. This is the cell's one +directly observed collision. The other five non-idle, stale, or unknown A +writes are classified only as unsafe writes; they are not counted as observed +collisions. + +This cell runs against PTY activity base +`46c71d31c0d6daee43adf568061b2b84a65ae8c0` and stacked guarded-send successor +`743ceb796a41a3282e31382575bff0d0e3826d59`. Set +`EVALS_PTY_PR133_ROOT` to a clean exact-head checkout after +`npm ci && npm run build`. + +The result proves the PTY-boundary strategy improves over aggressive input. +Configured generic-adapter selection, hook ownership, DND precedence, typed +receipts, and the same matrix through real generated task environments are now +covered separately by `ding-delivery-st2-rich-ab` against corrected st2 PR +#123. This cell remains the smaller PTY-boundary reference oracle. diff --git a/cells/ding-delivery-pty-ab/ding-delivery-pty-ab.kdl b/cells/ding-delivery-pty-ab/ding-delivery-pty-ab.kdl new file mode 100644 index 00000000..b9d87db1 --- /dev/null +++ b/cells/ding-delivery-pty-ab/ding-delivery-pty-ab.kdl @@ -0,0 +1,36 @@ +// Matched model-free A/B at the exact PTY boundary; no st2 configured-adapter claim. +eval { + copy "./fixture" + max-timeout "120s" + + run "delivery-ab" { + command "node ./delivery-ab.mjs" + } + + judges { + judge "EXACT SOURCE — activity and guarded-send heads are exact and clean" { + exec "grep -Fqx DING-AB-EXACT-PTY-HEADS-GREEN-57b1 $RUNS_DIR/delivery-ab.out" + } + judge "DURABILITY — A and B deliver identical bodies/hashes exactly once in the same order" { + exec "grep -Fqx DING-AB-DURABILITY-PARITY-GREEN-57b1 $RUNS_DIR/delivery-ab.out" + } + judge "ACTIVE TURN — one observed draft collision in A while explicit-idle B writes zero bytes" { + exec "grep -Fqx DING-AB-ACTIVE-TURN-IMPROVEMENT-GREEN-57b1 $RUNS_DIR/delivery-ab.out && jq -e '.summary.armAObservedPartialDraftCollisions == 1 and .summary.armBObservedPartialDraftCollisions == 0 and ([.cases[] | select(.caseId == \"active-turn\") | .observedPartialDraftCollision] == [{\"armA\":true,\"armB\":false,\"observation\":\"DING PTY marker arrived after the live partial draft\"}])' $CATALOG/delivery-ab-receipt.json >/dev/null" + } + judge "IDLE — B performs exactly one bounded guarded wake only for exact idle plus empty composer" { + exec "grep -Fqx DING-AB-IDLE-BOUNDED-WAKE-GREEN-57b1 $RUNS_DIR/delivery-ab.out" + } + judge "HOLDS — child, unknown, DND, hook failure, clear/compaction, and restart remain durable" { + exec "grep -Fqx DING-AB-HOLD-RECOVERY-GREEN-57b1 $RUNS_DIR/delivery-ab.out" + } + judge "FIFO — burst delivery order and ownership are identical with no duplicates" { + exec "grep -Fqx DING-AB-FIFO-GREEN-57b1 $RUNS_DIR/delivery-ab.out" + } + judge "IMPROVEMENT — B preserves delivery while reducing unsafe PTY writes from six to zero" { + exec "grep -Fqx DING-AB-IMPROVEMENT-GREEN-57b1 $RUNS_DIR/delivery-ab.out && jq -e '.summary.deliveryParity and .summary.armAUnsafeWrites == 6 and .summary.armBUnsafeWrites == 0' $CATALOG/delivery-ab-receipt.json >/dev/null" + } + judge "CLEANUP/MODEL FREE — both arms use zero models and leave no PTY sessions" { + exec "grep -Fqx DING-AB-CLEANUP-MODEL-FREE-GREEN-57b1 $RUNS_DIR/delivery-ab.out && jq -e '.modelCalls == 0 and .cleanup.sessions == 0' $CATALOG/delivery-ab-receipt.json >/dev/null" + } + } +} diff --git a/cells/ding-delivery-pty-ab/fixture/ab-child.mjs b/cells/ding-delivery-pty-ab/fixture/ab-child.mjs new file mode 100644 index 00000000..09438f8a --- /dev/null +++ b/cells/ding-delivery-pty-ab/fixture/ab-child.mjs @@ -0,0 +1,18 @@ +import { appendFileSync } from "node:fs"; + +const receivedPath = process.argv[2]; +if (!receivedPath) throw new Error("received-byte path is required"); + +process.stdin.setRawMode?.(true); +process.stdin.resume(); +process.stdin.on("data", (chunk) => appendFileSync(receivedPath, chunk)); +process.stdout.write("DING-AB-READY\r\n"); + +const timer = setInterval(() => {}, 1000); +function stop() { + clearInterval(timer); + process.exit(0); +} + +process.on("SIGTERM", stop); +process.on("SIGINT", stop); diff --git a/cells/ding-delivery-pty-ab/fixture/delivery-ab.mjs b/cells/ding-delivery-pty-ab/fixture/delivery-ab.mjs new file mode 100644 index 00000000..d6ce8732 --- /dev/null +++ b/cells/ding-delivery-pty-ab/fixture/delivery-ab.mjs @@ -0,0 +1,455 @@ +import assert from "node:assert/strict"; +import { execFile } from "node:child_process"; +import { createHash } from "node:crypto"; +import { readFile, writeFile } from "node:fs/promises"; +import path from "node:path"; +import { pathToFileURL } from "node:url"; +import { promisify } from "node:util"; + +const execFileAsync = promisify(execFile); +const ACTIVITY_HEAD = "46c71d31c0d6daee43adf568061b2b84a65ae8c0"; +const GUARDED_HEAD = "743ceb796a41a3282e31382575bff0d0e3826d59"; +const PACKAGE_LOCK_SHA256 = "43189b5b5b1d560be4b9102d2ed0793d89b9b01dbb3bbd976948f10c6669118d"; +const CASE_IDS = [ + "idle", + "active-turn", + "long-child", + "dnd", + "stale-unknown", + "hook-failure", + "crash-restart", + "compaction-clear", + "fifo-burst", +]; + +function requiredEnv(name) { + const value = process.env[name]; + assert.ok(value, `${name} is required`); + return path.resolve(value); +} + +function digest(value) { + return createHash("sha256").update(value).digest("hex"); +} + +async function sha256(file) { + return digest(await readFile(file)); +} + +async function sleep(milliseconds) { + await new Promise((resolve) => setTimeout(resolve, milliseconds)); +} + +async function waitFor(label, operation, predicate, attempts = 160) { + let lastError; + for (let attempt = 0; attempt < attempts; attempt += 1) { + try { + const value = await operation(); + if (predicate(value)) return value; + } catch (error) { + lastError = error; + } + await sleep(25); + } + throw new Error(`timed out waiting for ${label}${lastError ? `: ${lastError.message}` : ""}`); +} + +const packageRoot = requiredEnv("EVALS_PTY_PR133_ROOT"); +const catalog = requiredEnv("CATALOG"); +const ptyRoot = path.join(catalog, "pty-pr133-ab"); +const cli = path.join(packageRoot, "bin", "pty"); +const child = path.join(catalog, "ab-child.mjs"); +process.env.PTY_ROOT = ptyRoot; + +const gitHead = (await execFileAsync("git", ["-C", packageRoot, "rev-parse", "HEAD"])).stdout.trim(); +assert.equal(gitHead, GUARDED_HEAD); +await execFileAsync("git", ["-C", packageRoot, "merge-base", "--is-ancestor", ACTIVITY_HEAD, GUARDED_HEAD]); +assert.equal( + (await execFileAsync("git", ["-C", packageRoot, "status", "--porcelain"])).stdout, + "", +); +assert.equal(await sha256(path.join(packageRoot, "package-lock.json")), PACKAGE_LOCK_SHA256); + +const client = await import(pathToFileURL(path.join(packageRoot, "dist", "client-api.js"))); +const { + compareAndSend, + connectActivityPublisher, + queryStats, + sendData, +} = client; + +const cliEnv = { ...process.env, PTY_ROOT: ptyRoot }; +async function runCli(args) { + return execFileAsync(cli, args, { env: cliEnv, maxBuffer: 1024 * 1024 }); +} + +async function listSessions() { + const { stdout } = await runCli(["list", "--json"]); + return JSON.parse(stdout); +} + +async function removeSession(name) { + for (let attempt = 0; attempt < 120; attempt += 1) { + const session = (await listSessions()).find((entry) => entry.name === name); + if (!session) return; + if (session.status === "running") { + await runCli(["kill", name]).catch(() => {}); + } else { + await runCli(["rm", name]).catch(() => {}); + } + await sleep(25); + } + throw new Error(`session ${name} did not clean up`); +} + +async function readBytes(file) { + try { + return await readFile(file); + } catch (error) { + if (error.code === "ENOENT") return Buffer.alloc(0); + throw error; + } +} + +async function waitForMarker(file, marker) { + const bytes = Buffer.from(marker); + return waitFor( + `${path.basename(file)} marker ${marker}`, + () => readBytes(file), + (buffer) => buffer.includes(bytes), + ); +} + +function durableMessage(caseId, ordinal = 0) { + const body = `case=${caseId}\nordinal=${ordinal}\npayload=durable-ding-57\n`; + return { + id: `${caseId}-${ordinal}`, + body, + sha256: digest(body), + }; +} + +function createArm(id) { + return { + id, + name: `ding-ab-${id}`, + receivedPath: path.join(catalog, `received-${id}.bin`), + publisher: null, + publisherOrdinal: 0, + queue: [], + archive: [], + ptyBytes: 0, + unsafeWrites: 0, + wakeAttempts: 0, + }; +} + +const armA = createArm("a"); +const armB = createArm("b"); +const arms = [armA, armB]; + +async function spawnArm(arm) { + await runCli([ + "run", "-d", "--id", arm.name, "--", + process.execPath, child, arm.receivedPath, + ]); + return waitFor( + `${arm.name} startup`, + () => queryStats(arm.name), + (stats) => stats.process.alive, + ); +} + +async function restartArm(arm) { + arm.publisher?.close(); + arm.publisher = null; + await removeSession(arm.name); + return spawnArm(arm); +} + +async function ensurePublisher(arm) { + if (arm.publisher !== null) return arm.publisher; + arm.publisherOrdinal += 1; + arm.publisher = await connectActivityPublisher(arm.name, { + producerEpoch: `${arm.id}-epoch-${arm.publisherOrdinal}`, + source: "generic-eval", + }); + return arm.publisher; +} + +async function setActivity(arm, state, caseId) { + if (state === "unknown") { + arm.publisher?.close(); + arm.publisher = null; + await waitFor( + `${arm.name} unknown`, + () => queryStats(arm.name), + (stats) => stats.activity.state === "unknown" && stats.activity.producerEpoch === null, + ); + return; + } + const publisher = await ensurePublisher(arm); + await publisher.publish(state, { turnId: caseId }); +} + +function enqueue(arm, messages) { + arm.queue.push(...messages.map((message) => ({ ...message }))); +} + +function runHook(arm, available = true) { + if (!available) return []; + const delivered = arm.queue.splice(0); + arm.archive.push(...delivered); + return delivered; +} + +async function aggressiveWake(arm, marker, unsafe) { + arm.wakeAttempts += 1; + const before = (await readBytes(arm.receivedPath)).length; + await sendData({ name: arm.name, data: [marker] }); + const after = await waitForMarker(arm.receivedPath, marker); + arm.ptyBytes += Buffer.byteLength(marker); + assert.ok(after.length >= before + Buffer.byteLength(marker)); + if (unsafe) { + arm.unsafeWrites += 1; + } + return { attempted: true, accepted: true, bytes: Buffer.byteLength(marker) }; +} + +async function guardedWake(arm, marker, { composerEmpty, dnd = false }) { + if (dnd) return { attempted: false, accepted: false, reason: "dnd" }; + const observed = await queryStats(arm.name); + if (observed.activity.state !== "idle") { + return { attempted: false, accepted: false, reason: observed.activity.state }; + } + if (!composerEmpty) { + return { attempted: false, accepted: false, reason: "composer-nonempty" }; + } + arm.wakeAttempts += 1; + const before = (await readBytes(arm.receivedPath)).length; + const response = await compareAndSend(arm.name, { + generation: observed.generation, + ioRevision: observed.ioRevision, + data: marker, + }); + if (!response.ok) { + assert.equal((await readBytes(arm.receivedPath)).length, before); + return { attempted: true, accepted: false, reason: response.error }; + } + const after = await waitForMarker(arm.receivedPath, marker); + arm.ptyBytes += Buffer.byteLength(marker); + assert.ok(after.length >= before + Buffer.byteLength(marker)); + return { attempted: true, accepted: true, bytes: Buffer.byteLength(marker) }; +} + +function compareArchives(caseId, messages, startA, startB) { + const deliveredA = armA.archive.slice(startA); + const deliveredB = armB.archive.slice(startB); + const expected = messages.map(({ id, sha256 }) => ({ id, sha256 })); + assert.deepEqual( + deliveredA.map(({ id, sha256 }) => ({ id, sha256 })), + expected, + `${caseId} arm A delivery`, + ); + assert.deepEqual( + deliveredB.map(({ id, sha256 }) => ({ id, sha256 })), + expected, + `${caseId} arm B delivery`, + ); + assert.equal(new Set(deliveredA.map(({ id }) => id)).size, deliveredA.length); + assert.equal(new Set(deliveredB.map(({ id }) => id)).size, deliveredB.length); +} + +const receipt = { + dependency: { + activityHead: ACTIVITY_HEAD, + guardedHead: GUARDED_HEAD, + packageLockSha256: PACKAGE_LOCK_SHA256, + cliSha256: await sha256(cli), + clientSha256: await sha256(path.join(packageRoot, "dist", "client-api.js")), + evidenceClass: "local-exact-source-build", + }, + cases: [], + summary: {}, + modelCalls: 0, + cleanup: { sessions: null }, + boundary: { + proven: "external-reference-adapter-at-pty-boundary", + missing: "st2-configured-ding-adapter-and-provider-composer-hook-contract", + }, +}; + +try { + await Promise.all(arms.map(spawnArm)); + + for (const caseId of CASE_IDS) { + const count = caseId === "fifo-burst" ? 3 : 1; + const messages = Array.from({ length: count }, (_, ordinal) => durableMessage(caseId, ordinal)); + const startA = armA.archive.length; + const startB = armB.archive.length; + enqueue(armA, messages); + enqueue(armB, messages); + const caseReceipt = { caseId, messages, armA: {}, armB: {} }; + + if (caseId === "crash-restart") { + await Promise.all(arms.map(restartArm)); + caseReceipt.restarted = true; + } + + const activity = ( + caseId === "idle" || caseId === "dnd" || caseId === "fifo-burst" + ) ? "idle" : ( + caseId === "active-turn" || caseId === "hook-failure" + ) ? "active" : caseId === "long-child" ? "child_command" : "unknown"; + await Promise.all(arms.map((arm) => setActivity(arm, activity, caseId))); + + if (caseId === "active-turn") { + const draft = "PARTIAL-HUMAN-DRAFT-57"; + await Promise.all(arms.map(async (arm) => { + await sendData({ name: arm.name, data: [draft] }); + await waitForMarker(arm.receivedPath, draft); + })); + caseReceipt.partialDraft = draft; + const draftBytes = Buffer.from(draft); + for (const arm of arms) { + const beforeDelivery = await readBytes(arm.receivedPath); + assert.ok(beforeDelivery.subarray(-draftBytes.length).equals(draftBytes), + `${arm.id} partial draft was not live immediately before delivery`); + } + caseReceipt.partialDraftLiveBeforeDelivery = { armA: true, armB: true }; + } + + if (caseId === "dnd") { + caseReceipt.armA.held = { attempted: false, reason: "dnd" }; + caseReceipt.armB.held = await guardedWake(armB, "B:dnd:held", { + composerEmpty: true, + dnd: true, + }); + assert.equal(caseReceipt.armB.held.attempted, false); + caseReceipt.armA.wake = await aggressiveWake(armA, "A:dnd:0", false); + caseReceipt.armB.wake = await guardedWake(armB, "B:dnd:0", { + composerEmpty: true, + }); + runHook(armA); + runHook(armB); + } else if (caseId === "fifo-burst") { + caseReceipt.armA.wakes = []; + caseReceipt.armB.wakes = []; + for (let ordinal = 0; ordinal < messages.length; ordinal += 1) { + caseReceipt.armA.wakes.push( + await aggressiveWake(armA, `A:fifo:${ordinal}`, false), + ); + caseReceipt.armB.wakes.push( + await guardedWake(armB, `B:fifo:${ordinal}`, { composerEmpty: true }), + ); + } + runHook(armA); + runHook(armB); + } else { + const unsafe = activity !== "idle" || caseId === "active-turn"; + const markerA = `A:${caseId}:0`; + const markerB = `B:${caseId}:0`; + caseReceipt.armA.wake = await aggressiveWake(armA, markerA, unsafe); + caseReceipt.armB.wake = await guardedWake(armB, markerB, { + composerEmpty: caseId !== "active-turn", + }); + if (caseId === "active-turn") { + const draftBytes = Buffer.from(caseReceipt.partialDraft); + const collisionBytes = Buffer.concat([draftBytes, Buffer.from(markerA)]); + const afterA = await readBytes(armA.receivedPath); + const afterB = await readBytes(armB.receivedPath); + assert.ok(afterA.subarray(-collisionBytes.length).equals(collisionBytes), + "arm A DING marker was not observed after the live partial draft"); + assert.ok(afterB.subarray(-draftBytes.length).equals(draftBytes), + "arm B partial draft changed during guarded delivery"); + assert.equal(afterB.includes(markerB), false); + caseReceipt.observedPartialDraftCollision = { + armA: true, + armB: false, + observation: "DING PTY marker arrived after the live partial draft", + }; + } + + if (caseId === "hook-failure") { + assert.deepEqual(runHook(armA, false), []); + assert.deepEqual(runHook(armB, false), []); + assert.equal(armA.queue.length, 1); + assert.equal(armB.queue.length, 1); + } + runHook(armA); + runHook(armB); + } + + compareArchives(caseId, messages, startA, startB); + caseReceipt.armA.archive = armA.archive.slice(startA).map(({ id, sha256 }) => ({ id, sha256 })); + caseReceipt.armB.archive = armB.archive.slice(startB).map(({ id, sha256 }) => ({ id, sha256 })); + receipt.cases.push(caseReceipt); + } + + assert.equal(armA.queue.length, 0); + assert.equal(armB.queue.length, 0); + assert.deepEqual( + armA.archive.map(({ id, sha256 }) => ({ id, sha256 })), + armB.archive.map(({ id, sha256 }) => ({ id, sha256 })), + ); + + const activeCase = receipt.cases.find(({ caseId }) => caseId === "active-turn"); + const idleCase = receipt.cases.find(({ caseId }) => caseId === "idle"); + assert.equal(activeCase.armA.wake.accepted, true); + assert.equal(activeCase.armB.wake.accepted, false); + assert.equal(activeCase.armB.wake.reason, "active"); + assert.equal(idleCase.armB.wake.attempted, true); + assert.equal(idleCase.armB.wake.accepted, true); + + const observedPartialDraftCollisionsA = receipt.cases + .filter((entry) => entry.observedPartialDraftCollision?.armA) + .length; + const observedPartialDraftCollisionsB = receipt.cases + .filter((entry) => entry.observedPartialDraftCollision?.armB) + .length; + assert.deepEqual(activeCase.partialDraftLiveBeforeDelivery, { armA: true, armB: true }); + assert.deepEqual(activeCase.observedPartialDraftCollision, { + armA: true, + armB: false, + observation: "DING PTY marker arrived after the live partial draft", + }); + + receipt.summary = { + deliveryParity: true, + deliveredMessages: armA.archive.length, + armAPtyBytes: armA.ptyBytes, + armBPtyBytes: armB.ptyBytes, + armAUnsafeWrites: armA.unsafeWrites, + armBUnsafeWrites: armB.unsafeWrites, + armAObservedPartialDraftCollisions: observedPartialDraftCollisionsA, + armBObservedPartialDraftCollisions: observedPartialDraftCollisionsB, + armAWakeAttempts: armA.wakeAttempts, + armBWakeAttempts: armB.wakeAttempts, + }; + assert.equal(receipt.summary.armAUnsafeWrites, 6); + assert.equal(receipt.summary.armBUnsafeWrites, 0); + assert.equal(receipt.summary.armAObservedPartialDraftCollisions, 1); + assert.equal(receipt.summary.armBObservedPartialDraftCollisions, 0); +} finally { + for (const arm of arms) { + arm.publisher?.close(); + arm.publisher = null; + } + await Promise.all(arms.map((arm) => removeSession(arm.name))); +} + +receipt.cleanup.sessions = (await listSessions()).length; +assert.equal(receipt.cleanup.sessions, 0); +await writeFile( + path.join(catalog, "delivery-ab-receipt.json"), + `${JSON.stringify(receipt, null, 2)}\n`, +); + +console.log("DING-AB-EXACT-PTY-HEADS-GREEN-57b1"); +console.log("DING-AB-DURABILITY-PARITY-GREEN-57b1"); +console.log("DING-AB-ACTIVE-TURN-IMPROVEMENT-GREEN-57b1"); +console.log("DING-AB-IDLE-BOUNDED-WAKE-GREEN-57b1"); +console.log("DING-AB-HOLD-RECOVERY-GREEN-57b1"); +console.log("DING-AB-FIFO-GREEN-57b1"); +console.log("DING-AB-IMPROVEMENT-GREEN-57b1"); +console.log("DING-AB-CLEANUP-MODEL-FREE-GREEN-57b1"); diff --git a/cells/ding-delivery-st2-rich-ab/README.md b/cells/ding-delivery-st2-rich-ab/README.md new file mode 100644 index 00000000..363d0cbc --- /dev/null +++ b/cells/ding-delivery-st2-rich-ab/README.md @@ -0,0 +1,40 @@ +# ding-delivery-st2-rich-ab + +Integrated, model-free DING A/B for evals issue #57 against corrected st2 +PR #123. + +- **A** is the generated unconfigured DING sidecar and preserves the existing + aggressive PTY path. +- **B** is the generated configured sidecar. Its generic adapter argv is + expanded only from the final st2-managed task environment, publishes the + harness-neutral activity envelope, and reaches PTY only through the exact + generation/revision guard. Non-idle work is claimed by the generic + `ding-control hook-owned` ingress and stays off the PTY. + +The cell runs the same nine scenarios and 11 durable messages as the frozen +PTY-boundary A/B: idle, active turn with a partial draft, long child, DND, +stale activity, hook failure, crash/restart, compaction/clear, and a three-item +FIFO burst. It verifies exact bodies and SHA256 values at the inbox before +archiving, exactly-once order in both arms, real sidecar restart, typed rich +receipts, and zero residual PTY or exec processes. + +Across those scenarios, A makes six PTY writes while activity is non-idle, +stale, or unknown; B makes none. Collision evidence is intentionally narrower: +the active-turn case directly observes A's DING bytes arriving after a live +partial draft, while B leaves that draft untouched and transfers ownership to +the hook. The other five A writes are classified only as unsafe writes, not as +observed collisions. + +Set `EVALS_ST2_PR123_ROOT` to a clean exact checkout of +`d7500b0fcad8bb268da9da96c0226d9caddbe305` built with +`cargo build --release --locked`. The accepted local-source Linux release +binary SHA256 is +`705ad3ebd0bce497a4117c7c7993505c0579fc1f7df2421e0525a22208f6949f`. +Set `EVALS_PTY_PR133_ROOT` to the clean built stacked PTY head +`743ceb796a41a3282e31382575bff0d0e3826d59`. + +This is integrated st2 configured-DING evidence with a synthetic generic +adapter and deterministic hook consumer. It is not real-provider acceptance: +immutable Codex and Claude activity/hook fixtures, owner review, live-pane +acceptance, merged dependencies, and release artifacts remain separate gates. +No model or provider process is launched. diff --git a/cells/ding-delivery-st2-rich-ab/ding-delivery-st2-rich-ab.kdl b/cells/ding-delivery-st2-rich-ab/ding-delivery-st2-rich-ab.kdl new file mode 100644 index 00000000..b90aec98 --- /dev/null +++ b/cells/ding-delivery-st2-rich-ab/ding-delivery-st2-rich-ab.kdl @@ -0,0 +1,42 @@ +// Integrated exact-source st2 configured-DING versus aggressive control; no model seat. +eval { + copy "./fixture" + max-timeout "120s" + + run "integrated-ab" { + command "node ./integrated-ab.mjs" + } + + judges { + judge "EXACT ARTIFACTS — corrected st2 and stacked PTY sources/builds are exact and clean" { + exec "grep -Fqx ST2-RICH-AB-EXACT-ARTIFACTS-GREEN-57i1 $RUNS_DIR/integrated-ab.out" + } + judge "TASK ENV — generated rich argv launches from final managed env with exact boundaries" { + exec "grep -Fqx ST2-RICH-AB-TASK-ENV-GREEN-57i1 $RUNS_DIR/integrated-ab.out && jq -e '.taskEnvironment.correctedArgvExpansion and .taskEnvironment.noShell and .taskEnvironment.adapterLaunches >= 2' $CATALOG/integrated-ab-receipt.json >/dev/null" + } + judge "SELECTION — A uses fallback while B uses configured rich receipts only" { + exec "grep -Fqx ST2-RICH-AB-SELECTION-GREEN-57i1 $RUNS_DIR/integrated-ab.out" + } + judge "DURABILITY — both arms deliver the same 11 exact bodies/hashes once and in order" { + exec "grep -Fqx ST2-RICH-AB-DURABILITY-GREEN-57i1 $RUNS_DIR/integrated-ab.out && jq -e '.summary.deliveryParity and .summary.deliveredMessages == 11' $CATALOG/integrated-ab-receipt.json >/dev/null" + } + judge "DISCRIMINATOR — aggressive unsafe writes fall from six to zero" { + exec "grep -Fqx ST2-RICH-AB-IMPROVEMENT-GREEN-57i1 $RUNS_DIR/integrated-ab.out && jq -e '.summary.armAUnsafeWrites == 6 and .summary.armBUnsafeWrites == 0' $CATALOG/integrated-ab-receipt.json >/dev/null" + } + judge "OBSERVED COLLISION — active-turn partial draft collides only in aggressive A" { + exec "jq -e '.summary.armAObservedPartialDraftCollisions == 1 and .summary.armBObservedPartialDraftCollisions == 0 and ([.cases[] | select(.caseId == \"active-turn\") | .observedPartialDraftCollision] == [{\"armA\":true,\"armB\":false,\"observation\":\"DING PTY bytes arrived after the live partial draft\"}])' $CATALOG/integrated-ab-receipt.json >/dev/null" + } + judge "OWNERSHIP — B uses five guarded idle writes and six exact hook-owned deliveries" { + exec "grep -Fqx ST2-RICH-AB-OWNERSHIP-GREEN-57i1 $RUNS_DIR/integrated-ab.out && jq -e '.summary.armBGuardedWrites == 5 and .summary.armBHookOwned == 6' $CATALOG/integrated-ab-receipt.json >/dev/null" + } + judge "HOLDS/RESTART — DND, stale, hook failure, and real task restart fail closed" { + exec "grep -Fqx ST2-RICH-AB-HOLD-RESTART-GREEN-57i1 $RUNS_DIR/integrated-ab.out" + } + judge "FIFO — the three-message burst preserves exact order without duplicates" { + exec "grep -Fqx ST2-RICH-AB-FIFO-GREEN-57i1 $RUNS_DIR/integrated-ab.out" + } + judge "CLEANUP/MODEL FREE — no provider calls, PTYs, or sidecar processes remain" { + exec "grep -Fqx ST2-RICH-AB-CLEANUP-MODEL-FREE-GREEN-57i1 $RUNS_DIR/integrated-ab.out && jq -e '.modelCalls == 0 and .cleanup.sessions == 0 and .cleanup.execProcesses == 0' $CATALOG/integrated-ab-receipt.json >/dev/null" + } + } +} diff --git a/cells/ding-delivery-st2-rich-ab/fixture/ab-child.mjs b/cells/ding-delivery-st2-rich-ab/fixture/ab-child.mjs new file mode 100644 index 00000000..30d9cad0 --- /dev/null +++ b/cells/ding-delivery-st2-rich-ab/fixture/ab-child.mjs @@ -0,0 +1,33 @@ +import { appendFileSync } from "node:fs"; + +const receivedPath = process.argv[2]; +if (!receivedPath) throw new Error("received-byte path is required"); + +process.stdin.setRawMode?.(true); +process.stdin.resume(); +let pending = ""; +process.stdin.on("data", (chunk) => { + appendFileSync(receivedPath, chunk); + pending += chunk.toString("utf8"); + for (;;) { + const match = pending.match(/\u001b\[200~([\s\S]*?)\u001b\[201~[\r\n]/); + if (!match) break; + pending = pending.slice(match.index + match[0].length); + const notice = match[1]; + process.stdout.write( + `Messages to be submitted after next tool call:\r\n${notice}\r\n\r\n` + + "\u001b[1m›\u001b[1C\u001b[22;2mReady\r\n\r\n" + + " \u001b[0mgpt-5.6-sol xhigh · /workspace\r\n", + ); + } +}); +process.stdout.write("ST2-RICH-AB-READY\r\n"); + +const timer = setInterval(() => {}, 1000); +function stop() { + clearInterval(timer); + process.exit(0); +} + +process.on("SIGTERM", stop); +process.on("SIGINT", stop); diff --git a/cells/ding-delivery-st2-rich-ab/fixture/activity-adapter.mjs b/cells/ding-delivery-st2-rich-ab/fixture/activity-adapter.mjs new file mode 100644 index 00000000..b06ff881 --- /dev/null +++ b/cells/ding-delivery-st2-rich-ab/fixture/activity-adapter.mjs @@ -0,0 +1,122 @@ +import { appendFile, readFile } from "node:fs/promises"; +import path from "node:path"; +import { pathToFileURL } from "node:url"; + +function option(name) { + const index = process.argv.indexOf(name); + if (index < 0 || index + 1 >= process.argv.length) { + throw new Error(`${name} is required`); + } + return process.argv[index + 1]; +} + +const controlPath = option("--control"); +const proofPath = option("--proof"); +const tracePath = option("--trace"); +const session = process.env.ST_AGENT; +const packageRoot = process.env.PTY_PACKAGE_ROOT; +if (!session || !packageRoot) throw new Error("managed adapter environment is incomplete"); + +await appendFile( + proofPath, + `${JSON.stringify({ + pid: process.pid, + argv: process.argv.slice(2), + env: { + CATALOG: process.env.CATALOG, + ST_ROOT: process.env.ST_ROOT, + ST_AGENT: process.env.ST_AGENT, + PTY_ROOT: process.env.PTY_ROOT, + ADAPTER_ROOT: process.env.ADAPTER_ROOT, + ADAPTER_CONTROL: process.env.ADAPTER_CONTROL, + ADAPTER_PROOF: process.env.ADAPTER_PROOF, + ADAPTER_TRACE: process.env.ADAPTER_TRACE, + PTY_PACKAGE_ROOT: process.env.PTY_PACKAGE_ROOT, + PATH: process.env.PATH, + }, + })}\n`, +); + +const client = await import(pathToFileURL(path.join(packageRoot, "dist", "client-api.js"))); +const { connectActivityPublisher, queryStats } = client; + +async function sleep(milliseconds) { + await new Promise((resolve) => setTimeout(resolve, milliseconds)); +} + +async function waitForSession() { + for (let attempt = 0; attempt < 400; attempt += 1) { + try { + const stats = await queryStats(session); + if (stats.process.alive) return stats; + } catch {} + await sleep(25); + } + throw new Error(`timed out waiting for PTY session ${session}`); +} + +await waitForSession(); +const publisher = await connectActivityPublisher(session, { + producerEpoch: `st2-rich-adapter-${process.pid}`, + source: "generic-evals-57", +}); + +function ptyState(state) { + if (state === "child") return "child_command"; + if (state === "unknown") return "active"; + return state; +} + +async function publish(command) { + let finalEvent; + for (let observation = 0; observation < 2; observation += 1) { + await publisher.publish(ptyState(command.state), { turnId: command.id }); + const stats = await queryStats(session); + finalEvent = { + v: 1, + kind: "activity", + session, + incarnation: stats.activity.producerEpoch, + generation: stats.generation, + sequence: stats.activity.sequence, + state: command.state, + inputBuffer: command.inputBuffer, + validForMs: command.validForMs, + reason: `fixture:${command.id}:${observation + 1}`, + }; + process.stdout.write(`${JSON.stringify(finalEvent)}\n`); + } + await appendFile( + tracePath, + `${JSON.stringify({ id: command.id, event: finalEvent })}\n`, + ); +} + +let seen = null; +let running = false; +const timer = setInterval(async () => { + if (running) return; + running = true; + try { + const command = JSON.parse(await readFile(controlPath, "utf8")); + if (command.id !== seen) { + seen = command.id; + await publish(command); + } + } catch (error) { + if (error?.code !== "ENOENT") { + await appendFile(tracePath, `${JSON.stringify({ error: error.message })}\n`); + } + } finally { + running = false; + } +}, 20); + +function stop() { + clearInterval(timer); + publisher.close(); + process.exit(0); +} + +process.on("SIGTERM", stop); +process.on("SIGINT", stop); diff --git a/cells/ding-delivery-st2-rich-ab/fixture/integrated-ab.mjs b/cells/ding-delivery-st2-rich-ab/fixture/integrated-ab.mjs new file mode 100644 index 00000000..3124de60 --- /dev/null +++ b/cells/ding-delivery-st2-rich-ab/fixture/integrated-ab.mjs @@ -0,0 +1,992 @@ +import assert from "node:assert/strict"; +import { execFile } from "node:child_process"; +import { createHash } from "node:crypto"; +import { + mkdir, + readFile, + readdir, + readlink, + realpath, + rename, + stat, + writeFile, +} from "node:fs/promises"; +import path from "node:path"; +import { pathToFileURL } from "node:url"; +import { promisify } from "node:util"; + +const execFileAsync = promisify(execFile); + +const ST2_HEAD = "d7500b0fcad8bb268da9da96c0226d9caddbe305"; +const ST2_CARGO_LOCK_SHA256 = "56d7956f328d7525eea04c70f5767acb3bc207c9509e7e2cc332444a4ede2f3e"; +const ST2_BINARY_SHA256 = "705ad3ebd0bce497a4117c7c7993505c0579fc1f7df2421e0525a22208f6949f"; +const PTY_ACTIVITY_HEAD = "46c71d31c0d6daee43adf568061b2b84a65ae8c0"; +const PTY_GUARDED_HEAD = "743ceb796a41a3282e31382575bff0d0e3826d59"; +const PTY_PACKAGE_LOCK_SHA256 = "43189b5b5b1d560be4b9102d2ed0793d89b9b01dbb3bbd976948f10c6669118d"; +const PTY_CLI_SHA256 = "60e90a8a0041845c0e382e2cf3873c005bed2a95db8991697e0449c3a71a558d"; +const PTY_CLIENT_SHA256 = "b3026b70ea0aff6ab6a07c4371ce11cd3f600c75ac9584e653fe702212658bae"; +const CASES = [ + "idle", + "active-turn", + "long-child", + "dnd", + "stale-unknown", + "hook-failure", + "crash-restart", + "compaction-clear", + "fifo-burst", +]; +const UNSAFE_CASES = new Set([ + "active-turn", + "long-child", + "stale-unknown", + "hook-failure", + "crash-restart", + "compaction-clear", +]); +const TOTAL_MESSAGES = 11; + +function requiredEnv(name) { + const value = process.env[name]; + assert.ok(value, `${name} is required`); + return path.resolve(value); +} + +function digest(value) { + return createHash("sha256").update(value).digest("hex"); +} + +async function sha256(file) { + return digest(await readFile(file)); +} + +async function sleep(milliseconds) { + await new Promise((resolve) => setTimeout(resolve, milliseconds)); +} + +async function waitFor(label, operation, predicate, attempts = 400, pause = 25) { + let lastError; + for (let attempt = 0; attempt < attempts; attempt += 1) { + try { + const value = await operation(); + if (predicate(value)) return value; + } catch (error) { + lastError = error; + } + await sleep(pause); + } + throw new Error(`timed out waiting for ${label}${lastError ? `: ${lastError.message}` : ""}`); +} + +async function exists(file) { + try { + await stat(file); + return true; + } catch (error) { + if (error.code === "ENOENT") return false; + throw error; + } +} + +async function readBytes(file) { + try { + return await readFile(file); + } catch (error) { + if (error.code === "ENOENT") return Buffer.alloc(0); + throw error; + } +} + +function occurrences(buffer, text) { + const haystack = buffer.toString("utf8"); + let count = 0; + let offset = 0; + while ((offset = haystack.indexOf(text, offset)) >= 0) { + count += 1; + offset += text.length; + } + return count; +} + +function kdl(value) { + return JSON.stringify(value); +} + +const st2Root = requiredEnv("EVALS_ST2_PR123_ROOT"); +const ptyPackageRoot = requiredEnv("EVALS_PTY_PR133_ROOT"); +const catalog = requiredEnv("CATALOG"); +const st2Bin = path.join(st2Root, "target", "release", "st2"); +const ptyCli = path.join(ptyPackageRoot, "bin", "pty"); +const st2BinDir = path.dirname(st2Bin); +const ptyBinDir = path.dirname(ptyCli); +const net = path.join(catalog, "integrated-net"); +const ptyRoot = path.join(catalog, "i57pty"); +const stateRoot = path.join(catalog, "integrated-state"); +const execState = path.join(stateRoot, "st2", "matrix", "exec"); +const controlPath = path.join(net, "activity-control.json"); +const proofPath = path.join(net, "adapter-launch.jsonl"); +const tracePath = path.join(net, "adapter-trace.jsonl"); +const aBytesPath = path.join(net, "received-a.bin"); +const bBytesPath = path.join(net, "received-b.bin"); +const runtimePath = [st2BinDir, ptyBinDir, process.env.PATH].join(":"); +const runtimeEnv = { + ...process.env, + PATH: runtimePath, + PTY_ROOT: ptyRoot, + XDG_STATE_HOME: stateRoot, +}; +process.env.PTY_ROOT = ptyRoot; + +async function run(program, args, options = {}) { + return execFileAsync(program, args, { + env: runtimeEnv, + cwd: catalog, + maxBuffer: 4 * 1024 * 1024, + ...options, + }); +} + +async function runSt2(args) { + return run(st2Bin, args); +} + +async function runPty(args) { + return run(ptyCli, args); +} + +async function git(args, root) { + return run("git", ["-C", root, ...args]); +} + +assert.equal((await git(["rev-parse", "HEAD"], st2Root)).stdout.trim(), ST2_HEAD); +assert.equal((await git(["status", "--porcelain"], st2Root)).stdout, ""); +assert.equal(await sha256(path.join(st2Root, "Cargo.lock")), ST2_CARGO_LOCK_SHA256); +assert.equal(await sha256(st2Bin), ST2_BINARY_SHA256); +const st2Version = (await runSt2(["--version"])).stdout.trim(); +assert.match(st2Version, /^st2 0\.1\.0 — running from local source \(d7500b0, .+ ago\)$/); + +assert.equal((await git(["rev-parse", "HEAD"], ptyPackageRoot)).stdout.trim(), PTY_GUARDED_HEAD); +await git(["merge-base", "--is-ancestor", PTY_ACTIVITY_HEAD, PTY_GUARDED_HEAD], ptyPackageRoot); +assert.equal((await git(["status", "--porcelain"], ptyPackageRoot)).stdout, ""); +assert.equal(await sha256(path.join(ptyPackageRoot, "package-lock.json")), PTY_PACKAGE_LOCK_SHA256); +assert.equal(await sha256(ptyCli), PTY_CLI_SHA256); +assert.equal( + await sha256(path.join(ptyPackageRoot, "dist", "client-api.js")), + PTY_CLIENT_SHA256, +); + +const ptyClient = await import( + pathToFileURL(path.join(ptyPackageRoot, "dist", "client-api.js")) +); +const { connectActivityPublisher, queryStats, sendData } = ptyClient; + +await mkdir(path.join(net, "agents", "matrix", "a"), { recursive: true }); +await mkdir(path.join(net, "agents", "matrix", "b"), { recursive: true }); + +function agentDeclaration(identity, rich) { + const received = identity === "a" ? aBytesPath : bBytesPath; + const env = { + PATH: runtimePath, + ST_AGENT: `matrix.${identity}`, + NODE_BIN: process.execPath, + FIXTURE_ROOT: catalog, + ADAPTER_ROOT: catalog, + ADAPTER_CONTROL: controlPath, + ADAPTER_PROOF: proofPath, + ADAPTER_TRACE: tracePath, + PTY_PACKAGE_ROOT: ptyPackageRoot, + }; + const envLines = Object.entries(env) + .map(([key, value]) => ` ${key} ${kdl(value)}`) + .join("\n"); + const ding = rich + ? `ding { + adapter { + argv "$NODE_BIN" "$ADAPTER_ROOT/activity-adapter.mjs" "--control" "$ADAPTER_CONTROL" "--proof" "$ADAPTER_PROOF" "--trace" "$ADAPTER_TRACE" "space arg" "; touch forbidden" + } + }` + : "ding"; + return `agent ${kdl(identity)} { + host "matrix" + workspace "$CATALOG" + env { +${envLines} + } + argv "$NODE_BIN" "$FIXTURE_ROOT/ab-child.mjs" ${kdl(received)} + ${ding} +} +`; +} + +await writeFile( + path.join(net, "agents", "matrix", "a", "agent.kdl"), + agentDeclaration("a", false), +); +await writeFile( + path.join(net, "agents", "matrix", "b", "agent.kdl"), + agentDeclaration("b", true), +); + +await runSt2(["validate", "--catalog", net, "--host", "matrix", "--strict"]); + +function execPidPath(id) { + return path.join(execState, `${id}.pid`); +} + +async function readPid(id) { + return Number.parseInt((await readFile(execPidPath(id), "utf8")).trim(), 10); +} + +async function pidAlive(pid) { + if (!Number.isSafeInteger(pid) || pid <= 1) return false; + try { + process.kill(pid, 0); + const fields = (await readFile(`/proc/${pid}/stat`, "utf8")).split(" "); + return fields[2] !== "Z"; + } catch (error) { + if (error.code === "ESRCH" || error.code === "ENOENT") return false; + throw error; + } +} + +async function processIdentityAlive(identity) { + try { + const raw = await readFile(`/proc/${identity.pid}/stat`, "utf8"); + const close = raw.lastIndexOf(")"); + const fields = raw.slice(close + 2).split(" "); + if (fields[0] === "Z" || fields[19] !== identity.processStart) return false; + return await realpath(await readlink(`/proc/${identity.pid}/exe`)) === identity.executable; + } catch (error) { + if (error.code === "ENOENT" || error.code === "ESRCH") return false; + throw error; + } +} + +async function processGroupMembers(leaderPid) { + const members = []; + for (const name of await readdir("/proc")) { + if (!/^[0-9]+$/.test(name)) continue; + const pid = Number.parseInt(name, 10); + try { + const raw = await readFile(`/proc/${pid}/stat`, "utf8"); + const close = raw.lastIndexOf(")"); + const fields = raw.slice(close + 2).split(" "); + const processGroup = Number.parseInt(fields[2], 10); + if (processGroup !== leaderPid || fields[0] === "Z") continue; + members.push({ + pid, + processStart: fields[19], + executable: await realpath(await readlink(`/proc/${pid}/exe`)), + }); + } catch (error) { + if (error.code !== "ENOENT" && error.code !== "ESRCH") throw error; + } + } + members.sort((left, right) => left.pid - right.pid); + return members; +} + +async function listSessions() { + const { stdout } = await runPty(["list", "--json"]); + return JSON.parse(stdout); +} + +async function waitForSession(name) { + return waitFor( + `${name} running`, + () => listSessions(), + (sessions) => sessions.some((session) => session.name === name && session.status === "running"), + ); +} + +function sidecarLog(id) { + return path.join(net, "logs", `${id}.ding.log`); +} + +async function waitForLog(id, marker, start = 0) { + return waitFor( + `${id} log marker ${marker}`, + async () => (await readFile(sidecarLog(id), "utf8")).slice(start), + (text) => text.includes(marker), + ); +} + +async function jsonLines(file) { + try { + return (await readFile(file, "utf8")) + .split("\n") + .filter(Boolean) + .map((line) => JSON.parse(line)); + } catch (error) { + if (error.code === "ENOENT") return []; + throw error; + } +} + +const sidecarExecutables = []; +async function recordSidecarExecutable(id) { + const pid = await readPid(`${id}.ding`); + const exactSt2 = await realpath(st2Bin); + const members = await waitFor( + `${id} exact st2 process-group member`, + () => processGroupMembers(pid), + (entries) => entries.some((entry) => entry.executable === exactSt2), + ); + sidecarExecutables.push({ id, groupLeader: pid, members }); +} + +let expectedAdapterLaunches = 0; +async function startTeam() { + expectedAdapterLaunches += 1; + await runSt2(["up", "--catalog", net, "--host", "matrix", "--once"]); + await Promise.all([waitForSession("matrix.a"), waitForSession("matrix.b")]); + await waitForLog("matrix.a", "ready — found 0 existing unread message(s)"); + await waitForLog("matrix.b", "rich adapter ready — found 0 existing unread message(s)"); + await waitFor( + `${expectedAdapterLaunches} adapter launch proofs`, + () => jsonLines(proofPath), + (lines) => lines.length >= expectedAdapterLaunches, + ); + await recordSidecarExecutable("matrix.a"); + await recordSidecarExecutable("matrix.b"); +} + +async function setStatus(identity, state) { + await runSt2([ + "status", + identity, + "--set", + state, + "--catalog", + net, + "--host", + "matrix", + "--as", + identity, + ]); +} + +async function writeControl(command) { + const temporary = `${controlPath}.tmp`; + await writeFile(temporary, `${JSON.stringify(command)}\n`); + await rename(temporary, controlPath); + return waitFor( + `adapter trace ${command.id}`, + () => jsonLines(tracePath), + (lines) => lines.find((line) => line.id === command.id), + ); +} + +let aPublisher = null; +let aPublisherOrdinal = 0; +async function ensureAPublisher() { + if (aPublisher) return aPublisher; + aPublisherOrdinal += 1; + aPublisher = await connectActivityPublisher("matrix.a", { + producerEpoch: `aggressive-a-${aPublisherOrdinal}`, + source: "generic-evals-57-control", + }); + return aPublisher; +} + +async function setAActivity(state, id) { + if (state === "unknown") { + aPublisher?.close(); + aPublisher = null; + await waitFor( + `matrix.a unknown activity for ${id}`, + () => queryStats("matrix.a"), + (stats) => stats.activity.state === "unknown" && stats.activity.producerEpoch === null, + ); + return; + } + const publisher = await ensureAPublisher(); + await publisher.publish(state, { turnId: id }); + await waitFor( + `matrix.a ${state} activity for ${id}`, + () => queryStats("matrix.a"), + (stats) => stats.activity.state === state, + ); +} + +function activityFor(caseId) { + if (caseId === "idle" || caseId === "dnd" || caseId === "fifo-burst") return "idle"; + if (caseId === "active-turn" || caseId === "hook-failure") return "active"; + if (caseId === "long-child") return "child_command"; + return "unknown"; +} + +function richCommand(caseId, ordinal) { + const id = `${caseId}-${ordinal}-${Date.now()}`; + if (caseId === "stale-unknown") { + return { id, state: "idle", inputBuffer: "empty", validForMs: 25 }; + } + const activity = activityFor(caseId); + return { + id, + state: activity === "child_command" ? "child" : activity, + inputBuffer: caseId === "active-turn" ? "nonempty" : activity === "unknown" ? "unknown" : "empty", + validForMs: 1800, + }; +} + +async function sendMessage(identity, subject, body) { + const { stdout } = await runSt2([ + "message", + "send", + identity, + "--catalog", + net, + "--host", + "matrix", + "--as", + "matrix.sender", + "--subject", + subject, + "-m", + body, + ]); + const filename = stdout.trim(); + assert.match(filename, /^[0-9]{13}-[0-9a-z]{6}\.md$/); + return filename; +} + +async function readAndArchive(identity, filename, expected) { + const { stdout } = await runSt2([ + "message", + "read", + identity, + filename, + "--json", + "--catalog", + net, + "--host", + "matrix", + "--as", + identity, + ]); + const message = JSON.parse(stdout); + assert.equal(message.filename, filename); + assert.equal(message.subject, expected.subject); + assert.equal(message.body, expected.body); + assert.equal(digest(message.body), expected.sha256); + await runSt2([ + "message", + "archive", + identity, + filename, + "--catalog", + net, + "--host", + "matrix", + "--as", + identity, + ]); + return { id: expected.id, sha256: expected.sha256, filename }; +} + +async function hookOwn(filename) { + await runSt2([ + "ding-control", + "--identity", + "matrix.b", + "--catalog", + net, + "--host", + "matrix", + "hook-owned", + "--message", + filename, + ]); +} + +async function waitForNoOwnership(filename) { + const directory = path.join(net, "agents", "matrix", "b", "resources", "ding-control"); + await waitFor( + `ownership cleanup for ${filename}`, + async () => { + try { + return await Promise.all( + (await readdir(directory)).map(async (name) => ({ + name, + text: await readFile(path.join(directory, name), "utf8"), + })), + ); + } catch (error) { + if (error.code === "ENOENT") return []; + throw error; + } + }, + (records) => records.every((record) => !record.text.includes(filename)), + ); +} + +async function inboxCount(identity) { + const { stdout } = await runSt2([ + "message", + "ls", + identity, + "--count", + "--catalog", + net, + "--host", + "matrix", + "--as", + identity, + ]); + return Number.parseInt(stdout.trim(), 10); +} + +async function waitForExecTasksDead() { + for (const id of ["matrix.a.ding", "matrix.b.ding"]) { + if (!(await exists(execPidPath(id)))) continue; + const pid = await readPid(id); + await waitFor(`${id} stopped`, () => pidAlive(pid), (alive) => !alive); + } +} + +async function waitForTrackedProcessesDead() { + const identities = sidecarExecutables.flatMap((launch) => launch.members); + await waitFor( + "all tracked sidecar process identities stopped", + async () => Promise.all(identities.map((identity) => processIdentityAlive(identity))), + (alive) => alive.every((value) => !value), + ); +} + +async function fullRestart() { + aPublisher?.close(); + aPublisher = null; + await runSt2(["down", "--catalog", net, "--host", "matrix"]); + await waitForExecTasksDead(); + await waitForTrackedProcessesDead(); + await waitFor( + "matrix PTYs stopped", + () => listSessions(), + (sessions) => sessions.every((session) => session.status !== "running"), + ); + await startTeam(); + await Promise.all([setStatus("matrix.a", "available"), setStatus("matrix.b", "available")]); +} + +async function removeAllSessions() { + for (let attempt = 0; attempt < 160; attempt += 1) { + const sessions = await listSessions(); + if (sessions.length === 0) return; + for (const session of sessions) { + if (session.status === "running") { + await runPty(["kill", session.name]).catch(() => {}); + } else { + await runPty(["rm", session.name]).catch(() => {}); + } + } + await sleep(25); + } + throw new Error("integrated PTY sessions did not clean up"); +} + +const receipt = { + dependency: { + st2: { + head: ST2_HEAD, + cargoLockSha256: ST2_CARGO_LOCK_SHA256, + binarySha256: ST2_BINARY_SHA256, + version: st2Version, + evidenceClass: "local-exact-source-release-build", + }, + pty: { + activityHead: PTY_ACTIVITY_HEAD, + guardedHead: PTY_GUARDED_HEAD, + packageLockSha256: PTY_PACKAGE_LOCK_SHA256, + cliSha256: PTY_CLI_SHA256, + clientSha256: PTY_CLIENT_SHA256, + evidenceClass: "local-exact-source-build", + }, + }, + taskEnvironment: {}, + cases: [], + summary: {}, + modelCalls: 0, + cleanup: { sessions: null, execProcesses: null }, + boundary: { + proven: "integrated-st2-configured-generic-adapter-plus-hook-ownership", + missing: "immutable-real-provider-activity-and-hook-fixtures-live-pane-acceptance-and-release-artifacts", + }, +}; +const deliveredA = []; +const deliveredB = []; +const cleanupErrors = []; +let processedMessages = 0; + +try { + await startTeam(); + await Promise.all([setStatus("matrix.a", "available"), setStatus("matrix.b", "available")]); + + for (const caseId of CASES) { + if (caseId === "crash-restart") await fullRestart(); + const count = caseId === "fifo-burst" ? 3 : 1; + const caseReceipt = { + caseId, + messages: [], + armA: { mode: "generated-unconfigured-aggressive", ptyWrites: 0, unsafeWrites: 0 }, + armB: { mode: "generated-configured-rich", ptyWrites: 0, hookOwned: 0 }, + }; + if (caseId === "crash-restart") caseReceipt.restartedBeforeDelivery = true; + + if (caseId === "fifo-burst") { + await setAActivity("idle", caseId); + await writeControl(richCommand(caseId, 0)); + const beforeA = await readBytes(aBytesPath); + const beforeB = await readBytes(bBytesPath); + let bLogStart = (await readFile(sidecarLog("matrix.b"), "utf8")).length; + const records = []; + for (let ordinal = 0; ordinal < count; ordinal += 1) { + const id = `${caseId}-${ordinal}`; + const subject = `st2-rich-ab-${caseId}-${ordinal}`; + const body = `case=${caseId}\nordinal=${ordinal}\npayload=durable-st2-rich-ding-57\n`; + const message = { id, subject, body, sha256: digest(body) }; + const filenameA = await sendMessage("matrix.a", subject, body); + const filenameB = await sendMessage("matrix.b", subject, body); + records.push({ message, filenameA, filenameB }); + await sleep(5); + } + + await waitFor( + "aggressive FIFO burst", + () => readBytes(aBytesPath), + (bytes) => records.every(({ message }) => + occurrences(bytes, message.subject) === occurrences(beforeA, message.subject) + 1), + ); + for (let ordinal = 0; ordinal < records.length; ordinal += 1) { + const record = records[ordinal]; + if (ordinal > 0) { + bLogStart = (await readFile(sidecarLog("matrix.b"), "utf8")).length; + await writeControl(richCommand(caseId, ordinal)); + } + await waitFor( + `guarded FIFO marker ${record.message.subject}`, + () => readBytes(bBytesPath), + (bytes) => occurrences(bytes, record.message.subject) === + occurrences(beforeB, record.message.subject) + 1, + ); + await waitForLog("matrix.b", '"result":"pty-owned"', bLogStart); + const archivedB = await readAndArchive( + "matrix.b", + record.filenameB, + record.message, + ); + deliveredB.push(archivedB); + await waitForNoOwnership(record.filenameB); + } + + await sleep(1100); + const afterA = await readBytes(aBytesPath); + const afterB = await readBytes(bBytesPath); + const deltaA = afterA.subarray(beforeA.length).toString("utf8"); + const deltaB = afterB.subarray(beforeB.length).toString("utf8"); + let previousA = -1; + let previousB = -1; + for (const record of records) { + const indexA = deltaA.indexOf(record.message.subject); + const indexB = deltaB.indexOf(record.message.subject); + assert.ok(indexA > previousA, "aggressive FIFO order changed"); + assert.ok(indexB > previousB, "configured FIFO order changed"); + previousA = indexA; + previousB = indexB; + assert.equal( + occurrences(afterA, record.message.subject) - occurrences(beforeA, record.message.subject), + 1, + ); + assert.equal( + occurrences(afterB, record.message.subject) - occurrences(beforeB, record.message.subject), + 1, + ); + const archivedA = await readAndArchive( + "matrix.a", + record.filenameA, + record.message, + ); + deliveredA.push(archivedA); + caseReceipt.messages.push({ + id: record.message.id, + sha256: record.message.sha256, + filenameA: record.filenameA, + filenameB: record.filenameB, + }); + } + caseReceipt.armA.ptyWrites = records.length; + caseReceipt.armB.ptyWrites = records.length; + caseReceipt.burstQueuedBeforeArchive = true; + processedMessages += records.length; + receipt.cases.push(caseReceipt); + continue; + } + + for (let ordinal = 0; ordinal < count; ordinal += 1) { + const id = `${caseId}-${ordinal}`; + const subject = `st2-rich-ab-${caseId}-${ordinal}`; + const body = `case=${caseId}\nordinal=${ordinal}\npayload=durable-st2-rich-ding-57\n`; + const message = { id, subject, body, sha256: digest(body) }; + + if (caseId === "active-turn") { + const draft = "PARTIAL-HUMAN-DRAFT-ST2-57"; + await Promise.all([ + sendData({ name: "matrix.a", data: [draft] }), + sendData({ name: "matrix.b", data: [draft] }), + ]); + await Promise.all([ + waitFor("arm A partial draft", () => readBytes(aBytesPath), (bytes) => occurrences(bytes, draft) === 1), + waitFor("arm B partial draft", () => readBytes(bBytesPath), (bytes) => occurrences(bytes, draft) === 1), + ]); + caseReceipt.partialDraft = draft; + } + + await setAActivity(activityFor(caseId), id); + const command = richCommand(caseId, ordinal); + await writeControl(command); + if (caseId === "stale-unknown") await sleep(100); + + if (caseId === "dnd") { + await Promise.all([setStatus("matrix.a", "dnd"), setStatus("matrix.b", "dnd")]); + } + + const beforeA = await readBytes(aBytesPath); + const beforeB = await readBytes(bBytesPath); + if (caseId === "active-turn") { + const draftBytes = Buffer.from(caseReceipt.partialDraft); + assert.ok(beforeA.subarray(-draftBytes.length).equals(draftBytes), + "arm A partial draft was not live immediately before delivery"); + assert.ok(beforeB.subarray(-draftBytes.length).equals(draftBytes), + "arm B partial draft was not live immediately before delivery"); + caseReceipt.partialDraftLiveBeforeDelivery = { armA: true, armB: true }; + } + const bLogStart = (await readFile(sidecarLog("matrix.b"), "utf8")).length; + const filenameA = await sendMessage("matrix.a", subject, body); + const filenameB = await sendMessage("matrix.b", subject, body); + + if (caseId === "dnd") { + await waitForLog("matrix.b", '"reason":"presence-dnd"', bLogStart); + await sleep(250); + assert.equal(occurrences(await readBytes(aBytesPath), subject), occurrences(beforeA, subject)); + assert.equal(occurrences(await readBytes(bBytesPath), subject), occurrences(beforeB, subject)); + caseReceipt.dndHeldZeroBytes = true; + await Promise.all([setStatus("matrix.a", "available"), setStatus("matrix.b", "available")]); + await writeControl({ + ...command, + id: `${command.id}-released`, + validForMs: 1800, + }); + } + + await waitFor( + `aggressive marker ${subject}`, + () => readBytes(aBytesPath), + (bytes) => occurrences(bytes, subject) === occurrences(beforeA, subject) + 1, + ); + + if (UNSAFE_CASES.has(caseId)) { + const holdReason = caseId === "long-child" + ? "activity-child" + : caseId === "stale-unknown" + ? "activity-stale" + : caseId === "active-turn" || caseId === "hook-failure" + ? "activity-active" + : "activity-unknown"; + await waitForLog("matrix.b", `"reason":"${holdReason}"`, bLogStart); + if (caseId === "hook-failure") { + await sleep(300); + assert.equal(await inboxCount("matrix.b"), 1); + assert.equal(occurrences(await readBytes(bBytesPath), subject), occurrences(beforeB, subject)); + caseReceipt.hookFailureHeldUnread = true; + } + await hookOwn(filenameB); + await waitForLog("matrix.b", filenameB, bLogStart); + assert.equal(occurrences(await readBytes(bBytesPath), subject), occurrences(beforeB, subject)); + caseReceipt.armB.hookOwned += 1; + } else { + await waitFor( + `guarded marker ${subject}`, + () => readBytes(bBytesPath), + (bytes) => occurrences(bytes, subject) === occurrences(beforeB, subject) + 1, + ); + await waitForLog("matrix.b", '"result":"pty-owned"', bLogStart); + caseReceipt.armB.ptyWrites += 1; + } + + await sleep(1100); + const afterA = await readBytes(aBytesPath); + const afterB = await readBytes(bBytesPath); + const aWrites = occurrences(afterA, subject) - occurrences(beforeA, subject); + const bWrites = occurrences(afterB, subject) - occurrences(beforeB, subject); + assert.equal(aWrites, 1); + assert.equal(bWrites, UNSAFE_CASES.has(caseId) ? 0 : 1); + caseReceipt.armA.ptyWrites += aWrites; + if (UNSAFE_CASES.has(caseId)) { + caseReceipt.armA.unsafeWrites += aWrites; + } + if (caseId === "active-turn") { + const draft = caseReceipt.partialDraft; + const draftOffsetA = afterA.indexOf(draft); + const draftOffsetB = afterB.indexOf(draft); + const dingOffsetA = afterA.indexOf(subject, draftOffsetA + Buffer.byteLength(draft)); + assert.ok(draftOffsetA >= 0, "arm A partial draft disappeared"); + assert.ok(draftOffsetB >= 0, "arm B partial draft disappeared"); + assert.ok(dingOffsetA >= draftOffsetA + Buffer.byteLength(draft), + "arm A DING was not observed after the live partial draft"); + assert.equal(occurrences(afterB, subject), occurrences(beforeB, subject)); + caseReceipt.observedPartialDraftCollision = { + armA: true, + armB: false, + observation: "DING PTY bytes arrived after the live partial draft", + }; + } + + const archivedA = await readAndArchive("matrix.a", filenameA, message); + const archivedB = await readAndArchive("matrix.b", filenameB, message); + deliveredA.push(archivedA); + deliveredB.push(archivedB); + caseReceipt.messages.push({ id, sha256: message.sha256, filenameA, filenameB }); + await waitForNoOwnership(filenameB); + + processedMessages += 1; + } + receipt.cases.push(caseReceipt); + } + + assert.equal(processedMessages, TOTAL_MESSAGES); + assert.deepEqual( + deliveredA.map(({ id, sha256 }) => ({ id, sha256 })), + deliveredB.map(({ id, sha256 }) => ({ id, sha256 })), + ); + assert.equal(new Set(deliveredA.map(({ id }) => id)).size, TOTAL_MESSAGES); + assert.equal(new Set(deliveredB.map(({ id }) => id)).size, TOTAL_MESSAGES); + const archiveA = JSON.parse((await runSt2([ + "message", "ls", "matrix.a", "--archive", "--json", "--catalog", net, "--host", "matrix", "--as", "matrix.a", + ])).stdout); + const archiveB = JSON.parse((await runSt2([ + "message", "ls", "matrix.b", "--archive", "--json", "--catalog", net, "--host", "matrix", "--as", "matrix.b", + ])).stdout); + assert.equal(archiveA.length, TOTAL_MESSAGES); + assert.equal(archiveB.length, TOTAL_MESSAGES); + assert.equal(await inboxCount("matrix.a"), 0); + assert.equal(await inboxCount("matrix.b"), 0); + + const proofs = await jsonLines(proofPath); + assert.ok(proofs.length >= 2); + const expectedArgv = [ + "--control", controlPath, + "--proof", proofPath, + "--trace", tracePath, + "space arg", + "; touch forbidden", + ]; + for (const proof of proofs) { + assert.deepEqual(proof.argv, expectedArgv); + assert.equal(proof.env.CATALOG, net); + assert.equal(proof.env.ST_ROOT, net); + assert.equal(proof.env.ST_AGENT, "matrix.b"); + assert.equal(proof.env.PTY_ROOT, ptyRoot); + assert.equal(proof.env.ADAPTER_ROOT, catalog); + assert.equal(proof.env.ADAPTER_CONTROL, controlPath); + assert.equal(proof.env.ADAPTER_PROOF, proofPath); + assert.equal(proof.env.ADAPTER_TRACE, tracePath); + assert.equal(proof.env.PTY_PACKAGE_ROOT, ptyPackageRoot); + assert.equal(proof.env.PATH, runtimePath); + } + assert.equal(await exists(path.join(net, "agents", "matrix", "b", "forbidden")), false); + assert.equal(await exists(path.join(net, "forbidden")), false); + + const aLog = await readFile(sidecarLog("matrix.a"), "utf8"); + const bLogs = []; + for (const file of [sidecarLog("matrix.b"), `${sidecarLog("matrix.b")}.1`]) { + if (await exists(file)) bLogs.push(await readFile(file, "utf8")); + } + assert.match(aLog, /"result":"fallback"/); + assert.ok(bLogs.every((log) => !log.includes('"result":"fallback"'))); + assert.ok(bLogs.some((log) => log.includes('"result":"pty-owned"'))); + assert.ok(bLogs.some((log) => log.includes('"result":"hook-owned"'))); + + const unsafeA = receipt.cases.reduce((sum, entry) => sum + entry.armA.unsafeWrites, 0); + const unsafeB = receipt.cases + .filter((entry) => UNSAFE_CASES.has(entry.caseId)) + .reduce((sum, entry) => sum + entry.armB.ptyWrites, 0); + const hookOwnedB = receipt.cases.reduce((sum, entry) => sum + entry.armB.hookOwned, 0); + const guardedB = receipt.cases.reduce((sum, entry) => sum + entry.armB.ptyWrites, 0); + const observedPartialDraftCollisionsA = receipt.cases + .filter((entry) => entry.observedPartialDraftCollision?.armA) + .length; + const observedPartialDraftCollisionsB = receipt.cases + .filter((entry) => entry.observedPartialDraftCollision?.armB) + .length; + const activeTurn = receipt.cases.find((entry) => entry.caseId === "active-turn"); + assert.deepEqual(activeTurn.partialDraftLiveBeforeDelivery, { armA: true, armB: true }); + assert.deepEqual(activeTurn.observedPartialDraftCollision, { + armA: true, + armB: false, + observation: "DING PTY bytes arrived after the live partial draft", + }); + receipt.taskEnvironment = { + correctedArgvExpansion: true, + noShell: true, + adapterLaunches: proofs.length, + sidecarExecutables, + exactArgv: expectedArgv, + }; + receipt.summary = { + deliveryParity: true, + deliveredMessages: TOTAL_MESSAGES, + armAUnsafeWrites: unsafeA, + armBUnsafeWrites: unsafeB, + armAObservedPartialDraftCollisions: observedPartialDraftCollisionsA, + armBObservedPartialDraftCollisions: observedPartialDraftCollisionsB, + armBGuardedWrites: guardedB, + armBHookOwned: hookOwnedB, + }; + assert.equal(unsafeA, 6); + assert.equal(unsafeB, 0); + assert.equal(observedPartialDraftCollisionsA, 1); + assert.equal(observedPartialDraftCollisionsB, 0); + assert.equal(guardedB, 5); + assert.equal(hookOwnedB, 6); +} finally { + aPublisher?.close(); + aPublisher = null; + await runSt2(["down", "--catalog", net, "--host", "matrix"]).catch((error) => { + cleanupErrors.push(`st2 down: ${error.message}`); + }); + await waitForExecTasksDead().catch((error) => cleanupErrors.push(error.message)); + await waitForTrackedProcessesDead().catch((error) => cleanupErrors.push(error.message)); + await removeAllSessions().catch((error) => cleanupErrors.push(error.message)); +} + +assert.deepEqual(cleanupErrors, []); +receipt.cleanup.sessions = (await listSessions()).length; +const trackedProcesses = new Map(); +for (const launch of sidecarExecutables) { + for (const member of launch.members) { + trackedProcesses.set(`${member.pid}:${member.processStart}`, member); + } +} +let liveCleanupProcesses = 0; +for (const identity of trackedProcesses.values()) { + if (await processIdentityAlive(identity)) liveCleanupProcesses += 1; +} +receipt.cleanup.execProcesses = liveCleanupProcesses; +receipt.cleanup.trackedProcesses = trackedProcesses.size; +assert.equal(receipt.cleanup.sessions, 0); +assert.equal(receipt.cleanup.execProcesses, 0); + +await writeFile( + path.join(catalog, "integrated-ab-receipt.json"), + `${JSON.stringify(receipt, null, 2)}\n`, +); + +console.log("ST2-RICH-AB-EXACT-ARTIFACTS-GREEN-57i1"); +console.log("ST2-RICH-AB-TASK-ENV-GREEN-57i1"); +console.log("ST2-RICH-AB-SELECTION-GREEN-57i1"); +console.log("ST2-RICH-AB-DURABILITY-GREEN-57i1"); +console.log("ST2-RICH-AB-IMPROVEMENT-GREEN-57i1"); +console.log("ST2-RICH-AB-OWNERSHIP-GREEN-57i1"); +console.log("ST2-RICH-AB-HOLD-RESTART-GREEN-57i1"); +console.log("ST2-RICH-AB-FIFO-GREEN-57i1"); +console.log("ST2-RICH-AB-CLEANUP-MODEL-FREE-GREEN-57i1"); diff --git a/cells/ding-pty-activity-lease/README.md b/cells/ding-pty-activity-lease/README.md new file mode 100644 index 00000000..de6e09a6 --- /dev/null +++ b/cells/ding-pty-activity-lease/README.md @@ -0,0 +1,18 @@ +# ding-pty-activity-lease + +Model-free black-box coverage for the generic PTY activity lease introduced by +draft PTY PR #131 at exact head +`46c71d31c0d6daee43adf568061b2b84a65ae8c0`. The runnable experiment uses the +stacked PR #133 successor source at exact head +`743ceb796a41a3282e31382575bff0d0e3826d59`, which contains that exact base +and the owner-reviewed combined fixtures. + +Set `EVALS_PTY_PR133_ROOT` to a clean checkout at the stacked head after +`npm ci && npm run build`. The cell verifies both Git identities before using +the public `connectActivityPublisher()` and `queryStats()` APIs. This is +exact-source experiment evidence, not a release artifact. + +The cell proves ordered activity transitions, single-publisher ownership, +fail-closed sequence handling, diagnostic-only alternate-screen state, +generation reset, and zero remaining PTY sessions. It launches no provider or +model. diff --git a/cells/ding-pty-activity-lease/ding-pty-activity-lease.kdl b/cells/ding-pty-activity-lease/ding-pty-activity-lease.kdl new file mode 100644 index 00000000..b6b453cb --- /dev/null +++ b/cells/ding-pty-activity-lease/ding-pty-activity-lease.kdl @@ -0,0 +1,36 @@ +// Exact-source, model-free black-box contract for PTY PR #131 through stacked PR #133. +eval { + copy "./fixture" + max-timeout "60s" + + run "activity-lease" { + command "node ./activity-lease.mjs" + } + + judges { + judge "EXACT SOURCE — stacked PTY head contains the exact activity base and clean package lock" { + exec "grep -Fqx PTY-PR131-EXACT-HEAD-GREEN-57a1 $RUNS_DIR/activity-lease.out" + } + judge "GENERATION BOUND — initial and disconnected activity are unknown in the live daemon generation" { + exec "grep -Fqx ACTIVITY-GENERATION-UNKNOWN-GREEN-57a1 $RUNS_DIR/activity-lease.out" + } + judge "ORDERED STATES — active, child_command, and idle publish in one epoch with strict sequence" { + exec "grep -Fqx ACTIVITY-ORDERED-STATES-GREEN-57a1 $RUNS_DIR/activity-lease.out" + } + judge "SINGLE OWNER — a competing publisher is rejected without disturbing the live owner" { + exec "grep -Fqx ACTIVITY-SINGLE-OWNER-GREEN-57a1 $RUNS_DIR/activity-lease.out" + } + judge "FAIL CLOSED — a skipped owner sequence resets authority to unknown" { + exec "grep -Fqx ACTIVITY-SEQUENCE-FAIL-CLOSED-GREEN-57a1 $RUNS_DIR/activity-lease.out" + } + judge "DIAGNOSTIC ONLY — alternate-screen state never implies semantic idle" { + exec "grep -Fqx ACTIVITY-DIAGNOSTIC-NONAUTHORITY-GREEN-57a1 $RUNS_DIR/activity-lease.out" + } + judge "RESTART — daemon replacement changes generation and starts unknown" { + exec "grep -Fqx ACTIVITY-RESTART-GENERATION-GREEN-57a1 $RUNS_DIR/activity-lease.out" + } + judge "CLEANUP/MODEL FREE — receipt records zero model calls and no PTY sessions remain" { + exec "grep -Fqx ACTIVITY-CLEANUP-MODEL-FREE-GREEN-57a1 $RUNS_DIR/activity-lease.out && jq -e '.modelCalls == 0 and .cleanup.sessions == 0' $CATALOG/activity-lease-receipt.json >/dev/null" + } + } +} diff --git a/cells/ding-pty-activity-lease/fixture/activity-lease.mjs b/cells/ding-pty-activity-lease/fixture/activity-lease.mjs new file mode 100644 index 00000000..f54ec998 --- /dev/null +++ b/cells/ding-pty-activity-lease/fixture/activity-lease.mjs @@ -0,0 +1,256 @@ +import assert from "node:assert/strict"; +import { execFile } from "node:child_process"; +import { createHash } from "node:crypto"; +import { readFile, writeFile } from "node:fs/promises"; +import net from "node:net"; +import path from "node:path"; +import { pathToFileURL } from "node:url"; +import { promisify } from "node:util"; + +const execFileAsync = promisify(execFile); +const ACTIVITY_HEAD = "46c71d31c0d6daee43adf568061b2b84a65ae8c0"; +const STACKED_HEAD = "743ceb796a41a3282e31382575bff0d0e3826d59"; +const PACKAGE_LOCK_SHA256 = "43189b5b5b1d560be4b9102d2ed0793d89b9b01dbb3bbd976948f10c6669118d"; + +function requiredEnv(name) { + const value = process.env[name]; + assert.ok(value, `${name} is required`); + return path.resolve(value); +} + +async function sha256(file) { + return createHash("sha256").update(await readFile(file)).digest("hex"); +} + +async function sleep(milliseconds) { + await new Promise((resolve) => setTimeout(resolve, milliseconds)); +} + +async function waitFor(label, operation, predicate, attempts = 120) { + let lastError; + for (let attempt = 0; attempt < attempts; attempt += 1) { + try { + const value = await operation(); + if (predicate(value)) return value; + } catch (error) { + lastError = error; + } + await sleep(25); + } + throw new Error(`timed out waiting for ${label}${lastError ? `: ${lastError.message}` : ""}`); +} + +const packageRoot = requiredEnv("EVALS_PTY_PR133_ROOT"); +const catalog = requiredEnv("CATALOG"); +const ptyRoot = path.join(catalog, "pty-pr133-activity"); +const cli = path.join(packageRoot, "bin", "pty"); +const child = path.join(catalog, "alternate-screen-child.mjs"); +process.env.PTY_ROOT = ptyRoot; + +const gitHead = (await execFileAsync("git", ["-C", packageRoot, "rev-parse", "HEAD"])).stdout.trim(); +assert.equal(gitHead, STACKED_HEAD); +await execFileAsync("git", ["-C", packageRoot, "merge-base", "--is-ancestor", ACTIVITY_HEAD, STACKED_HEAD]); +assert.equal( + (await execFileAsync("git", ["-C", packageRoot, "status", "--porcelain"])).stdout, + "", +); +assert.equal(await sha256(path.join(packageRoot, "package-lock.json")), PACKAGE_LOCK_SHA256); + +const client = await import(pathToFileURL(path.join(packageRoot, "dist", "client-api.js"))); +const { + MessageType, + PacketReader, + connectActivityPublisher, + encodeActivity, + getSocketPath, + queryStats, +} = client; + +const cliEnv = { ...process.env, PTY_ROOT: ptyRoot }; +async function runCli(args) { + return execFileAsync(cli, args, { env: cliEnv, maxBuffer: 1024 * 1024 }); +} + +async function listSessions() { + const { stdout } = await runCli(["list", "--json"]); + return JSON.parse(stdout); +} + +async function removeSession(name) { + for (let attempt = 0; attempt < 120; attempt += 1) { + const session = (await listSessions()).find((entry) => entry.name === name); + if (!session) return; + if (session.status === "running") { + await runCli(["kill", name]).catch(() => {}); + } else { + await runCli(["rm", name]).catch(() => {}); + } + await sleep(25); + } + throw new Error(`session ${name} did not clean up`); +} + +async function spawnSession(name) { + await runCli(["run", "-d", "--id", name, "--", process.execPath, child]); + return waitFor( + `${name} alternate-screen startup`, + () => queryStats(name), + (stats) => stats.process.alive && stats.modes.alternateScreen, + ); +} + +async function connectRawActivity(name) { + const socket = net.createConnection(getSocketPath(name)); + const reader = new PacketReader(); + let pending = null; + socket.on("data", (data) => { + for (const packet of reader.feed(Buffer.isBuffer(data) ? data : Buffer.from(data))) { + if (packet.type !== MessageType.ACTIVITY || pending === null) continue; + const resolve = pending; + pending = null; + resolve(JSON.parse(packet.payload.toString("utf8"))); + } + }); + await new Promise((resolve, reject) => { + socket.once("connect", resolve); + socket.once("error", reject); + }); + return { + async request(command) { + assert.equal(pending, null, "raw activity request already pending"); + const response = new Promise((resolve, reject) => { + const timer = setTimeout(() => { + pending = null; + reject(new Error("raw activity response timeout")); + }, 2000); + pending = (value) => { + clearTimeout(timer); + resolve(value); + }; + }); + socket.write(encodeActivity(command)); + return response; + }, + close() { + socket.destroy(); + }, + }; +} + +const name = "ding-activity-lease"; +const receipt = { + dependency: { + activityHead: ACTIVITY_HEAD, + stackedHead: STACKED_HEAD, + packageLockSha256: PACKAGE_LOCK_SHA256, + cliSha256: await sha256(cli), + clientSha256: await sha256(path.join(packageRoot, "dist", "client-api.js")), + evidenceClass: "local-exact-source-build", + }, + activity: {}, + modelCalls: 0, + cleanup: { sessions: null }, +}; + +try { + const initial = await spawnSession(name); + assert.equal(initial.activity.state, "unknown"); + assert.equal(initial.activity.generation, initial.generation); + assert.equal(initial.activity.producerEpoch, null); + assert.equal(initial.activity.sequence, 0); + assert.equal(initial.modes.alternateScreen, true); + receipt.activity.initial = initial.activity; + + const owner = await connectActivityPublisher(name, { + producerEpoch: "eval-epoch-a", + source: "generic-eval", + }); + const claimed = await waitFor( + "activity claim", + () => queryStats(name), + (stats) => stats.activity.producerEpoch === "eval-epoch-a", + ); + assert.equal(claimed.activity.state, "unknown"); + assert.equal(claimed.activity.sequence, 0); + + await assert.rejects( + connectActivityPublisher(name, { producerEpoch: "competing-epoch" }), + /activity lease already held/, + ); + const afterCompetition = await queryStats(name); + assert.equal(afterCompetition.activity.producerEpoch, "eval-epoch-a"); + assert.equal(afterCompetition.activity.sequence, 0); + + const active = await owner.publish("active", { turnId: "turn-57" }); + const childCommand = await owner.publish("child_command", { turnId: "turn-57" }); + const idle = await owner.publish("idle", { turnId: "turn-57" }); + assert.deepEqual( + [active.state, childCommand.state, idle.state], + ["active", "child_command", "idle"], + ); + assert.deepEqual( + [active.sequence, childCommand.sequence, idle.sequence], + [1, 2, 3], + ); + assert.equal(idle.generation, initial.generation); + assert.equal((await queryStats(name)).modes.alternateScreen, true); + receipt.activity.ordered = [active, childCommand, idle]; + owner.close(); + + const disconnected = await waitFor( + "activity disconnect reset", + () => queryStats(name), + (stats) => stats.activity.state === "unknown" && stats.activity.producerEpoch === null, + ); + assert.equal(disconnected.activity.generation, initial.generation); + + const raw = await connectRawActivity(name); + const rawClaim = await raw.request({ op: "claim", producerEpoch: "eval-epoch-skipped" }); + assert.equal(rawClaim.ok, true); + const skipped = await raw.request({ + op: "set", + producerEpoch: "eval-epoch-skipped", + sequence: 2, + state: "idle", + }); + assert.equal(skipped.ok, false); + assert.match(skipped.error, /activity sequence must be 1/); + assert.equal(skipped.activity.state, "unknown"); + assert.equal(skipped.activity.producerEpoch, null); + raw.close(); + const afterSkipped = await queryStats(name); + assert.equal(afterSkipped.activity.state, "unknown"); + assert.equal(afterSkipped.activity.sequence, 0); + receipt.activity.skippedSequence = skipped; + + const firstGeneration = afterSkipped.generation; + await removeSession(name); + const restarted = await spawnSession(name); + assert.notEqual(restarted.generation, firstGeneration); + assert.equal(restarted.activity.generation, restarted.generation); + assert.equal(restarted.activity.state, "unknown"); + assert.equal(restarted.activity.producerEpoch, null); + receipt.activity.restart = { + before: firstGeneration, + after: restarted.generation, + activity: restarted.activity, + }; +} finally { + await removeSession(name); +} + +receipt.cleanup.sessions = (await listSessions()).length; +assert.equal(receipt.cleanup.sessions, 0); +await writeFile( + path.join(catalog, "activity-lease-receipt.json"), + `${JSON.stringify(receipt, null, 2)}\n`, +); + +console.log("PTY-PR131-EXACT-HEAD-GREEN-57a1"); +console.log("ACTIVITY-GENERATION-UNKNOWN-GREEN-57a1"); +console.log("ACTIVITY-ORDERED-STATES-GREEN-57a1"); +console.log("ACTIVITY-SINGLE-OWNER-GREEN-57a1"); +console.log("ACTIVITY-SEQUENCE-FAIL-CLOSED-GREEN-57a1"); +console.log("ACTIVITY-DIAGNOSTIC-NONAUTHORITY-GREEN-57a1"); +console.log("ACTIVITY-RESTART-GENERATION-GREEN-57a1"); +console.log("ACTIVITY-CLEANUP-MODEL-FREE-GREEN-57a1"); diff --git a/cells/ding-pty-activity-lease/fixture/alternate-screen-child.mjs b/cells/ding-pty-activity-lease/fixture/alternate-screen-child.mjs new file mode 100644 index 00000000..863ac027 --- /dev/null +++ b/cells/ding-pty-activity-lease/fixture/alternate-screen-child.mjs @@ -0,0 +1,13 @@ +process.stdin.setRawMode?.(true); +process.stdin.resume(); +process.stdout.write("\u001b[?1049hACTIVITY-LEASE-READY\r\n"); + +const timer = setInterval(() => {}, 1000); +function stop() { + clearInterval(timer); + process.stdout.write("\u001b[?1049l"); + process.exit(0); +} + +process.on("SIGTERM", stop); +process.on("SIGINT", stop); diff --git a/cells/ding-pty-guarded-send/README.md b/cells/ding-pty-guarded-send/README.md new file mode 100644 index 00000000..76c0458e --- /dev/null +++ b/cells/ding-pty-guarded-send/README.md @@ -0,0 +1,15 @@ +# ding-pty-guarded-send + +Model-free black-box coverage for revision-guarded PTY input introduced by +draft PTY PR #133 at exact head +`743ceb796a41a3282e31382575bff0d0e3826d59`, stacked on activity PR #131 at +`46c71d31c0d6daee43adf568061b2b84a65ae8c0`. + +Set `EVALS_PTY_PR133_ROOT` to a clean checkout at the stacked head after +`npm ci && npm run build`. The cell verifies the exact Git identities and uses +only public package APIs. + +It proves one exact guarded write, replay rejection, zero candidate bytes after +ordinary input, output, resize, activity, or generation races, viewer-without-I/O +eligibility, malformed/empty/oversize rejection, and complete cleanup. PTY does +not prove provider idle or an empty composer; those remain adapter authority. diff --git a/cells/ding-pty-guarded-send/ding-pty-guarded-send.kdl b/cells/ding-pty-guarded-send/ding-pty-guarded-send.kdl new file mode 100644 index 00000000..399a74f0 --- /dev/null +++ b/cells/ding-pty-guarded-send/ding-pty-guarded-send.kdl @@ -0,0 +1,36 @@ +// Exact-source, model-free black-box contract for PTY PR #133. +eval { + copy "./fixture" + max-timeout "90s" + + run "guarded-send" { + command "node ./guarded-send.mjs" + } + + judges { + judge "EXACT SOURCE — guarded-send head and activity base are exact and clean" { + exec "grep -Fqx PTY-PR133-EXACT-HEAD-GREEN-57g1 $RUNS_DIR/guarded-send.out" + } + judge "EXACTLY ONCE — one exact generation/revision token writes once and replay writes zero" { + exec "grep -Fqx GUARDED-SEND-EXACTLY-ONCE-GREEN-57g1 $RUNS_DIR/guarded-send.out" + } + judge "INPUT RACES — key, newline, paste, escape, and supervisor input reject candidate bytes" { + exec "grep -Fqx GUARDED-SEND-INPUT-RACES-GREEN-57g1 $RUNS_DIR/guarded-send.out" + } + judge "OUTPUT/RESIZE/ACTIVITY — each generic revision race rejects candidate bytes" { + exec "grep -Fqx GUARDED-SEND-REVISION-RACES-GREEN-57g1 $RUNS_DIR/guarded-send.out" + } + judge "VIEWER — an attached non-typing viewer leaves the token valid" { + exec "grep -Fqx GUARDED-SEND-VIEWER-NO-IO-GREEN-57g1 $RUNS_DIR/guarded-send.out" + } + judge "RESTART — an old generation token rejects against a replacement daemon" { + exec "grep -Fqx GUARDED-SEND-RESTART-GREEN-57g1 $RUNS_DIR/guarded-send.out" + } + judge "INVALID — wrong generation, empty data, and oversize data write zero bytes" { + exec "grep -Fqx GUARDED-SEND-INVALID-ZERO-BYTES-GREEN-57g1 $RUNS_DIR/guarded-send.out" + } + judge "CLEANUP/MODEL FREE — receipt records zero model calls and no PTY sessions remain" { + exec "grep -Fqx GUARDED-SEND-CLEANUP-MODEL-FREE-GREEN-57g1 $RUNS_DIR/guarded-send.out && jq -e '.modelCalls == 0 and .cleanup.sessions == 0' $CATALOG/guarded-send-receipt.json >/dev/null" + } + } +} diff --git a/cells/ding-pty-guarded-send/fixture/guard-child.mjs b/cells/ding-pty-guarded-send/fixture/guard-child.mjs new file mode 100644 index 00000000..8cc223d9 --- /dev/null +++ b/cells/ding-pty-guarded-send/fixture/guard-child.mjs @@ -0,0 +1,19 @@ +import { appendFileSync } from "node:fs"; + +const receivedPath = process.argv[2]; +if (!receivedPath) throw new Error("received-byte path is required"); + +process.stdin.setRawMode?.(true); +process.stdin.resume(); +process.stdin.on("data", (chunk) => appendFileSync(receivedPath, chunk)); +process.on("SIGUSR1", () => process.stdout.write("OUTPUT-RACE\r\n")); +process.stdout.write("GUARDED-SEND-READY\r\n"); + +const timer = setInterval(() => {}, 1000); +function stop() { + clearInterval(timer); + process.exit(0); +} + +process.on("SIGTERM", stop); +process.on("SIGINT", stop); diff --git a/cells/ding-pty-guarded-send/fixture/guarded-send.mjs b/cells/ding-pty-guarded-send/fixture/guarded-send.mjs new file mode 100644 index 00000000..65362a2d --- /dev/null +++ b/cells/ding-pty-guarded-send/fixture/guarded-send.mjs @@ -0,0 +1,353 @@ +import assert from "node:assert/strict"; +import { execFile } from "node:child_process"; +import { createHash } from "node:crypto"; +import { readFile, writeFile } from "node:fs/promises"; +import path from "node:path"; +import { pathToFileURL } from "node:url"; +import { promisify } from "node:util"; + +const execFileAsync = promisify(execFile); +const ACTIVITY_HEAD = "46c71d31c0d6daee43adf568061b2b84a65ae8c0"; +const GUARDED_HEAD = "743ceb796a41a3282e31382575bff0d0e3826d59"; +const PACKAGE_LOCK_SHA256 = "43189b5b5b1d560be4b9102d2ed0793d89b9b01dbb3bbd976948f10c6669118d"; +const CANDIDATE = "DING-CANDIDATE-57"; + +function requiredEnv(name) { + const value = process.env[name]; + assert.ok(value, `${name} is required`); + return path.resolve(value); +} + +async function sha256(file) { + return createHash("sha256").update(await readFile(file)).digest("hex"); +} + +async function sleep(milliseconds) { + await new Promise((resolve) => setTimeout(resolve, milliseconds)); +} + +async function waitFor(label, operation, predicate, attempts = 160) { + let lastError; + for (let attempt = 0; attempt < attempts; attempt += 1) { + try { + const value = await operation(); + if (predicate(value)) return value; + } catch (error) { + lastError = error; + } + await sleep(25); + } + throw new Error(`timed out waiting for ${label}${lastError ? `: ${lastError.message}` : ""}`); +} + +const packageRoot = requiredEnv("EVALS_PTY_PR133_ROOT"); +const catalog = requiredEnv("CATALOG"); +const ptyRoot = path.join(catalog, "pty-pr133-guarded"); +const cli = path.join(packageRoot, "bin", "pty"); +const child = path.join(catalog, "guard-child.mjs"); +const receivedPath = path.join(catalog, "received.bin"); +process.env.PTY_ROOT = ptyRoot; + +const gitHead = (await execFileAsync("git", ["-C", packageRoot, "rev-parse", "HEAD"])).stdout.trim(); +assert.equal(gitHead, GUARDED_HEAD); +await execFileAsync("git", ["-C", packageRoot, "merge-base", "--is-ancestor", ACTIVITY_HEAD, GUARDED_HEAD]); +assert.equal( + (await execFileAsync("git", ["-C", packageRoot, "status", "--porcelain"])).stdout, + "", +); +assert.equal(await sha256(path.join(packageRoot, "package-lock.json")), PACKAGE_LOCK_SHA256); + +const client = await import(pathToFileURL(path.join(packageRoot, "dist", "client-api.js"))); +const { + MAX_GUARDED_DATA_BYTES, + SessionConnection, + compareAndSend, + connectActivityPublisher, + queryStats, + sendData, +} = client; + +const cliEnv = { ...process.env, PTY_ROOT: ptyRoot }; +async function runCli(args) { + return execFileAsync(cli, args, { env: cliEnv, maxBuffer: 1024 * 1024 }); +} + +async function listSessions() { + const { stdout } = await runCli(["list", "--json"]); + return JSON.parse(stdout); +} + +async function removeSession(name) { + for (let attempt = 0; attempt < 120; attempt += 1) { + const session = (await listSessions()).find((entry) => entry.name === name); + if (!session) return; + if (session.status === "running") { + await runCli(["kill", name]).catch(() => {}); + } else { + await runCli(["rm", name]).catch(() => {}); + } + await sleep(25); + } + throw new Error(`session ${name} did not clean up`); +} + +async function spawnSession(name) { + await runCli(["run", "-d", "--id", name, "--", process.execPath, child, receivedPath]); + return waitFor( + `${name} startup`, + () => queryStats(name), + (stats) => stats.process.alive && stats.process.pid !== null, + ); +} + +async function stableStats(name) { + let previous = await queryStats(name); + for (let attempt = 0; attempt < 80; attempt += 1) { + await sleep(25); + const current = await queryStats(name); + if (current.ioRevision === previous.ioRevision) return current; + previous = current; + } + throw new Error("PTY I/O revision did not settle"); +} + +async function received() { + try { + return await readFile(receivedPath); + } catch (error) { + if (error.code === "ENOENT") return Buffer.alloc(0); + throw error; + } +} + +async function waitForReceivedLength(minimum) { + return waitFor( + `received-byte length >= ${minimum}`, + received, + (buffer) => buffer.length >= minimum, + ); +} + +function assertCandidateAbsent(buffer) { + assert.equal(buffer.includes(Buffer.from(CANDIDATE)), false); +} + +const name = "ding-guarded-send"; +const receipt = { + dependency: { + activityHead: ACTIVITY_HEAD, + guardedHead: GUARDED_HEAD, + packageLockSha256: PACKAGE_LOCK_SHA256, + cliSha256: await sha256(cli), + clientSha256: await sha256(path.join(packageRoot, "dist", "client-api.js")), + evidenceClass: "local-exact-source-build", + }, + cases: { + inputRaces: [], + revisionRaces: [], + invalid: [], + }, + modelCalls: 0, + cleanup: { sessions: null }, +}; + +try { + const initial = await spawnSession(name); + const quiet = await stableStats(name); + assert.equal(quiet.generation, initial.generation); + + const beforeSuccess = (await received()).length; + const exact = await compareAndSend(name, { + generation: quiet.generation, + ioRevision: quiet.ioRevision, + data: "Q", + }); + assert.equal(exact.ok, true); + await waitForReceivedLength(beforeSuccess + 1); + const afterSuccess = await received(); + assert.equal(afterSuccess.subarray(beforeSuccess).toString("utf8"), "Q"); + + const replay = await compareAndSend(name, { + generation: quiet.generation, + ioRevision: quiet.ioRevision, + data: CANDIDATE, + }); + assert.equal(replay.ok, false); + await sleep(50); + assertCandidateAbsent(await received()); + receipt.cases.exactlyOnce = { exact, replay }; + + const inputCases = [ + ["recent-key", { data: ["k"] }], + ["newline", { data: ["\r"] }], + ["paste", { data: ["PASTE-57"], paste: true }], + ["escape", { data: ["\u001b"] }], + ["supervisor-input", { data: ["s"] }], + ]; + for (const [caseId, options] of inputCases) { + const token = await stableStats(name); + const before = (await received()).length; + await sendData({ name, ...options }); + const afterInput = await waitForReceivedLength(before + 1); + const response = await compareAndSend(name, { + generation: token.generation, + ioRevision: token.ioRevision, + data: CANDIDATE, + }); + assert.equal(response.ok, false, caseId); + await sleep(30); + const afterCandidate = await received(); + assert.equal(afterCandidate.length, afterInput.length, caseId); + assertCandidateAbsent(afterCandidate); + receipt.cases.inputRaces.push({ + caseId, + beforeRevision: token.ioRevision, + response, + }); + } + + const outputToken = await stableStats(name); + process.kill(outputToken.process.pid, "SIGUSR1"); + await waitFor( + "child output revision", + () => queryStats(name), + (stats) => stats.ioRevision > outputToken.ioRevision, + ); + const outputBytes = (await received()).length; + const outputRace = await compareAndSend(name, { + generation: outputToken.generation, + ioRevision: outputToken.ioRevision, + data: CANDIDATE, + }); + assert.equal(outputRace.ok, false); + assert.equal((await received()).length, outputBytes); + assertCandidateAbsent(await received()); + receipt.cases.revisionRaces.push({ caseId: "output", response: outputRace }); + + const viewer = new SessionConnection({ + name, + rows: (await queryStats(name)).terminal.rows, + cols: (await queryStats(name)).terminal.cols, + }); + await viewer.connect(); + const viewerToken = await stableStats(name); + await sleep(50); + assert.equal((await queryStats(name)).ioRevision, viewerToken.ioRevision); + const viewerBefore = (await received()).length; + const viewerSend = await compareAndSend(name, { + generation: viewerToken.generation, + ioRevision: viewerToken.ioRevision, + data: "V", + }); + assert.equal(viewerSend.ok, true); + await waitForReceivedLength(viewerBefore + 1); + receipt.cases.viewerNoIo = viewerSend; + viewer.disconnect(); + + const resizeViewer = new SessionConnection({ + name, + rows: (await queryStats(name)).terminal.rows, + cols: (await queryStats(name)).terminal.cols, + }); + await resizeViewer.connect(); + const resizeToken = await stableStats(name); + resizeViewer.resize(resizeToken.terminal.rows + 1, resizeToken.terminal.cols + 1); + await waitFor( + "resize revision", + () => queryStats(name), + (stats) => stats.ioRevision > resizeToken.ioRevision, + ); + const resizeBytes = (await received()).length; + const resizeRace = await compareAndSend(name, { + generation: resizeToken.generation, + ioRevision: resizeToken.ioRevision, + data: CANDIDATE, + }); + assert.equal(resizeRace.ok, false); + assert.equal((await received()).length, resizeBytes); + assertCandidateAbsent(await received()); + receipt.cases.revisionRaces.push({ caseId: "resize", response: resizeRace }); + resizeViewer.disconnect(); + + const activity = await connectActivityPublisher(name, { + producerEpoch: "guarded-send-epoch", + source: "generic-eval", + }); + await activity.publish("idle", { turnId: "turn-57" }); + const activityToken = await stableStats(name); + await activity.publish("active", { turnId: "turn-57" }); + const activityBytes = (await received()).length; + const activityRace = await compareAndSend(name, { + generation: activityToken.generation, + ioRevision: activityToken.ioRevision, + data: CANDIDATE, + }); + assert.equal(activityRace.ok, false); + assert.equal((await received()).length, activityBytes); + assertCandidateAbsent(await received()); + receipt.cases.revisionRaces.push({ caseId: "activity", response: activityRace }); + activity.close(); + + const invalidToken = await stableStats(name); + for (const [caseId, options] of [ + ["wrong-generation", { + generation: "wrong-generation", + ioRevision: invalidToken.ioRevision, + data: CANDIDATE, + }], + ["empty", { + generation: invalidToken.generation, + ioRevision: invalidToken.ioRevision, + data: "", + }], + ["oversize", { + generation: invalidToken.generation, + ioRevision: invalidToken.ioRevision, + data: "x".repeat(MAX_GUARDED_DATA_BYTES + 1), + }], + ]) { + const before = (await received()).length; + const response = await compareAndSend(name, options); + assert.equal(response.ok, false, caseId); + assert.equal((await received()).length, before, caseId); + assertCandidateAbsent(await received()); + receipt.cases.invalid.push({ caseId, response }); + } + + const oldToken = await stableStats(name); + await removeSession(name); + const restarted = await spawnSession(name); + assert.notEqual(restarted.generation, oldToken.generation); + const restartBytes = (await received()).length; + const restartRace = await compareAndSend(name, { + generation: oldToken.generation, + ioRevision: oldToken.ioRevision, + data: CANDIDATE, + }); + assert.equal(restartRace.ok, false); + assert.equal((await received()).length, restartBytes); + assertCandidateAbsent(await received()); + receipt.cases.restart = { + before: oldToken.generation, + after: restarted.generation, + response: restartRace, + }; +} finally { + await removeSession(name); +} + +receipt.cleanup.sessions = (await listSessions()).length; +assert.equal(receipt.cleanup.sessions, 0); +await writeFile( + path.join(catalog, "guarded-send-receipt.json"), + `${JSON.stringify(receipt, null, 2)}\n`, +); + +console.log("PTY-PR133-EXACT-HEAD-GREEN-57g1"); +console.log("GUARDED-SEND-EXACTLY-ONCE-GREEN-57g1"); +console.log("GUARDED-SEND-INPUT-RACES-GREEN-57g1"); +console.log("GUARDED-SEND-REVISION-RACES-GREEN-57g1"); +console.log("GUARDED-SEND-VIEWER-NO-IO-GREEN-57g1"); +console.log("GUARDED-SEND-RESTART-GREEN-57g1"); +console.log("GUARDED-SEND-INVALID-ZERO-BYTES-GREEN-57g1"); +console.log("GUARDED-SEND-CLEANUP-MODEL-FREE-GREEN-57g1"); diff --git a/evidence/harness-exclusions.tsv b/evidence/harness-exclusions.tsv index fb928d87..cb8ce5e6 100644 --- a/evidence/harness-exclusions.tsv +++ b/evidence/harness-exclusions.tsv @@ -1,5 +1,11 @@ cell surface reason context-resource-continuity cell deterministic native context/resource restart probe; no harness seat +ding-adapter-fixture-contract cell deterministic fixture provenance and blocker contract; no bus-connected agent +ding-delivery-a-aggressive-control cell deterministic unconfigured DING ordinary-input control; no harness or model agent +ding-delivery-st2-rich-ab cell deterministic exact-source configured generic-adapter versus aggressive DING A/B; no harness or model agent +ding-delivery-pty-ab cell deterministic PTY-boundary aggressive-versus-guarded delivery A/B; no harness or model agent +ding-pty-activity-lease cell deterministic exact-source PTY activity-lease probe; no harness or model agent +ding-pty-guarded-send cell deterministic exact-source PTY compare-and-send race probe; no harness or model agent docs judge:cold-reader one-shot offline Claude print grader; no bus identity, DING, or hook surface exec-lifecycle-logging cell deterministic native exec lifecycle probe; no harness seat hook-integrity cell deterministic native hook-installation probe; no live harness seat