diff --git a/AGENT-SPEC.md b/AGENT-SPEC.md index 366f526..7284702 100644 --- a/AGENT-SPEC.md +++ b/AGENT-SPEC.md @@ -245,6 +245,85 @@ task named `agent`. Network startup materializes active declarations for the sel reconciliation; retired and other-host declarations are skipped. A gating failure suppresses only that agent. Prefer a catalog-owned `.st2/` overlay and locally excluded tool loaders. +## Experimental read-only plans + +This candidate contract is not part of the released `0fed14b` corpus +conformance claim above. The paired model-free experiment is isolated to +[`catalog-plan-vs-direct-brief`](cells/catalog-plan-vs-direct-brief/) and runs +against st2 draft PR +[#115](https://github.com/compoundingtech/st2/pull/115) at exact current-main source +`8a76b6e71355140e5b89cd9313fcfd88c82b5cad`. + +The experiment shape descends from +[source sketch revision `5c1d142`](https://gist.github.com/myobie/d5ecfac24cd3965e095a5031cd2e00cb/5c1d1427c0556d95d13890e5c5086cd85b25d994); +the executable discovery and schema authority is the exact PR115 source above. +An agent discovers a plan only through a childless Resource link: + +```kdl +agent "app-web" { + resource "ship-remote-approvals" _tag="plan" uri="file:plans/ship-remote-approvals/plan.kdl" +} +``` + +The positional Resource name is an agent-local role. The referenced +`plan.kdl`, not the Resource envelope, owns all plan truth: + +```kdl +plan "ship-remote-approvals" { + owner "app-web" + version "0000" content="file:versions/0000.md" + version "0001" content="file:versions/0001.md" { + parent "0000" + why "Browser proof exposed an approval race." + } +} +``` + +Instead of external Markdown content, a version may keep its complete intent +inline in the same referenced `plan.kdl`: + +```kdl +plan "review-follow-up" { + owner "app-web" + version "0000" { + intent "Review every unresolved comment and report the exact final head." + } +} +``` + +Plan identity is the explicit KDL value, never its directory. External plans +require one `owner`. Each version requires exactly one `content="file:..."` or +one child `intent`; both or neither fail. A version may repeat `parent` for +multiple declared parents; parents must exist, be unique, and form an acyclic +graph. A version with parents requires one non-empty `why`. Versions are sorted, +and the frontier is derived as every version with no child, retaining concurrent +siblings. Resource URIs resolve relative to the agent KDL, while content URIs +resolve relative to the referenced `plan.kdl`; both must be relative `file:` +references to regular files inside the selected catalog. The Resource adds the +agent to `referencedBy` but owns no plan fields. Legacy `plan-ref`, childful plan +Resources, and agent-owned inline plan truth are unsupported. The experiment +stores no content digest or history. It cannot prove that an earlier declaration +or content file stayed unchanged. + +The only supported CLI is read-only: + +```sh +st2 plan validate [PATH] [--json] +st2 plan list [PATH] [--json] +st2 plan show [PATH] [--json] +st2 plan inspect [PATH] [--json] +``` + +Global `--catalog` selects the same input. The experiment has no current +pointer, execution, controller, schedule, step graph, retry, claim, receipt, +event, reconciliation mutation, agent restart, CAS, merge, or deployment +behavior. Direct KDL and direct human-to-agent planning remain supported. +The current model-free A/B finds no plan advantage for cold resume, exact +intent recovery, or acceptance evidence: both durable paths recover equally, +and neither surface reports worker acceptance. Static validation and resolved +provenance are plan-only authoring evidence; live correctness, traffic, cost, +and duration remain unresolved. + ## Validation, health, and lifecycle Canonical validation is: diff --git a/CATALOG.md b/CATALOG.md index a4f1270..a1ceef1 100644 --- a/CATALOG.md +++ b/CATALOG.md @@ -27,6 +27,7 @@ harness-native loader plus canonical hook file. | `assignment-contract-hot-assignment` | Codex | `gpt-5.6-sol` / medium | 2 | medium | `1200s` | 4 | — | **NO STRUCTURED RUN** | | `assignment-contract-hot-focus` | Codex | `gpt-5.6-sol` / medium | 2 | medium | `1200s` | 4 | — | **NO STRUCTURED RUN** | | `assignment-contract-hot-resources` | Codex | `gpt-5.6-sol` / medium | 2 | medium | `1200s` | 4 | — | **NO STRUCTURED RUN** | +| `catalog-plan-vs-direct-brief` | model-free | — | 0 | none | `60s` | 22 | — | **NO STRUCTURED RUN** | | `context-resource-continuity` | model-free | — | 0 | none | `90s` | 4 | — | **NO STRUCTURED RUN** | | `crash-ding` | mixed | `claude-sonnet-5+gpt-5.6-sol` / medium | 4 | high | `180s` | 5 | — | **NO STRUCTURED RUN** | | `ding-mode` | Claude | `claude-sonnet-5` / medium | 2 | medium | `1200s` | 4 | — | **NO STRUCTURED RUN** | @@ -95,6 +96,7 @@ while the last-run column makes a recorded failure distinct from a cell with no ## Harness hook exclusions +- `catalog-plan-vs-direct-brief` / `cell` — deterministic plan-versus-brief fixture and recovery contract; no harness seat - `adopt-only-migration` / `cell` — deterministic native adoption/replacement lifecycle probe; no harness seat - `agent-spec-resource-bindings` / `cell` — deterministic native Agent Spec Resource-envelope probe; no harness seat - `context-resource-continuity` / `cell` — deterministic native context/resource restart probe; no harness seat diff --git a/cells/catalog-plan-vs-direct-brief/README.md b/cells/catalog-plan-vs-direct-brief/README.md new file mode 100644 index 0000000..0ef697f --- /dev/null +++ b/cells/catalog-plan-vs-direct-brief/README.md @@ -0,0 +1,65 @@ +# catalog-plan-vs-direct-brief + +Model-free contract for the first experiment proposed by +[`st2 plans: today → dream`](https://gist.github.com/myobie/d5ecfac24cd3965e095a5031cd2e00cb/5c1d1427c0556d95d13890e5c5086cd85b25d994): +compare one realistic repository task delivered as a versioned local catalog +plan (A) with the same task delivered as an ordinary durable direct brief (B). + +The frozen task, repository, tools, budgets, intent revisions, judges, and done +condition are identical. The two intent documents are byte-identical at each +revision. Both arms receive durable local input and use the same evaluator-owned +receipt fields, so direct planning remains a valid control rather than an +intentionally fragile baseline. + +The deterministic fixture now runs both durable product paths: + +- the seeded repository needs a real implementation and the arm-neutral + reference solution passes public and held-out correctness tests; +- arm A recovers an initial and then revised catalog snapshot through exact + `st2 plan show`/`inspect` calls; +- arm B receives the byte-identical revisions through real isolated + `st2 message send` deliveries with `inReplyTo` lineage, then recovers them + through `message ls`/`read`; +- both arms recover the same intent after cold state loss with their remote + source offline, so cold resume, intent recovery, and steering are ties; +- neither read-only plan inspection nor direct-message delivery reports worker + acceptance, so acceptance evidence is evaluator-owned in both arms and is + also a tie; +- the plan adds native static validation and resolved provenance, but the + current model-free result is `no-measured-advantage`; and +- no model or provider is launched. + +Correctness, coordination traffic, token use, cost, and wall duration remain +unresolved live-run endpoints. The reference solution establishes a valid task +and neutral judge; it is not substituted for two agent executions. + +Arm A uses the exact Resource-linked external `plan.kdl` contract from +[st2 draft PR #115](https://github.com/compoundingtech/st2/pull/115) at source +`8a76b6e71355140e5b89cd9313fcfd88c82b5cad`, while retaining the experiment +shape from source gist revision +`5c1d1427c0556d95d13890e5c5086cd85b25d994`. A childless Agent Spec Resource +with `_tag="plan"` supplies only the agent-local role and source-relative file +link. The referenced `plan.kdl` owns the plan identity, owner, versions, and +intent truth. The comparison path keeps its external Markdown content, and one +focused second target proves the supported inline-intent form without changing +the A/B scenario. + +The accepted Linux artifact SHA256 is +`214e08874720bc546d4adf7d7977e614237baf7989cc09f6932cd991f497a753`; +hosted Nix run +[30835684680](https://github.com/compoundingtech/st2/actions/runs/30835684680), +job `91760161352`, passed. The fixture exercises only `plan validate`, `list`, +`show`, and `inspect`, and proves they do not alter the catalog. Legacy +`plan-ref` and agent-owned inline plan truth are not used. + +This plan model stores no content digest or history, so it cannot prove that an +earlier declaration, parent link, or content file stayed unchanged. + +Arm B uses the product's ordinary durable message store rather than a synthetic +thread sidecar. Neither arm receives a pre-authored acceptance receipt. The +experiment uses plain copied folders, requires no server, and sets +`casRequired` to false. The product experiment adds no current pointer, +execution, scheduling, steps, retries, progress claims, events, +reconciliation, or CAS. A provider-backed A/B remains a separate authorization +with an exact model, effort, budget, run order, spend ceiling, cleanup, and +rollback. diff --git a/cells/catalog-plan-vs-direct-brief/catalog-plan-vs-direct-brief.kdl b/cells/catalog-plan-vs-direct-brief/catalog-plan-vs-direct-brief.kdl new file mode 100644 index 0000000..e48d310 --- /dev/null +++ b/cells/catalog-plan-vs-direct-brief/catalog-plan-vs-direct-brief.kdl @@ -0,0 +1,81 @@ +// Neutral, model-free fixture contract for a catalog-plan versus direct-brief A/B. +eval { + copy "./fixture" + max-timeout "60s" + + run "fixture-contract" { + command "bash ./check-fixtures.sh" + } + run "plan-contract" { + command "bash ./check-plan-contract.sh" + } + + judges { + judge "TASK — frozen repository starts red and its arm-neutral reference implementation passes" { + exec "grep -Fqx TASK-CONTRACT-GREEN-43af $RUNS_DIR/fixture-contract.out" + } + judge "PARITY — task intent, tools, budgets, judges, and done condition match across arms" { + exec "grep -Fqx ARM-PARITY-GREEN-43af $RUNS_DIR/fixture-contract.out" + } + judge "RESTART — catalog plan and real direct-message inbox both recover after cold state loss" { + exec "grep -Fqx COLD-RESTART-TIE-43af $RUNS_DIR/fixture-contract.out" + } + judge "PARTITION — both arms recover locally while their remote sources are unavailable" { + exec "grep -Fqx LOCAL-PARTITION-TIE-43af $RUNS_DIR/fixture-contract.out" + } + judge "INTENT — both arms recover byte-identical initial and revised intent" { + exec "grep -Fqx INTENT-RECOVERY-TIE-43af $RUNS_DIR/fixture-contract.out" + } + judge "STEERING — a complete plan version and reply-linked direct brief recover equally" { + exec "grep -Fqx HUMAN-STEERING-TIE-43af $RUNS_DIR/fixture-contract.out" + } + judge "ACCEPTANCE — neither read-only plan inspection nor message delivery proves worker acceptance" { + exec "grep -Fqx ACCEPTANCE-EVIDENCE-TIE-43af $RUNS_DIR/fixture-contract.out" + } + judge "STATIC VALUE — plan validation and provenance are plan-only authoring evidence" { + exec "grep -Fqx PLAN-STATIC-EVIDENCE-GREEN-43af $RUNS_DIR/fixture-contract.out" + } + judge "DIRECT REALISM — the control uses durable st2 messages and reply lineage" { + exec "grep -Fqx DIRECT-MESSAGE-EVIDENCE-GREEN-43af $RUNS_DIR/fixture-contract.out" + } + judge "MEASUREMENT — correctness gates are separate from neutral traffic and cost observations" { + exec "grep -Fqx MEASUREMENT-CONTRACT-GREEN-43af $RUNS_DIR/fixture-contract.out" + } + judge "PROVENANCE — every authoritative input has a verified content hash" { + exec "grep -Fqx PROVENANCE-GREEN-43af $RUNS_DIR/fixture-contract.out" + } + judge "PAIRING — exact experimental source and artifact are frozen while live execution stays blocked" { + exec "grep -Fqx PRODUCT-PAIRING-GREEN-43af $RUNS_DIR/fixture-contract.out" + } + judge "STORAGE — plain local folders suffice and CAS is not required" { + exec "grep -Fqx PLAIN-FOLDER-NO-CAS-GREEN-43af $RUNS_DIR/fixture-contract.out" + } + judge "VERDICT — the model-free endpoints show no plan advantage and live endpoints stay unresolved" { + exec "grep -Fqx MODEL-FREE-OUTCOME-GREEN-43af $RUNS_DIR/fixture-contract.out" + } + judge "RUNNER — the exact experimental st2 binary matches the accepted source and SHA256" { + exec "grep -Fqx PLAN-RUNNER-PIN-GREEN-43af $RUNS_DIR/plan-contract.out" + } + judge "PLAN VALIDATE — Resource-linked external plans normalize without errors" { + exec "grep -Fqx PLAN-VALIDATE-GREEN-43af $RUNS_DIR/plan-contract.out" + } + judge "PLAN TARGETS — external Markdown content and inline intent are both exact" { + exec "grep -Fqx PLAN-TARGET-FORMS-GREEN-43af $RUNS_DIR/plan-contract.out" + } + judge "PLAN LIST — explicit identity, owner, and derived frontier are exact" { + exec "grep -Fqx PLAN-LIST-GREEN-43af $RUNS_DIR/plan-contract.out" + } + judge "PLAN SHOW — normalized intent exposes versions but not provenance paths" { + exec "grep -Fqx PLAN-SHOW-GREEN-43af $RUNS_DIR/plan-contract.out" + } + judge "PLAN INSPECT — provenance and resolved content are exact and read-only" { + exec "grep -Fqx PLAN-INSPECT-READONLY-GREEN-43af $RUNS_DIR/plan-contract.out" + } + judge "PLAN BOUNDARY — a mutable current pointer is rejected with a classified error" { + exec "grep -Fqx PLAN-BOUNDARY-GREEN-43af $RUNS_DIR/plan-contract.out" + } + judge "HERMETIC — the scaffold launches no model, provider, network client, or mutating product runtime" { + exec "grep -Fqx HERMETIC-SCAFFOLD-GREEN-43af $RUNS_DIR/fixture-contract.out" + } + } +} diff --git a/cells/catalog-plan-vs-direct-brief/fixture/arm-a/catalog-initial/agents/eval/receipt-worker/agent.kdl b/cells/catalog-plan-vs-direct-brief/fixture/arm-a/catalog-initial/agents/eval/receipt-worker/agent.kdl new file mode 100644 index 0000000..66133e0 --- /dev/null +++ b/cells/catalog-plan-vs-direct-brief/fixture/arm-a/catalog-initial/agents/eval/receipt-worker/agent.kdl @@ -0,0 +1,6 @@ +agent "receipt-worker" { + host "eval" + command "true" + resource "receipt-report" _tag="plan" uri="file:plans/receipt-report/plan.kdl" + resource "inline-intent" _tag="plan" uri="file:plans/inline-intent/plan.kdl" +} diff --git a/cells/catalog-plan-vs-direct-brief/fixture/arm-a/catalog-initial/agents/eval/receipt-worker/plans/inline-intent/plan.kdl b/cells/catalog-plan-vs-direct-brief/fixture/arm-a/catalog-initial/agents/eval/receipt-worker/plans/inline-intent/plan.kdl new file mode 100644 index 0000000..7e37f03 --- /dev/null +++ b/cells/catalog-plan-vs-direct-brief/fixture/arm-a/catalog-initial/agents/eval/receipt-worker/plans/inline-intent/plan.kdl @@ -0,0 +1,6 @@ +plan "inline-intent" { + owner "receipt-worker" + version "0000" { + intent "Keep the complete inline intent in plan.kdl." + } +} diff --git a/cells/catalog-plan-vs-direct-brief/fixture/arm-a/catalog-initial/agents/eval/receipt-worker/plans/receipt-report/plan.kdl b/cells/catalog-plan-vs-direct-brief/fixture/arm-a/catalog-initial/agents/eval/receipt-worker/plans/receipt-report/plan.kdl new file mode 100644 index 0000000..63844e9 --- /dev/null +++ b/cells/catalog-plan-vs-direct-brief/fixture/arm-a/catalog-initial/agents/eval/receipt-worker/plans/receipt-report/plan.kdl @@ -0,0 +1,4 @@ +plan "receipt-report" { + owner "receipt-worker" + version "0000" content="file:versions/0000.md" +} diff --git a/cells/catalog-plan-vs-direct-brief/fixture/arm-a/catalog-initial/agents/eval/receipt-worker/plans/receipt-report/versions/0000.md b/cells/catalog-plan-vs-direct-brief/fixture/arm-a/catalog-initial/agents/eval/receipt-worker/plans/receipt-report/versions/0000.md new file mode 100644 index 0000000..7f75d24 --- /dev/null +++ b/cells/catalog-plan-vs-direct-brief/fixture/arm-a/catalog-initial/agents/eval/receipt-worker/plans/receipt-report/versions/0000.md @@ -0,0 +1,37 @@ +# Receipt report: latest accepted pass and last run + +Revision: `0000` + +## Goal + +Implement the report core in `src/report.mjs`. The report must retain the most +recent accepted `PASS` for each cell independently from that cell's last run, +which may be a failure. + +## Scope and owner + +You own the frozen `receipt-report` repository for this task. Change only +`src/report.mjs`. + +## Required behavior + +- `parseJsonLines(text)` ignores blank lines and returns records in input order. +- Malformed JSON, non-object JSON, or a record missing non-empty `run_id`, + `cell`, or `result` fields throws a `TypeError` that identifies the input line. +- `summarize(records)` returns one row per cell, sorted by cell name. +- Each row is `{ cell, accepted_pass, last_run }`. +- `accepted_pass` is the complete most recent `PASS` record for the cell, or + `null` when the cell has no pass. +- `last_run` is the complete last record for the cell. +- Neither exported function mutates caller-owned records. + +## Invariants and allowed actions + +Use only the repository's existing Node, Bash, and Git tools. Do not access the +network or add dependencies. Do not weaken or replace tests. No action outside +the frozen repository is authorized. + +## Evidence and done condition + +Run `npm test`. Completion additionally requires the evaluator's held-out tests +to pass and a diff containing only `src/report.mjs`. diff --git a/cells/catalog-plan-vs-direct-brief/fixture/arm-a/catalog/agents/eval/receipt-worker/agent.kdl b/cells/catalog-plan-vs-direct-brief/fixture/arm-a/catalog/agents/eval/receipt-worker/agent.kdl new file mode 100644 index 0000000..66133e0 --- /dev/null +++ b/cells/catalog-plan-vs-direct-brief/fixture/arm-a/catalog/agents/eval/receipt-worker/agent.kdl @@ -0,0 +1,6 @@ +agent "receipt-worker" { + host "eval" + command "true" + resource "receipt-report" _tag="plan" uri="file:plans/receipt-report/plan.kdl" + resource "inline-intent" _tag="plan" uri="file:plans/inline-intent/plan.kdl" +} diff --git a/cells/catalog-plan-vs-direct-brief/fixture/arm-a/catalog/agents/eval/receipt-worker/plans/inline-intent/plan.kdl b/cells/catalog-plan-vs-direct-brief/fixture/arm-a/catalog/agents/eval/receipt-worker/plans/inline-intent/plan.kdl new file mode 100644 index 0000000..7e37f03 --- /dev/null +++ b/cells/catalog-plan-vs-direct-brief/fixture/arm-a/catalog/agents/eval/receipt-worker/plans/inline-intent/plan.kdl @@ -0,0 +1,6 @@ +plan "inline-intent" { + owner "receipt-worker" + version "0000" { + intent "Keep the complete inline intent in plan.kdl." + } +} diff --git a/cells/catalog-plan-vs-direct-brief/fixture/arm-a/catalog/agents/eval/receipt-worker/plans/receipt-report/plan.kdl b/cells/catalog-plan-vs-direct-brief/fixture/arm-a/catalog/agents/eval/receipt-worker/plans/receipt-report/plan.kdl new file mode 100644 index 0000000..c556d30 --- /dev/null +++ b/cells/catalog-plan-vs-direct-brief/fixture/arm-a/catalog/agents/eval/receipt-worker/plans/receipt-report/plan.kdl @@ -0,0 +1,9 @@ +plan "receipt-report" { + owner "receipt-worker" + + version "0000" content="file:versions/0000.md" + version "0001" content="file:versions/0001.md" { + parent "0000" + why "Human steering tightens input validation without changing scope." + } +} diff --git a/cells/catalog-plan-vs-direct-brief/fixture/arm-a/catalog/agents/eval/receipt-worker/plans/receipt-report/versions/0000.md b/cells/catalog-plan-vs-direct-brief/fixture/arm-a/catalog/agents/eval/receipt-worker/plans/receipt-report/versions/0000.md new file mode 100644 index 0000000..7f75d24 --- /dev/null +++ b/cells/catalog-plan-vs-direct-brief/fixture/arm-a/catalog/agents/eval/receipt-worker/plans/receipt-report/versions/0000.md @@ -0,0 +1,37 @@ +# Receipt report: latest accepted pass and last run + +Revision: `0000` + +## Goal + +Implement the report core in `src/report.mjs`. The report must retain the most +recent accepted `PASS` for each cell independently from that cell's last run, +which may be a failure. + +## Scope and owner + +You own the frozen `receipt-report` repository for this task. Change only +`src/report.mjs`. + +## Required behavior + +- `parseJsonLines(text)` ignores blank lines and returns records in input order. +- Malformed JSON, non-object JSON, or a record missing non-empty `run_id`, + `cell`, or `result` fields throws a `TypeError` that identifies the input line. +- `summarize(records)` returns one row per cell, sorted by cell name. +- Each row is `{ cell, accepted_pass, last_run }`. +- `accepted_pass` is the complete most recent `PASS` record for the cell, or + `null` when the cell has no pass. +- `last_run` is the complete last record for the cell. +- Neither exported function mutates caller-owned records. + +## Invariants and allowed actions + +Use only the repository's existing Node, Bash, and Git tools. Do not access the +network or add dependencies. Do not weaken or replace tests. No action outside +the frozen repository is authorized. + +## Evidence and done condition + +Run `npm test`. Completion additionally requires the evaluator's held-out tests +to pass and a diff containing only `src/report.mjs`. diff --git a/cells/catalog-plan-vs-direct-brief/fixture/arm-a/catalog/agents/eval/receipt-worker/plans/receipt-report/versions/0001.md b/cells/catalog-plan-vs-direct-brief/fixture/arm-a/catalog/agents/eval/receipt-worker/plans/receipt-report/versions/0001.md new file mode 100644 index 0000000..299b7a5 --- /dev/null +++ b/cells/catalog-plan-vs-direct-brief/fixture/arm-a/catalog/agents/eval/receipt-worker/plans/receipt-report/versions/0001.md @@ -0,0 +1,41 @@ +# Receipt report: latest accepted pass and last run + +Revision: `0001` + +## Goal + +Implement the report core in `src/report.mjs`. The report must retain the most +recent accepted `PASS` for each cell independently from that cell's last run, +which may be a failure. + +## Scope and owner + +You own the frozen `receipt-report` repository for this task. Change only +`src/report.mjs`. + +## Required behavior + +- `parseJsonLines(text)` ignores blank lines and returns records in input order. +- Malformed JSON, non-object JSON, or a record missing non-empty `run_id`, + `cell`, or `result` fields throws a `TypeError` that identifies the input line. +- Only `PASS` and `FAIL` results are valid. An unknown result throws a + `TypeError` that identifies its `run_id`. +- A duplicate `run_id` is invalid. Both `parseJsonLines(text)` and + `summarize(records)` throw a `TypeError` that identifies the duplicate. +- `summarize(records)` returns one row per cell, sorted by cell name. +- Each row is `{ cell, accepted_pass, last_run }`. +- `accepted_pass` is the complete most recent `PASS` record for the cell, or + `null` when the cell has no pass. +- `last_run` is the complete last record for the cell. +- Neither exported function mutates caller-owned records. + +## Invariants and allowed actions + +Use only the repository's existing Node, Bash, and Git tools. Do not access the +network or add dependencies. Do not weaken or replace tests. No action outside +the frozen repository is authorized. + +## Evidence and done condition + +Run `npm test`. Completion additionally requires the evaluator's held-out tests +to pass and a diff containing only `src/report.mjs`. diff --git a/cells/catalog-plan-vs-direct-brief/fixture/arm-b/inbox/brief-0000.md b/cells/catalog-plan-vs-direct-brief/fixture/arm-b/inbox/brief-0000.md new file mode 100644 index 0000000..7f75d24 --- /dev/null +++ b/cells/catalog-plan-vs-direct-brief/fixture/arm-b/inbox/brief-0000.md @@ -0,0 +1,37 @@ +# Receipt report: latest accepted pass and last run + +Revision: `0000` + +## Goal + +Implement the report core in `src/report.mjs`. The report must retain the most +recent accepted `PASS` for each cell independently from that cell's last run, +which may be a failure. + +## Scope and owner + +You own the frozen `receipt-report` repository for this task. Change only +`src/report.mjs`. + +## Required behavior + +- `parseJsonLines(text)` ignores blank lines and returns records in input order. +- Malformed JSON, non-object JSON, or a record missing non-empty `run_id`, + `cell`, or `result` fields throws a `TypeError` that identifies the input line. +- `summarize(records)` returns one row per cell, sorted by cell name. +- Each row is `{ cell, accepted_pass, last_run }`. +- `accepted_pass` is the complete most recent `PASS` record for the cell, or + `null` when the cell has no pass. +- `last_run` is the complete last record for the cell. +- Neither exported function mutates caller-owned records. + +## Invariants and allowed actions + +Use only the repository's existing Node, Bash, and Git tools. Do not access the +network or add dependencies. Do not weaken or replace tests. No action outside +the frozen repository is authorized. + +## Evidence and done condition + +Run `npm test`. Completion additionally requires the evaluator's held-out tests +to pass and a diff containing only `src/report.mjs`. diff --git a/cells/catalog-plan-vs-direct-brief/fixture/arm-b/inbox/brief-0001.md b/cells/catalog-plan-vs-direct-brief/fixture/arm-b/inbox/brief-0001.md new file mode 100644 index 0000000..299b7a5 --- /dev/null +++ b/cells/catalog-plan-vs-direct-brief/fixture/arm-b/inbox/brief-0001.md @@ -0,0 +1,41 @@ +# Receipt report: latest accepted pass and last run + +Revision: `0001` + +## Goal + +Implement the report core in `src/report.mjs`. The report must retain the most +recent accepted `PASS` for each cell independently from that cell's last run, +which may be a failure. + +## Scope and owner + +You own the frozen `receipt-report` repository for this task. Change only +`src/report.mjs`. + +## Required behavior + +- `parseJsonLines(text)` ignores blank lines and returns records in input order. +- Malformed JSON, non-object JSON, or a record missing non-empty `run_id`, + `cell`, or `result` fields throws a `TypeError` that identifies the input line. +- Only `PASS` and `FAIL` results are valid. An unknown result throws a + `TypeError` that identifies its `run_id`. +- A duplicate `run_id` is invalid. Both `parseJsonLines(text)` and + `summarize(records)` throw a `TypeError` that identifies the duplicate. +- `summarize(records)` returns one row per cell, sorted by cell name. +- Each row is `{ cell, accepted_pass, last_run }`. +- `accepted_pass` is the complete most recent `PASS` record for the cell, or + `null` when the cell has no pass. +- `last_run` is the complete last record for the cell. +- Neither exported function mutates caller-owned records. + +## Invariants and allowed actions + +Use only the repository's existing Node, Bash, and Git tools. Do not access the +network or add dependencies. Do not weaken or replace tests. No action outside +the frozen repository is authorized. + +## Evidence and done condition + +Run `npm test`. Completion additionally requires the evaluator's held-out tests +to pass and a diff containing only `src/report.mjs`. diff --git a/cells/catalog-plan-vs-direct-brief/fixture/arms.tsv b/cells/catalog-plan-vs-direct-brief/fixture/arms.tsv new file mode 100644 index 0000000..c67b5f4 --- /dev/null +++ b/cells/catalog-plan-vs-direct-brief/fixture/arms.tsv @@ -0,0 +1,3 @@ +arm input_kind initial_source local_resume_source steering_source acceptance_surface allowed_tools wall_budget_seconds model_token_budget judge_profile done_condition +A versioned-catalog-plan arm-a/catalog-initial/agents/eval/receipt-worker/plans/receipt-report/versions/0000.md local plan catalog inspected through st2 plan arm-a/catalog/agents/eval/receipt-worker/plans/receipt-report/versions/0001.md evaluator-owned receipt only node,bash,git 1200 40000 arm-neutral-public-plus-held-out public and held-out tests pass; only src/report.mjs differs from the seed; no dependency or network change +B durable-direct-brief arm-b/inbox/brief-0000.md local st2 message inbox read through st2 message arm-b/inbox/brief-0001.md evaluator-owned receipt only node,bash,git 1200 40000 arm-neutral-public-plus-held-out public and held-out tests pass; only src/report.mjs differs from the seed; no dependency or network change diff --git a/cells/catalog-plan-vs-direct-brief/fixture/blockers.tsv b/cells/catalog-plan-vs-direct-brief/fixture/blockers.tsv new file mode 100644 index 0000000..c5b3c69 --- /dev/null +++ b/cells/catalog-plan-vs-direct-brief/fixture/blockers.tsv @@ -0,0 +1,3 @@ +blocker_id owner dependency required_evidence status +st2-plan-runtime st2-owner https://github.com/compoundingtech/st2/pull/115 exact Resource-link source 8a76b6e71355140e5b89cd9313fcfd88c82b5cad and binary SHA256 214e08874720bc546d4adf7d7977e614237baf7989cc09f6932cd991f497a753 with prior gates and hosted run 30835684680 job 91760161352 green ready +live-ab evals-owner future provider-backed execution separate authorization naming exact plan artifact, model, effort, per-arm token and wall budgets, arm order or counterbalancing, maximum spend, cleanup, and rollback blocked diff --git a/cells/catalog-plan-vs-direct-brief/fixture/check-fixtures.sh b/cells/catalog-plan-vs-direct-brief/fixture/check-fixtures.sh new file mode 100755 index 0000000..42c338f --- /dev/null +++ b/cells/catalog-plan-vs-direct-brief/fixture/check-fixtures.sh @@ -0,0 +1,228 @@ +#!/usr/bin/env bash +set -euo pipefail + +root="${CATALOG:?CATALOG must be set}" +experiment="$root/experiment.tsv" +arms="$root/arms.tsv" +scenario="$root/scenario.tsv" +measurements="$root/measurement-schema.tsv" +receipts="$root/receipt-schema.tsv" +blockers="$root/blockers.tsv" +runner="$root/runner.tsv" +plan_declaration="$root/arm-a/catalog/agents/eval/receipt-worker/plans/receipt-report/plan.kdl" +agent_declaration="$root/arm-a/catalog/agents/eval/receipt-worker/agent.kdl" +inline_plan_declaration="$root/arm-a/catalog/agents/eval/receipt-worker/plans/inline-intent/plan.kdl" +initial_plan_declaration="$root/arm-a/catalog-initial/agents/eval/receipt-worker/plans/receipt-report/plan.kdl" +initial_agent_declaration="$root/arm-a/catalog-initial/agents/eval/receipt-worker/agent.kdl" +initial_inline_plan_declaration="$root/arm-a/catalog-initial/agents/eval/receipt-worker/plans/inline-intent/plan.kdl" +initial_plan_0000="$root/arm-a/catalog-initial/agents/eval/receipt-worker/plans/receipt-report/versions/0000.md" +plan_0000="$root/arm-a/catalog/agents/eval/receipt-worker/plans/receipt-report/versions/0000.md" +plan_0001="$root/arm-a/catalog/agents/eval/receipt-worker/plans/receipt-report/versions/0001.md" +brief_0000="$root/arm-b/inbox/brief-0000.md" +brief_0001="$root/arm-b/inbox/brief-0001.md" + +expect_header() { + file="$1" + expected="$2" + test "$(head -n 1 "$file")" = "$expected" +} + +tree_hash() { + ( + cd "$1" + find . -type f -print0 | + LC_ALL=C sort -z | + xargs -0 sha256sum | + sha256sum | + awk '{print $1}' + ) +} + +expect_header "$experiment" $'field\tvalue' +expect_header "$arms" $'arm\tinput_kind\tinitial_source\tlocal_resume_source\tsteering_source\tacceptance_surface\tallowed_tools\twall_budget_seconds\tmodel_token_budget\tjudge_profile\tdone_condition' +expect_header "$scenario" $'ordinal\tevent\tarm_a_input\tarm_b_input\tinvariant' +expect_header "$measurements" $'metric\ttype\tunit\trole\tpreferred_direction\tmodel_free_arm_a\tmodel_free_arm_b\tmodel_free_verdict\tlive_source' +expect_header "$receipts" $'field\ttype\trequired\tdescription' +expect_header "$blockers" $'blocker_id\towner\tdependency\trequired_evidence\tstatus' +expect_header "$runner" $'field\tvalue' + +test "$(awk -F '\t' 'NR > 1 { count++ } END { print count + 0 }' "$arms")" -eq 2 +test "$(awk -F '\t' 'NR > 1 { count++ } END { print count + 0 }' "$scenario")" -eq 6 +test "$(awk -F '\t' 'NR > 1 { count++ } END { print count + 0 }' "$measurements")" -eq 14 +test "$(awk -F '\t' 'NR > 1 { count++ } END { print count + 0 }' "$receipts")" -eq 24 +test "$(awk -F '\t' 'NR > 1 { count++ } END { print count + 0 }' "$blockers")" -eq 2 +test "$(awk -F '\t' 'NR > 1 { count++ } END { print count + 0 }' "$runner")" -eq 8 + +grep -Fqx $'allowed_tools\tnode,bash,git' "$experiment" +grep -Fqx $'network_policy\tdisabled' "$experiment" +grep -Fqx $'wall_budget_seconds\t1200' "$experiment" +grep -Fqx $'model_token_budget\t40000' "$experiment" +grep -Fqx $'initial_revision\t0000' "$experiment" +grep -Fqx $'steering_revision\t0001' "$experiment" +grep -Fqx $'model_free_result\tno-measured-advantage-for-cold-resume-intent-recovery-or-acceptance-evidence' "$experiment" +grep -Fqx $'source_gist\thttps://gist.github.com/myobie/d5ecfac24cd3965e095a5031cd2e00cb/5c1d1427c0556d95d13890e5c5086cd85b25d994' "$experiment" +grep -Fqx $'source_gist_revision\t5c1d1427c0556d95d13890e5c5086cd85b25d994' "$experiment" +grep -Fqx $'st2_plan_source\t8a76b6e71355140e5b89cd9313fcfd88c82b5cad' "$experiment" +grep -Fqx $'st2_plan_binary_sha256\t214e08874720bc546d4adf7d7977e614237baf7989cc09f6932cd991f497a753' "$experiment" +grep -Fqx $'live_run_status\tblocked-pending-separate-provider-authorization' "$experiment" +grep -Fqx $'st2_pr\thttps://github.com/compoundingtech/st2/pull/115' "$runner" +grep -Fqx $'source_full\t8a76b6e71355140e5b89cd9313fcfd88c82b5cad' "$runner" +grep -Fqx $'source_short\t8a76b6e' "$runner" +grep -Fqx $'binary_sha256\t214e08874720bc546d4adf7d7977e614237baf7989cc09f6932cd991f497a753' "$runner" +grep -Fqx $'source_gist_revision\t5c1d1427c0556d95d13890e5c5086cd85b25d994' "$runner" +grep -Fqx $'runtime_scope\tread-only-validate-list-show-inspect' "$runner" +grep -Fqx $'hosted_run\thttps://github.com/compoundingtech/st2/actions/runs/30835684680' "$runner" +grep -Fqx $'hosted_status\tpass' "$runner" + +cmp -s "$initial_plan_0000" "$plan_0000" +cmp -s "$plan_0000" "$brief_0000" +cmp -s "$plan_0001" "$brief_0001" +test "$(sha256sum "$plan_0000" | awk '{print $1}')" != \ + "$(sha256sum "$plan_0001" | awk '{print $1}')" + +awk -F '\t' ' + NR == 2 { + tools = $7 + wall = $8 + tokens = $9 + judge = $10 + done = $11 + next + } + NR == 3 { + if ($7 != tools || $8 != wall || $9 != tokens || $10 != judge || $11 != done) { + exit 1 + } + } +' "$arms" +grep -Fqx $'A\tversioned-catalog-plan\tarm-a/catalog-initial/agents/eval/receipt-worker/plans/receipt-report/versions/0000.md\tlocal plan catalog inspected through st2 plan\tarm-a/catalog/agents/eval/receipt-worker/plans/receipt-report/versions/0001.md\tevaluator-owned receipt only\tnode,bash,git\t1200\t40000\tarm-neutral-public-plus-held-out\tpublic and held-out tests pass; only src/report.mjs differs from the seed; no dependency or network change' "$arms" +grep -Fqx $'B\tdurable-direct-brief\tarm-b/inbox/brief-0000.md\tlocal st2 message inbox read through st2 message\tarm-b/inbox/brief-0001.md\tevaluator-owned receipt only\tnode,bash,git\t1200\t40000\tarm-neutral-public-plus-held-out\tpublic and held-out tests pass; only src/report.mjs differs from the seed; no dependency or network change' "$arms" + +grep -Fqx 'plan "receipt-report" {' "$plan_declaration" +grep -Fqx ' owner "receipt-worker"' "$plan_declaration" +grep -Fqx ' version "0000" content="file:versions/0000.md"' "$plan_declaration" +grep -Fqx ' version "0001" content="file:versions/0001.md" {' "$plan_declaration" +grep -Fqx ' parent "0000"' "$plan_declaration" +grep -Fqx ' why "Human steering tightens input validation without changing scope."' "$plan_declaration" +grep -Fqx 'plan "receipt-report" {' "$initial_plan_declaration" +grep -Fqx ' owner "receipt-worker"' "$initial_plan_declaration" +grep -Fqx ' version "0000" content="file:versions/0000.md"' "$initial_plan_declaration" +if grep -Fq 'version "0001"' "$initial_plan_declaration"; then + echo "initial plan snapshot already contains the steering revision" >&2 + exit 1 +fi +grep -Fqx 'agent "receipt-worker" {' "$agent_declaration" +grep -Fqx ' resource "receipt-report" _tag="plan" uri="file:plans/receipt-report/plan.kdl"' "$agent_declaration" +grep -Fqx ' resource "inline-intent" _tag="plan" uri="file:plans/inline-intent/plan.kdl"' "$agent_declaration" +grep -Fqx 'plan "inline-intent" {' "$inline_plan_declaration" +grep -Fqx ' owner "receipt-worker"' "$inline_plan_declaration" +grep -Fqx ' intent "Keep the complete inline intent in plan.kdl."' "$inline_plan_declaration" +cmp -s "$initial_inline_plan_declaration" "$inline_plan_declaration" +cmp -s "$initial_agent_declaration" "$agent_declaration" +if rg -n 'plan-ref|^[[:space:]]+plan[[:space:]]' "$initial_agent_declaration" "$agent_declaration"; then + echo "agent declaration still owns plan truth" >&2 + exit 1 +fi + +if ( + cd "$root/task-repo" + npm test >/dev/null 2>&1 +); then + echo "seed repository unexpectedly passes before implementation" >&2 + exit 1 +fi + +scratch="$(mktemp -d)" +cleanup() { + rm -rf -- "$scratch" +} +trap cleanup EXIT +cp -a "$root/task-repo" "$scratch/candidate" +cp "$root/reference/report.mjs" "$scratch/candidate/src/report.mjs" +"$root/judges/judge-task.sh" "$scratch/candidate" >/dev/null +echo "TASK-CONTRACT-GREEN-43af" + +if rg -n -i 'arm[-_ ]?[ab]|catalog[ -]?plan|direct[ -]?brief' \ + "$root/judges/judge-task.sh" "$root/judges/report.test.mjs"; then + echo "arm-neutral correctness judge contains delivery-arm identity" >&2 + exit 1 +fi +echo "ARM-PARITY-GREEN-43af" + +matrix_output="$(CATALOG="$root" "$root/run-recovery-matrix.sh")" +for marker in \ + COLD-RESTART-TIE-43af \ + LOCAL-PARTITION-TIE-43af \ + INTENT-RECOVERY-TIE-43af \ + HUMAN-STEERING-TIE-43af \ + ACCEPTANCE-EVIDENCE-TIE-43af \ + PLAN-STATIC-EVIDENCE-GREEN-43af \ + DIRECT-MESSAGE-EVIDENCE-GREEN-43af \ + MODEL-FREE-COMPARISON-GREEN-43af \ + PLAIN-FOLDER-NO-CAS-GREEN-43af; do + grep -Fqx "$marker" <<<"$matrix_output" +done +jq -e ' + .outcomes.coldResume.verdict == "tie" and + .outcomes.intentRecovery.verdict == "tie" and + .outcomes.humanSteering.verdict == "tie" and + .outcomes.acceptanceEvidence.verdict == "tie" and + .conclusion == "no-measured-advantage" +' "$root/comparison-receipt.json" >/dev/null +echo "COLD-RESTART-TIE-43af" +echo "LOCAL-PARTITION-TIE-43af" +echo "INTENT-RECOVERY-TIE-43af" +echo "HUMAN-STEERING-TIE-43af" +echo "ACCEPTANCE-EVIDENCE-TIE-43af" +echo "PLAN-STATIC-EVIDENCE-GREEN-43af" +echo "DIRECT-MESSAGE-EVIDENCE-GREEN-43af" +echo "PLAIN-FOLDER-NO-CAS-GREEN-43af" +echo "MODEL-FREE-OUTCOME-GREEN-43af" + +test "$(awk -F '\t' 'NR > 1 && $4 == "model-free-gating" { count++ } END { print count + 0 }' "$measurements")" -eq 3 +test "$(awk -F '\t' 'NR > 1 && $4 == "live-gating" { count++ } END { print count + 0 }' "$measurements")" -eq 2 +test "$(awk -F '\t' 'NR > 1 && $4 == "reporting" { count++ } END { print count + 0 }' "$measurements")" -eq 3 +test "$(awk -F '\t' 'NR > 1 && $4 == "live-reporting" { count++ } END { print count + 0 }' "$measurements")" -eq 6 +awk -F '\t' ' + NR > 1 && ($4 == "model-free-gating" || $4 == "live-gating" || $4 == "reporting") && $5 != "pass" && $1 != "acceptance_evidence" { exit 1 } + NR > 1 && $4 == "live-reporting" && $5 != "lower" { exit 1 } +' "$measurements" +test "$(awk -F '\t' 'NR > 1 && $3 != "yes" { count++ } END { print count + 0 }' "$receipts")" -eq 0 +echo "MEASUREMENT-CONTRACT-GREEN-43af" + +checked=0 +while IFS=$'\t' read -r path expected; do + if test "$path" = "path"; then + test "$expected" = "sha256" + continue + fi + if test "$path" = "task-repo.tree"; then + test "$(tree_hash "$root/task-repo")" = "$expected" + checked=$((checked + 1)) + continue + fi + test -f "$root/$path" + actual="$(sha256sum "$root/$path" | awk '{print $1}')" + test "$actual" = "$expected" + checked=$((checked + 1)) +done <"$root/provenance.tsv" +expected_provenance="$(awk -F '\t' 'NR > 1 { count++ } END { print count + 0 }' "$root/provenance.tsv")" +test "$checked" -eq "$expected_provenance" +echo "PROVENANCE-GREEN-43af" + +test "$(awk -F '\t' 'NR > 1 && $5 == "ready" { count++ } END { print count + 0 }' "$blockers")" -eq 1 +test "$(awk -F '\t' 'NR > 1 && $5 == "blocked" { count++ } END { print count + 0 }' "$blockers")" -eq 1 +grep -Fq '8a76b6e71355140e5b89cd9313fcfd88c82b5cad' "$blockers" +grep -Fq '214e08874720bc546d4adf7d7977e614237baf7989cc09f6932cd991f497a753' "$blockers" +grep -Fq 'hosted run 30835684680 job 91760161352 green' "$blockers" +grep -Fq 'separate authorization' "$blockers" +test "$(find "$root/arm-a" -type f -name '*.kdl' | wc -l)" -eq 6 +echo "PRODUCT-PAIRING-GREEN-43af" + +if rg -n --pcre2 \ + '(^|[;&|][[:space:]]*)(claude|codex|curl|wget|ssh|gh|st2[[:space:]]+(up|eval|ping)|pty[[:space:]]+(run|send)|eval)[[:space:]]' \ + "$root" -g '*.sh' -g '*.kdl'; then + echo "fixture contains a provider, network, mutating product runtime, or nested eval command" >&2 + exit 1 +fi +echo "HERMETIC-SCAFFOLD-GREEN-43af" diff --git a/cells/catalog-plan-vs-direct-brief/fixture/check-plan-contract.sh b/cells/catalog-plan-vs-direct-brief/fixture/check-plan-contract.sh new file mode 100755 index 0000000..cb08fea --- /dev/null +++ b/cells/catalog-plan-vs-direct-brief/fixture/check-plan-contract.sh @@ -0,0 +1,139 @@ +#!/usr/bin/env bash +set -euo pipefail + +root="${CATALOG:?CATALOG must be set}" +runner="$root/runner.tsv" +catalog="$root/arm-a/catalog" +invalid="$root/invalid/current" +st2_path="$(command -v st2)" + +value() { + awk -F '\t' -v key="$1" '$1 == key { print $2 }' "$runner" +} + +source_short="$(value source_short)" +expected_sha="$(value binary_sha256)" +version_regex="^st2 0\\.1\\.0 — running from local source \\(${source_short}, .+ ago\\)$" +actual_version="$("$st2_path" --version)" +actual_sha="$(sha256sum "$st2_path" | awk '{print $1}')" +[[ "$actual_version" =~ $version_regex ]] +test "$actual_sha" = "$expected_sha" +echo "PLAN-RUNNER-PIN-GREEN-43af" + +before="$(mktemp)" +after="$(mktemp)" +invalid_json="$(mktemp)" +cleanup() { + rm -f -- "$before" "$after" "$invalid_json" +} +trap cleanup EXIT + +find "$catalog" -type f -print0 | + LC_ALL=C sort -z | + xargs -0 sha256sum >"$before" + +validate_json="$("$st2_path" plan validate --catalog "$catalog" --json)" +jq -e ' + .result == "valid" and + .plans == 2 and + .errors == 0 +' <<<"$validate_json" >/dev/null +echo "PLAN-VALIDATE-GREEN-43af" + +list_json="$("$st2_path" plan list --catalog "$catalog" --json)" +jq -e ' + length == 2 and + .[0] == { + "identity": "inline-intent", + "owner": "receipt-worker", + "frontier": ["0000"] + } and + .[1] == { + "identity": "receipt-report", + "owner": "receipt-worker", + "frontier": ["0001"] + } +' <<<"$list_json" >/dev/null +echo "PLAN-LIST-GREEN-43af" + +show_json="$("$st2_path" plan show receipt-report --catalog "$catalog" --json)" +jq -e ' + .identity == "receipt-report" and + .owner == "receipt-worker" and + .frontier == ["0001"] and + (.versions | length) == 2 and + .versions[0] == { + "identity": "0000", + "parents": [], + "why": null, + "content": "file:versions/0000.md" + } and + .versions[1] == { + "identity": "0001", + "parents": ["0000"], + "why": "Human steering tightens input validation without changing scope.", + "content": "file:versions/0001.md" + } and + (has("source") | not) and + (.versions[0] | has("resolvedContent") | not) +' <<<"$show_json" >/dev/null +echo "PLAN-SHOW-GREEN-43af" + +inspect_json="$("$st2_path" plan inspect receipt-report --catalog "$catalog" --json)" +jq -e ' + .identity == "receipt-report" and + .owner == "receipt-worker" and + .sourceKind == "external" and + .referencedBy == ["receipt-worker"] and + .frontier == ["0001"] and + (.source | endswith("/agents/eval/receipt-worker/plans/receipt-report/plan.kdl")) and + (.versions[0].resolvedContent | endswith("/agents/eval/receipt-worker/plans/receipt-report/versions/0000.md")) and + (.versions[1].resolvedContent | endswith("/agents/eval/receipt-worker/plans/receipt-report/versions/0001.md")) +' <<<"$inspect_json" >/dev/null + +inline_show_json="$("$st2_path" plan show inline-intent --catalog "$catalog" --json)" +jq -e ' + .identity == "inline-intent" and + .owner == "receipt-worker" and + .frontier == ["0000"] and + .versions == [{ + "identity": "0000", + "parents": [], + "why": null, + "intent": "Keep the complete inline intent in plan.kdl." + }] and + (has("source") | not) and + (.versions[0] | has("content") | not) and + (.versions[0] | has("resolvedContent") | not) +' <<<"$inline_show_json" >/dev/null + +inline_inspect_json="$("$st2_path" plan inspect inline-intent --catalog "$catalog" --json)" +jq -e ' + .identity == "inline-intent" and + .sourceKind == "external" and + .referencedBy == ["receipt-worker"] and + (.source | endswith("/agents/eval/receipt-worker/plans/inline-intent/plan.kdl")) and + .versions[0].intent == "Keep the complete inline intent in plan.kdl." and + (.versions[0] | has("content") | not) and + (.versions[0] | has("resolvedContent") | not) +' <<<"$inline_inspect_json" >/dev/null +echo "PLAN-TARGET-FORMS-GREEN-43af" + +find "$catalog" -type f -print0 | + LC_ALL=C sort -z | + xargs -0 sha256sum >"$after" +cmp -s "$before" "$after" +test ! -e "$catalog/.st2" +echo "PLAN-INSPECT-READONLY-GREEN-43af" + +if "$st2_path" plan validate --catalog "$invalid" --json >"$invalid_json" 2>/dev/null; then + echo "unsupported current pointer unexpectedly validated" >&2 + exit 1 +fi +jq -e ' + .result == "invalid" and + .code == "unsupported-plan-field" and + (.error | contains("unsupported-current")) and + (.error | contains("current")) +' "$invalid_json" >/dev/null +echo "PLAN-BOUNDARY-GREEN-43af" diff --git a/cells/catalog-plan-vs-direct-brief/fixture/experiment.tsv b/cells/catalog-plan-vs-direct-brief/fixture/experiment.tsv new file mode 100644 index 0000000..e5eff76 --- /dev/null +++ b/cells/catalog-plan-vs-direct-brief/fixture/experiment.tsv @@ -0,0 +1,20 @@ +field value +experiment_id catalog-plan-vs-direct-brief-v1 +source_gist https://gist.github.com/myobie/d5ecfac24cd3965e095a5031cd2e00cb/5c1d1427c0556d95d13890e5c5086cd85b25d994 +source_gist_revision 5c1d1427c0556d95d13890e5c5086cd85b25d994 +st2_plan_pr https://github.com/compoundingtech/st2/pull/115 +st2_plan_source 8a76b6e71355140e5b89cd9313fcfd88c82b5cad +st2_plan_binary_sha256 214e08874720bc546d4adf7d7977e614237baf7989cc09f6932cd991f497a753 +task_id receipt-report-latest-pass +seed_repository task-repo +allowed_change src/report.mjs +allowed_tools node,bash,git +network_policy disabled +wall_budget_seconds 1200 +model_token_budget 40000 +judge_profile arm-neutral-public-plus-held-out +done_condition public and held-out tests pass; only src/report.mjs differs from the seed; no dependency or network change +initial_revision 0000 +steering_revision 0001 +model_free_result no-measured-advantage-for-cold-resume-intent-recovery-or-acceptance-evidence +live_run_status blocked-pending-separate-provider-authorization diff --git a/cells/catalog-plan-vs-direct-brief/fixture/invalid/current/0000.md b/cells/catalog-plan-vs-direct-brief/fixture/invalid/current/0000.md new file mode 100644 index 0000000..ef57ad9 --- /dev/null +++ b/cells/catalog-plan-vs-direct-brief/fixture/invalid/current/0000.md @@ -0,0 +1 @@ +# Unsupported mutable current pointer diff --git a/cells/catalog-plan-vs-direct-brief/fixture/invalid/current/plan.kdl b/cells/catalog-plan-vs-direct-brief/fixture/invalid/current/plan.kdl new file mode 100644 index 0000000..999c07d --- /dev/null +++ b/cells/catalog-plan-vs-direct-brief/fixture/invalid/current/plan.kdl @@ -0,0 +1,5 @@ +plan "unsupported-current" { + owner "receipt-worker" + current "0000" + version "0000" content="file:0000.md" +} diff --git a/cells/catalog-plan-vs-direct-brief/fixture/judges/judge-task.sh b/cells/catalog-plan-vs-direct-brief/fixture/judges/judge-task.sh new file mode 100755 index 0000000..fbc47dc --- /dev/null +++ b/cells/catalog-plan-vs-direct-brief/fixture/judges/judge-task.sh @@ -0,0 +1,34 @@ +#!/usr/bin/env bash +set -euo pipefail + +fixture_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +candidate="${1:?usage: judge-task.sh CANDIDATE_REPOSITORY}" +seed="$fixture_root/task-repo" + +test -f "$candidate/src/report.mjs" + +mapfile -t seed_files < <( + cd "$seed" + find . -type f -print | LC_ALL=C sort +) +mapfile -t candidate_files < <( + cd "$candidate" + find . -type f ! -path './.git/*' -print | LC_ALL=C sort +) +test "${seed_files[*]}" = "${candidate_files[*]}" + +while IFS= read -r relative; do + case "$relative" in + ./src/report.mjs) continue ;; + esac + cmp -s "$seed/${relative#./}" "$candidate/${relative#./}" +done < <(printf '%s\n' "${seed_files[@]}") + +( + cd "$candidate" + npm test +) +CANDIDATE_REPORT="$candidate/src/report.mjs" \ + node --test "$fixture_root/judges/report.test.mjs" + +echo "ARM-NEUTRAL-TASK-JUDGE-GREEN-43af" diff --git a/cells/catalog-plan-vs-direct-brief/fixture/judges/report.test.mjs b/cells/catalog-plan-vs-direct-brief/fixture/judges/report.test.mjs new file mode 100644 index 0000000..c4abd76 --- /dev/null +++ b/cells/catalog-plan-vs-direct-brief/fixture/judges/report.test.mjs @@ -0,0 +1,111 @@ +import assert from "node:assert/strict"; +import test from "node:test"; +import { pathToFileURL } from "node:url"; + +const candidate = process.env.CANDIDATE_REPORT; +if (!candidate) { + throw new Error("CANDIDATE_REPORT is required"); +} + +const { parseJsonLines, summarize } = await import(pathToFileURL(candidate)); + +test("retains complete records without mutating caller input", () => { + const records = [ + { run_id: "r1", cell: "zeta", result: "PASS", usage: { input: 10 } }, + { run_id: "r2", cell: "alpha", result: "FAIL", note: "first" }, + { run_id: "r3", cell: "zeta", result: "FAIL", usage: { input: 11 } }, + { run_id: "r4", cell: "alpha", result: "PASS", note: "accepted" }, + ]; + const original = structuredClone(records); + + assert.deepEqual(summarize(records), [ + { + cell: "alpha", + accepted_pass: { + run_id: "r4", + cell: "alpha", + result: "PASS", + note: "accepted", + }, + last_run: { + run_id: "r4", + cell: "alpha", + result: "PASS", + note: "accepted", + }, + }, + { + cell: "zeta", + accepted_pass: { + run_id: "r1", + cell: "zeta", + result: "PASS", + usage: { input: 10 }, + }, + last_run: { + run_id: "r3", + cell: "zeta", + result: "FAIL", + usage: { input: 11 }, + }, + }, + ]); + assert.deepEqual(records, original); +}); + +test("rejects duplicate run ids in parsed and direct records", () => { + const duplicateText = [ + '{"run_id":"same","cell":"alpha","result":"PASS"}', + '{"run_id":"same","cell":"beta","result":"FAIL"}', + ].join("\n"); + + assert.throws( + () => parseJsonLines(duplicateText), + (error) => + error instanceof TypeError && + /duplicate/.test(error.message) && + /same/.test(error.message), + ); + assert.throws( + () => + summarize([ + { run_id: "same", cell: "alpha", result: "PASS" }, + { run_id: "same", cell: "alpha", result: "FAIL" }, + ]), + (error) => + error instanceof TypeError && + /duplicate/.test(error.message) && + /same/.test(error.message), + ); +}); + +test("rejects unknown results and identifies the run", () => { + assert.throws( + () => + parseJsonLines( + '{"run_id":"mystery-7","cell":"alpha","result":"CANCELLED"}', + ), + (error) => + error instanceof TypeError && + /unknown result/.test(error.message) && + /mystery-7/.test(error.message), + ); +}); + +test("reports malformed and structurally invalid input lines", () => { + assert.throws( + () => parseJsonLines('\n{"run_id":"ok","cell":"alpha","result":"PASS"}\n{'), + (error) => error instanceof TypeError && /line 3/.test(error.message), + ); + assert.throws( + () => parseJsonLines('["not","an","object"]'), + (error) => error instanceof TypeError && /line 1/.test(error.message), + ); + assert.throws( + () => parseJsonLines('{"run_id":"","cell":"alpha","result":"PASS"}'), + (error) => + error instanceof TypeError && + /line 1/.test(error.message) && + /run_id/.test(error.message), + ); +}); diff --git a/cells/catalog-plan-vs-direct-brief/fixture/measurement-schema.tsv b/cells/catalog-plan-vs-direct-brief/fixture/measurement-schema.tsv new file mode 100644 index 0000000..eb33629 --- /dev/null +++ b/cells/catalog-plan-vs-direct-brief/fixture/measurement-schema.tsv @@ -0,0 +1,15 @@ +metric type unit role preferred_direction model_free_arm_a model_free_arm_b model_free_verdict live_source +correctness boolean pass/fail live-gating pass not-run not-run unresolved-live arm-neutral held-out judge +cold_restart_recovery boolean pass/fail model-free-gating pass pass pass tie comparison receipt +partition_recovery boolean pass/fail model-free-gating pass pass pass tie comparison receipt +human_steering boolean pass/fail model-free-gating pass pass pass tie comparison receipt +acceptance_evidence enum evaluator-only|product-native live-gating product-native evaluator-only evaluator-only tie comparison receipt plus future worker claim +static_intent_validation boolean pass/fail reporting pass pass n/a plan-only st2 plan validate +lineage_observation boolean pass/fail reporting pass pass pass tie st2 plan show and st2 message inReplyTo +provenance_resolution boolean pass/fail reporting pass pass n/a plan-only st2 plan inspect +coordination_messages integer count live-reporting lower unmeasured unmeasured unresolved-live st2 message receipt +coordination_bytes integer bytes live-reporting lower unmeasured unmeasured unresolved-live st2 message and sync receipt +model_input_tokens integer tokens live-reporting lower 0 0 model-free-only harness usage receipt +model_output_tokens integer tokens live-reporting lower 0 0 model-free-only harness usage receipt +cost_usd decimal USD live-reporting lower 0 0 model-free-only harness usage receipt +wall_duration decimal seconds live-reporting lower unmeasured unmeasured unresolved-live evaluator run receipt diff --git a/cells/catalog-plan-vs-direct-brief/fixture/provenance.tsv b/cells/catalog-plan-vs-direct-brief/fixture/provenance.tsv new file mode 100644 index 0000000..0a0466c --- /dev/null +++ b/cells/catalog-plan-vs-direct-brief/fixture/provenance.tsv @@ -0,0 +1,31 @@ +path sha256 +experiment.tsv 7eb1c81bb4aa26c37aed62488602fc8ddda83fe946301832682252c440bc605e +arms.tsv bd7c41b75fc29920b5ba1d1298649f5b94e65cf85ed9b8192def51e22357e7b2 +scenario.tsv 884a9c83b983128c8e642bf3a945d6f7edb544ac224f5934a4468f17a48d282b +measurement-schema.tsv 01763a0c090b9c2fd39e5f1b7995380e2dc0d2e5d4230227749ddd41a133f9ab +receipt-schema.tsv c996eaee81301640442b04d9fca31937bb840ed10ef78cca035e5990d1fe6441 +blockers.tsv 140a4fb38bae58ef5dbace2a10574000808d3d1fd5b8e609b829850dc5fb49ba +runner.tsv 8114588d818d2306d944785953447cb6114bdb83e0e72c6ebf3bc060016b19b8 +arm-a/catalog-initial/agents/eval/receipt-worker/plans/receipt-report/plan.kdl 08dca1863eadce1d1616d29d1f01de21435b07015d36066f42e5dbe8d68edbd1 +arm-a/catalog-initial/agents/eval/receipt-worker/plans/receipt-report/versions/0000.md 4f81d370fe6c09c5c199046cc948f68ebdc8a510a4c30fde8edc6993d1abef11 +arm-a/catalog-initial/agents/eval/receipt-worker/plans/inline-intent/plan.kdl c1cf3ed40cc909edb8bcb69418dcbe21daf2ed074be34638e632cc31c4f0ae23 +arm-a/catalog-initial/agents/eval/receipt-worker/agent.kdl e347308269970ab6abde6c1f99a96e0619783f55081ec348d9107c10a10a4e92 +arm-a/catalog/agents/eval/receipt-worker/plans/receipt-report/plan.kdl 9781d56aceee726cf26c639b195d100fb66229b0855fa9ba5ce2f1e56024e9ba +arm-a/catalog/agents/eval/receipt-worker/plans/receipt-report/versions/0000.md 4f81d370fe6c09c5c199046cc948f68ebdc8a510a4c30fde8edc6993d1abef11 +arm-a/catalog/agents/eval/receipt-worker/plans/receipt-report/versions/0001.md aa3075e6fcbec4648209f77cd0b3b9866d70f4a354bb16554d3e58babd7f04b4 +arm-a/catalog/agents/eval/receipt-worker/plans/inline-intent/plan.kdl c1cf3ed40cc909edb8bcb69418dcbe21daf2ed074be34638e632cc31c4f0ae23 +arm-a/catalog/agents/eval/receipt-worker/agent.kdl e347308269970ab6abde6c1f99a96e0619783f55081ec348d9107c10a10a4e92 +arm-b/inbox/brief-0000.md 4f81d370fe6c09c5c199046cc948f68ebdc8a510a4c30fde8edc6993d1abef11 +arm-b/inbox/brief-0001.md aa3075e6fcbec4648209f77cd0b3b9866d70f4a354bb16554d3e58babd7f04b4 +invalid/current/plan.kdl a7d5792410bcd4f6ba073a1fecaee536adec7a40f82298c2d3022a75f9a0b687 +invalid/current/0000.md 1a18971d45c564ee22de2496c1a086b6a799b9cf809a6d7351236d3c5b4e788a +task-repo/package.json 8f756153985ab193ec21992e4adf874db4b02330db019006c5214550c2ff9618 +task-repo/README.md c50a603eac8bb3dad994dfc96f682f7f8d9a56300e21a7f7be0d040308ff580b +task-repo/src/report.mjs e2bc26ea2e19e14290585388c4f7d2cc10f56294cb4547068af884847197f020 +task-repo/test/report.test.mjs e9008004b55687df44025f66abe2e5ff739de8d8c7bec27d8d2815defd3a3a31 +reference/report.mjs 7343d91050754693f1ced37a97dae6f8701549af27ee78e08bab260a5d6550c9 +judges/report.test.mjs 24f41a2e596c940a8a3c5123e53b635bc66deb11d48bc0911a280d7a5485e670 +judges/judge-task.sh 23680852e5d3ffd744eb00b6edf12c85f5560cbe3b406da45169e710eff2a10b +run-recovery-matrix.sh 29aab23eeca0c1af1f76c6745da11c37da4e8a1800592a8a53df4eb963941533 +check-plan-contract.sh 474a5b9ffa62d899ed81f4867f245202e6c161593d2e7cec5c2bcdfb969f5335 +task-repo.tree 22c368e3d2b9722c0fe53710dbd16e2010e98d7215db0aa42fe7532cfa9233c5 diff --git a/cells/catalog-plan-vs-direct-brief/fixture/receipt-schema.tsv b/cells/catalog-plan-vs-direct-brief/fixture/receipt-schema.tsv new file mode 100644 index 0000000..63d7983 --- /dev/null +++ b/cells/catalog-plan-vs-direct-brief/fixture/receipt-schema.tsv @@ -0,0 +1,25 @@ +field type required description +schemaVersion integer yes comparison receipt schema version +experimentId string yes frozen experiment identity +runner.source git-sha yes exact experimental st2 source +runner.sha256 sha256 yes exact experimental st2 executable +modelCalls nonnegative-integer yes provider/model calls made by this run +remoteSourcesOfflineDuringRecovery boolean yes both recoveries occurred with remote sources unavailable +storage.plainFoldersSufficient boolean yes no service or content-addressed store is required +storage.casRequired boolean yes whether content-addressed storage is required +arms.A.transport enum:catalog-plan yes arm A durable transport +arms.B.transport enum:st2-direct-messages yes arm B durable transport +arms.A.initialIntentSha256 sha256 yes recovered initial intent hash +arms.B.initialIntentSha256 sha256 yes recovered initial intent hash +arms.A.steeredIntentSha256 sha256 yes recovered revised intent hash +arms.B.steeredIntentSha256 sha256 yes recovered revised intent hash +outcomes.coldResume.verdict enum:tie|arm-a|arm-b yes comparative cold-resume result +outcomes.intentRecovery.verdict enum:tie|arm-a|arm-b yes comparative exact-intent result +outcomes.humanSteering.verdict enum:tie|arm-a|arm-b yes comparative steering result +outcomes.acceptanceEvidence.verdict enum:tie|arm-a|arm-b yes comparative worker-acceptance evidence result +planOnly.staticIntentValidation enum:pass|fail yes read-only plan schema and graph validation +planOnly.provenanceResolution enum:pass|fail yes read-only resolved resource provenance +directOnly.messageDelivery enum:pass|fail yes durable direct-message delivery +directOnly.replyLineage enum:pass|fail yes direct-message inReplyTo lineage +liveUnresolved array yes endpoints requiring separately authorized model runs +conclusion enum:no-measured-advantage|arm-a-advantage|arm-b-advantage yes model-free comparative conclusion diff --git a/cells/catalog-plan-vs-direct-brief/fixture/reference/report.mjs b/cells/catalog-plan-vs-direct-brief/fixture/reference/report.mjs new file mode 100644 index 0000000..6b8f744 --- /dev/null +++ b/cells/catalog-plan-vs-direct-brief/fixture/reference/report.mjs @@ -0,0 +1,76 @@ +function validateRecord(record, context) { + if (record === null || typeof record !== "object" || Array.isArray(record)) { + throw new TypeError(`${context}: expected an object record`); + } + + for (const field of ["run_id", "cell", "result"]) { + if (typeof record[field] !== "string" || record[field].length === 0) { + throw new TypeError(`${context}: missing non-empty ${field}`); + } + } + + if (record.result !== "PASS" && record.result !== "FAIL") { + throw new TypeError( + `${context}: unknown result ${record.result} for ${record.run_id}`, + ); + } +} + +function validateUnique(records, contextForIndex) { + const seen = new Set(); + records.forEach((record, index) => { + const context = contextForIndex(index); + validateRecord(record, context); + if (seen.has(record.run_id)) { + throw new TypeError(`${context}: duplicate run_id ${record.run_id}`); + } + seen.add(record.run_id); + }); +} + +export function parseJsonLines(text) { + const records = []; + const sourceLines = text.split("\n"); + + sourceLines.forEach((line, index) => { + if (line.trim().length === 0) { + return; + } + + let record; + try { + record = JSON.parse(line); + } catch { + throw new TypeError(`line ${index + 1}: malformed JSON`); + } + records.push({ record, lineNumber: index + 1 }); + }); + + validateUnique( + records.map(({ record }) => record), + (index) => `line ${records[index].lineNumber}`, + ); + return records.map(({ record }) => record); +} + +export function summarize(records) { + validateUnique(records, (index) => `record ${index + 1}`); + + const byCell = new Map(); + for (const record of records) { + const current = byCell.get(record.cell) ?? { + cell: record.cell, + accepted_pass: null, + last_run: null, + }; + if (record.result === "PASS") { + current.accepted_pass = { ...record }; + } + current.last_run = { ...record }; + byCell.set(record.cell, current); + } + + return [...byCell.values()].sort((left, right) => + left.cell.localeCompare(right.cell), + ); +} diff --git a/cells/catalog-plan-vs-direct-brief/fixture/run-recovery-matrix.sh b/cells/catalog-plan-vs-direct-brief/fixture/run-recovery-matrix.sh new file mode 100755 index 0000000..fc84238 --- /dev/null +++ b/cells/catalog-plan-vs-direct-brief/fixture/run-recovery-matrix.sh @@ -0,0 +1,226 @@ +#!/usr/bin/env bash +set -euo pipefail + +root="${CATALOG:?CATALOG must be set}" +runner="$root/runner.tsv" +receipt="$root/comparison-receipt.json" +st2_path="$(command -v st2)" +scratch="$(mktemp -d)" + +cleanup() { + rm -rf -- "$scratch" +} +trap cleanup EXIT + +runner_value() { + awk -F '\t' -v key="$1" '$1 == key { print $2 }' "$runner" +} + +hash_file() { + sha256sum "$1" | awk '{print $1}' +} + +assert_no_acceptance_claim() { + jq -e ' + (has("accepted") | not) and + (has("acceptance") | not) and + (has("completed") | not) and + (has("receipt") | not) and + (has("receipts") | not) + ' >/dev/null +} + +source_full="$(runner_value source_full)" +source_short="$(runner_value source_short)" +expected_binary_sha="$(runner_value binary_sha256)" +version_regex="^st2 0\\.1\\.0 — running from local source \\(${source_short}, .+ ago\\)$" +actual_version="$($st2_path --version)" +actual_binary_sha="$(hash_file "$st2_path")" +[[ "$actual_version" =~ $version_regex ]] +test "$actual_binary_sha" = "$expected_binary_sha" + +remote="$scratch/remote" +local_store="$scratch/local" +session="$scratch/session" +plan_local="$local_store/plan-catalog" +direct_bus="$local_store/direct-bus" +mkdir -p "$remote" "$local_store" "$session" "$direct_bus" +cp -a "$root/arm-a/catalog-initial" "$remote/plan-initial" +cp -a "$root/arm-a/catalog" "$remote/plan-steered" +cp -a "$root/arm-b/inbox" "$remote/direct-source" + +# Initial delivery: sync only the initial plan snapshot, and deliver only the +# initial direct brief through the real st2 message store. +cp -a "$remote/plan-initial" "$plan_local" +initial_message="$($st2_path message send eval.worker \ + --catalog "$direct_bus" \ + --as eval.supervisor \ + --subject 'intent revision 0000' \ + <"$remote/direct-source/brief-0000.md")" +test -n "$initial_message" + +# Cold restart and partition: volatile state disappears and both remote input +# sources go offline. Recovery must use only each arm's local durable store. +printf 'volatile\n' >"$session/state" +rm -rf -- "$session" +mv "$remote" "$remote.offline" +test ! -e "$remote" + +plan_initial_show="$($st2_path plan show receipt-report --catalog "$plan_local" --json)" +plan_initial_inspect="$($st2_path plan inspect receipt-report --catalog "$plan_local" --json)" +jq -e ' + .frontier == ["0000"] and + (.versions | length) == 1 and + .versions[0].identity == "0000" +' <<<"$plan_initial_show" >/dev/null +plan_initial_path="$(jq -r '.versions[0].resolvedContent' <<<"$plan_initial_inspect")" +plan_initial_sha="$(hash_file "$plan_initial_path")" +test "$plan_initial_sha" = "$(hash_file "$root/arm-a/catalog-initial/agents/eval/receipt-worker/plans/receipt-report/versions/0000.md")" +assert_no_acceptance_claim <<<"$plan_initial_show" +assert_no_acceptance_claim <<<"$plan_initial_inspect" + +direct_initial_list="$($st2_path message ls eval.worker --catalog "$direct_bus" --json)" +jq -e --arg filename "$initial_message" ' + length == 1 and + .[0].filename == $filename and + .[0].from == "eval.supervisor" and + .[0].subject == "intent revision 0000" and + .[0].inReplyTo == null +' <<<"$direct_initial_list" >/dev/null +direct_initial_read="$($st2_path message read eval.worker "$initial_message" --catalog "$direct_bus" --json)" +jq -rj '.body' <<<"$direct_initial_read" >"$scratch/direct-initial.md" +direct_initial_sha="$(hash_file "$scratch/direct-initial.md")" +test "$direct_initial_sha" = "$(hash_file "$root/arm-b/inbox/brief-0000.md")" +assert_no_acceptance_claim <<<"$direct_initial_read" +echo "COLD-RESTART-TIE-43af" +echo "LOCAL-PARTITION-TIE-43af" + +# Human steering: restore the remote source, sync a complete new plan snapshot, +# and deliver a reply-linked direct brief. Then repeat cold recovery offline. +mv "$remote.offline" "$remote" +rm -rf -- "$plan_local" +cp -a "$remote/plan-steered" "$plan_local" +steered_message="$($st2_path message send eval.worker \ + --catalog "$direct_bus" \ + --as eval.supervisor \ + --subject 'intent revision 0001' \ + --in-reply-to "$initial_message" \ + <"$remote/direct-source/brief-0001.md")" +test -n "$steered_message" + +mkdir -p "$session" +printf 'volatile\n' >"$session/state" +rm -rf -- "$session" +mv "$remote" "$remote.offline" +test ! -e "$remote" + +plan_steered_show="$($st2_path plan show receipt-report --catalog "$plan_local" --json)" +plan_steered_inspect="$($st2_path plan inspect receipt-report --catalog "$plan_local" --json)" +jq -e ' + .frontier == ["0001"] and + (.versions | length) == 2 and + .versions[0].identity == "0000" and + .versions[1].identity == "0001" and + .versions[1].parents == ["0000"] +' <<<"$plan_steered_show" >/dev/null +plan_steered_path="$(jq -r '.versions[] | select(.identity == "0001") | .resolvedContent' <<<"$plan_steered_inspect")" +plan_steered_sha="$(hash_file "$plan_steered_path")" +test "$plan_steered_sha" = "$(hash_file "$root/arm-a/catalog/agents/eval/receipt-worker/plans/receipt-report/versions/0001.md")" +assert_no_acceptance_claim <<<"$plan_steered_show" +assert_no_acceptance_claim <<<"$plan_steered_inspect" + +direct_steered_list="$($st2_path message ls eval.worker --catalog "$direct_bus" --json)" +jq -e --arg initial "$initial_message" --arg steered "$steered_message" ' + length == 2 and + any(.[]; .filename == $initial and .inReplyTo == null) and + any(.[]; .filename == $steered and .inReplyTo == $initial and .subject == "intent revision 0001") +' <<<"$direct_steered_list" >/dev/null +direct_steered_read="$($st2_path message read eval.worker "$steered_message" --catalog "$direct_bus" --json)" +jq -rj '.body' <<<"$direct_steered_read" >"$scratch/direct-steered.md" +direct_steered_sha="$(hash_file "$scratch/direct-steered.md")" +test "$direct_steered_sha" = "$(hash_file "$root/arm-b/inbox/brief-0001.md")" +test "$(jq -r '.inReplyTo' <<<"$direct_steered_read")" = "$initial_message" +assert_no_acceptance_claim <<<"$direct_steered_read" + +test "$plan_initial_sha" = "$direct_initial_sha" +test "$plan_steered_sha" = "$direct_steered_sha" +echo "INTENT-RECOVERY-TIE-43af" +echo "HUMAN-STEERING-TIE-43af" +echo "ACCEPTANCE-EVIDENCE-TIE-43af" +echo "PLAN-STATIC-EVIDENCE-GREEN-43af" +echo "DIRECT-MESSAGE-EVIDENCE-GREEN-43af" + +jq -n \ + --arg experiment_id "catalog-plan-vs-direct-brief-v1" \ + --arg source "$source_full" \ + --arg sha "$actual_binary_sha" \ + --arg initial_sha "$plan_initial_sha" \ + --arg steered_sha "$plan_steered_sha" \ + '{ + schemaVersion: 1, + experimentId: $experiment_id, + runner: {source: $source, sha256: $sha}, + modelCalls: 0, + remoteSourcesOfflineDuringRecovery: true, + storage: {plainFoldersSufficient: true, casRequired: false}, + arms: { + A: { + transport: "catalog-plan", + initialIntentSha256: $initial_sha, + steeredIntentSha256: $steered_sha + }, + B: { + transport: "st2-direct-messages", + initialIntentSha256: $initial_sha, + steeredIntentSha256: $steered_sha + } + }, + outcomes: { + coldResume: {A: "pass", B: "pass", verdict: "tie"}, + intentRecovery: {A: "pass", B: "pass", verdict: "tie"}, + humanSteering: {A: "pass", B: "pass", verdict: "tie"}, + acceptanceEvidence: { + A: "evaluator-only", + B: "evaluator-only", + verdict: "tie", + reason: "Neither read-only plan inspection nor direct-message delivery reports worker acceptance." + } + }, + planOnly: { + staticIntentValidation: "pass", + provenanceResolution: "pass" + }, + directOnly: { + messageDelivery: "pass", + replyLineage: "pass" + }, + liveUnresolved: [ + "correctness", + "coordination_messages", + "coordination_bytes", + "model_input_tokens", + "model_output_tokens", + "cost_usd", + "wall_duration" + ], + conclusion: "no-measured-advantage" + }' >"$receipt" + +jq -e ' + .modelCalls == 0 and + .remoteSourcesOfflineDuringRecovery == true and + .storage.plainFoldersSufficient == true and + .storage.casRequired == false and + .outcomes.coldResume.verdict == "tie" and + .outcomes.intentRecovery.verdict == "tie" and + .outcomes.humanSteering.verdict == "tie" and + .outcomes.acceptanceEvidence.verdict == "tie" and + .planOnly.staticIntentValidation == "pass" and + .planOnly.provenanceResolution == "pass" and + .directOnly.messageDelivery == "pass" and + .directOnly.replyLineage == "pass" and + .conclusion == "no-measured-advantage" +' "$receipt" >/dev/null + +echo "MODEL-FREE-COMPARISON-GREEN-43af" +echo "PLAIN-FOLDER-NO-CAS-GREEN-43af" diff --git a/cells/catalog-plan-vs-direct-brief/fixture/runner.tsv b/cells/catalog-plan-vs-direct-brief/fixture/runner.tsv new file mode 100644 index 0000000..2a807f5 --- /dev/null +++ b/cells/catalog-plan-vs-direct-brief/fixture/runner.tsv @@ -0,0 +1,9 @@ +field value +st2_pr https://github.com/compoundingtech/st2/pull/115 +source_full 8a76b6e71355140e5b89cd9313fcfd88c82b5cad +source_short 8a76b6e +binary_sha256 214e08874720bc546d4adf7d7977e614237baf7989cc09f6932cd991f497a753 +source_gist_revision 5c1d1427c0556d95d13890e5c5086cd85b25d994 +runtime_scope read-only-validate-list-show-inspect +hosted_run https://github.com/compoundingtech/st2/actions/runs/30835684680 +hosted_status pass diff --git a/cells/catalog-plan-vs-direct-brief/fixture/scenario.tsv b/cells/catalog-plan-vs-direct-brief/fixture/scenario.tsv new file mode 100644 index 0000000..221af70 --- /dev/null +++ b/cells/catalog-plan-vs-direct-brief/fixture/scenario.tsv @@ -0,0 +1,7 @@ +ordinal event arm_a_input arm_b_input invariant +1 initial_delivery catalog snapshot with version 0000 st2 message carrying brief 0000 byte-identical intent; neither delivery claims worker acceptance +2 cold_restart st2 plan show on local snapshot st2 message read from local inbox no volatile conversation or process state +3 simulated_partition local catalog only local message inbox only remote input source unavailable during recovery +4 human_steering catalog snapshot adds version 0001 second st2 message replies to brief 0000 byte-identical complete revised intent +5 restart_after_steering frontier 0001 with version 0000 retained brief 0001 with inReplyTo 0000 retained revision 0001 recovered and lineage remains inspectable +6 model_free_verdict static validation and provenance only message delivery and reply lineage only cold resume, intent recovery, and acceptance evidence tie; correctness, traffic, cost, and duration remain live-run endpoints diff --git a/cells/catalog-plan-vs-direct-brief/fixture/task-repo/README.md b/cells/catalog-plan-vs-direct-brief/fixture/task-repo/README.md new file mode 100644 index 0000000..093850f --- /dev/null +++ b/cells/catalog-plan-vs-direct-brief/fixture/task-repo/README.md @@ -0,0 +1,11 @@ +# receipt-report + +A dependency-free report core for append-only eval run receipts. + +Run the visible test suite with: + +```sh +npm test +``` + +The task brief is delivered separately. Only `src/report.mjs` is in scope. diff --git a/cells/catalog-plan-vs-direct-brief/fixture/task-repo/package.json b/cells/catalog-plan-vs-direct-brief/fixture/task-repo/package.json new file mode 100644 index 0000000..a34c746 --- /dev/null +++ b/cells/catalog-plan-vs-direct-brief/fixture/task-repo/package.json @@ -0,0 +1,9 @@ +{ + "name": "receipt-report", + "version": "0.0.0", + "private": true, + "type": "module", + "scripts": { + "test": "node --test" + } +} diff --git a/cells/catalog-plan-vs-direct-brief/fixture/task-repo/src/report.mjs b/cells/catalog-plan-vs-direct-brief/fixture/task-repo/src/report.mjs new file mode 100644 index 0000000..7b5ae6e --- /dev/null +++ b/cells/catalog-plan-vs-direct-brief/fixture/task-repo/src/report.mjs @@ -0,0 +1,13 @@ +export function parseJsonLines(text) { + return text + .split("\n") + .filter(Boolean) + .map((line) => JSON.parse(line)); +} + +export function summarize(records) { + return records.map((record) => ({ + cell: record.cell, + last_run: record.result, + })); +} diff --git a/cells/catalog-plan-vs-direct-brief/fixture/task-repo/test/report.test.mjs b/cells/catalog-plan-vs-direct-brief/fixture/task-repo/test/report.test.mjs new file mode 100644 index 0000000..f3cacc1 --- /dev/null +++ b/cells/catalog-plan-vs-direct-brief/fixture/task-repo/test/report.test.mjs @@ -0,0 +1,54 @@ +import assert from "node:assert/strict"; +import test from "node:test"; + +import { parseJsonLines, summarize } from "../src/report.mjs"; + +test("reports the latest accepted pass separately from the last run", () => { + const records = parseJsonLines(` +{"run_id":"r1","cell":"alpha","result":"PASS","score":"2/2"} +{"run_id":"r2","cell":"alpha","result":"FAIL","score":"1/2"} +{"run_id":"r3","cell":"beta","result":"FAIL","score":"0/1"} +`); + + assert.deepEqual(summarize(records), [ + { + cell: "alpha", + accepted_pass: { + run_id: "r1", + cell: "alpha", + result: "PASS", + score: "2/2", + }, + last_run: { + run_id: "r2", + cell: "alpha", + result: "FAIL", + score: "1/2", + }, + }, + { + cell: "beta", + accepted_pass: null, + last_run: { + run_id: "r3", + cell: "beta", + result: "FAIL", + score: "0/1", + }, + }, + ]); +}); + +test("ignores blank lines and sorts cells", () => { + const records = parseJsonLines(` + +{"run_id":"r1","cell":"zeta","result":"PASS"} + +{"run_id":"r2","cell":"alpha","result":"PASS"} +`); + + assert.deepEqual( + summarize(records).map(({ cell }) => cell), + ["alpha", "zeta"], + ); +}); diff --git a/evidence/harness-exclusions.tsv b/evidence/harness-exclusions.tsv index 020ab44..f417fd9 100644 --- a/evidence/harness-exclusions.tsv +++ b/evidence/harness-exclusions.tsv @@ -1,4 +1,5 @@ cell surface reason +catalog-plan-vs-direct-brief cell deterministic plan-versus-brief fixture and recovery contract; no harness seat adopt-only-migration cell deterministic native adoption/replacement lifecycle probe; no harness seat agent-spec-resource-bindings cell deterministic native Agent Spec Resource-envelope probe; no harness seat context-resource-continuity cell deterministic native context/resource restart probe; no harness seat