From 2b94a09e551c80b7c026d00714e007e66f2da9f5 Mon Sep 17 00:00:00 2001 From: Nathan Date: Fri, 31 Jul 2026 15:28:59 +0200 Subject: [PATCH 1/4] test: freeze Agent Spec field-change matrix --- CATALOG.md | 6 +- .../agent-spec-field-change-matrix/README.md | 47 ++ .../agent-spec-field-change-matrix.kdl | 28 + .../fixture/cases.tsv | 19 + .../fixture/matrix.sh | 615 ++++++++++++++++++ .../judges/contract.sh | 125 ++++ .../judges/self-test.sh | 62 ++ evidence/harness-exclusions.tsv | 1 + 8 files changed, 901 insertions(+), 2 deletions(-) create mode 100644 cells/agent-spec-field-change-matrix/README.md create mode 100644 cells/agent-spec-field-change-matrix/agent-spec-field-change-matrix.kdl create mode 100644 cells/agent-spec-field-change-matrix/fixture/cases.tsv create mode 100755 cells/agent-spec-field-change-matrix/fixture/matrix.sh create mode 100755 cells/agent-spec-field-change-matrix/judges/contract.sh create mode 100755 cells/agent-spec-field-change-matrix/judges/self-test.sh diff --git a/CATALOG.md b/CATALOG.md index 233e7a6..94fa153 100644 --- a/CATALOG.md +++ b/CATALOG.md @@ -1,7 +1,7 @@ # Current eval catalog and evidence -This is the sole current corpus surface: **58 included cells** (18 model-free, +This is the sole current corpus surface: **59 included cells** (19 model-free, 40 model-backed) and **4 retired exclusions**. Inventory fields are derived from the active KDL and executable judge scripts. Latest accepted PASS evidence remains distinct from the append-only run history: a failed last run is visible without being advertised as accepted evidence, and cells with no @@ -17,6 +17,7 @@ harness-native loader plus canonical hook file. | Cell | Harness | Model(s) / effort | Model seats | Cost | Timeout | Held-out judges | Latest accepted PASS | Last recorded run | |---|---|---|---:|---|---|---:|---|---| | `adopt-only-migration` | model-free | — | 0 | none | `90s` | 6 | — | **NO STRUCTURED RUN** | +| `agent-spec-field-change-matrix` | model-free | — | 0 | none | `180s` | 5 | — | **NO STRUCTURED RUN** | | `agent-spec-resource-bindings` | model-free | — | 0 | none | `60s` | 4 | — | **NO STRUCTURED RUN** | | `assignment-contract-cold-assignment` | Codex | `gpt-5.6-sol` / medium | 2 | medium | `1200s` | 3 | — | **NO STRUCTURED RUN** | | `assignment-contract-cold-focus` | Codex | `gpt-5.6-sol` / medium | 2 | medium | `1200s` | 3 | — | **NO STRUCTURED RUN** | @@ -95,6 +96,7 @@ while the last-run column makes a recorded failure distinct from a cell with no ## Harness hook exclusions - `adopt-only-migration` / `cell` — deterministic native adoption/replacement lifecycle probe; no harness seat +- `agent-spec-field-change-matrix` / `cell` — deterministic native Agent Spec field-change contract probe; no harness seat - `agent-spec-resource-bindings` / `cell` — deterministic native Agent Spec Resource-envelope probe; no harness seat - `context-resource-continuity` / `cell` — deterministic native context/resource restart probe; no harness seat - `docs` / `judge:cold-reader` — one-shot offline Claude print grader; no bus identity, DING, or hook surface @@ -114,7 +116,7 @@ while the last-run column makes a recorded failure distinct from a cell with no - `targeted-reconcile-isolation` / `cell` — deterministic selected reconcile isolation probe; no bus-connected agent - `two-networks-coexist` / `cell` — deterministic catalog-isolation probe; no harness seat -These **19** rows are generated from structured exclusions. A cell-level row is required for +These **20** rows are generated from structured exclusions. A cell-level row is required for every derived model-free cell. The one-shot docs cold reader is a model-backed grader, but not a bus-connected agent: it has no st2 identity, DING sidecar, or lifecycle hooks. diff --git a/cells/agent-spec-field-change-matrix/README.md b/cells/agent-spec-field-change-matrix/README.md new file mode 100644 index 0000000..d374548 --- /dev/null +++ b/cells/agent-spec-field-change-matrix/README.md @@ -0,0 +1,47 @@ +# agent-spec-field-change-matrix + +Contract-first, model-free acceptance matrix for the field-change rules in merged +[`compoundingtech/st2#102`](https://github.com/compoundingtech/st2/pull/102) at exact merge +[`e54a04a3ee8af6fe0b0bce4cd961f8188ac90525`](https://github.com/compoundingtech/st2/commit/e54a04a3ee8af6fe0b0bce4cd961f8188ac90525). + +The cell is intentionally product-red. PR #102 changes documentation, not runtime behavior, and names eight +implementation gaps. A red case is accepted by the classification judges only when its public-CLI observation +matches the exact frozen map below; it is never counted as a conformance pass. The run step remains red until +all contract cases pass and the manifest is deliberately advanced with matching product evidence. + +**Capabilities required:** `st2,pty,jq,awk,grep,sed`. No model and no provider. Every catalog, workspace, exec state, +message, PTY root, process, and receipt is created below the eval-owned temporary root. + +## Closed-set map + +| Outcome | Cases | Field / gap | +| --- | --- | --- | +| Existing pass | `invalid-type-refuses`, `role-metadata-adopts`, `compact-lowering-equal`, `provider-fields-core-noop` | F04, F05, F14, F15 | +| Expected red | `source-noop-heals` | F01 / G01 | +| Expected red | `workspace-survivor-event`, `resource-survivor-event`, `render-survivor-event` | F06-F08 / G02, G04, G06 | +| Expected red | `identity-remove-add`, `task-set-remove`, `task-id-change`, `spawn-drift-visible`, `invalid-policy-refuses`, `retire-removed-child` | F02, F09-F13 / G03, G04 | +| Expected red | `host-projection` | F03 / G05 | +| Expected red | `invalid-agent-isolated`, `plan-dry-run-order` | F16, shared R05 / G01, G07 | +| Expected red | `moved-intent-refusal` | unsupported moved intent / G08 | + +The probes use only public `st2 validate`, `agents`, `up --once`, message, and task observation surfaces plus +isolated exec and PTY state. They do not import st2 source modules or treat unit tests as conformance evidence. + +## Gates + +- `judges/self-test.sh` plants missing-field, unknown-gap, expected-red-as-pass, result-drift, and residue + mutations against the static oracle. +- The focused eval runs all public-behavior probes, prints one receipt row per closed-set case, cleans every + exact eval-owned process/session, and exits nonzero while any contract case is red. +- `bin/check-corpus.sh` remains the model-free repository preflight and does not execute the product-red cell. + +Run the static oracle and the expected-red product matrix with: + +```sh +bash ./cells/agent-spec-field-change-matrix/judges/self-test.sh +st2 eval ./cells/agent-spec-field-change-matrix/ --keep +``` + +At the `e54a04a3` product pin the honest focused verdict is expected to contain five green classification +judges and one red run-step gate. Any different pass/red map is an unexpected product change and fails the +classification judge rather than silently changing the contract. diff --git a/cells/agent-spec-field-change-matrix/agent-spec-field-change-matrix.kdl b/cells/agent-spec-field-change-matrix/agent-spec-field-change-matrix.kdl new file mode 100644 index 0000000..bd4c59e --- /dev/null +++ b/cells/agent-spec-field-change-matrix/agent-spec-field-change-matrix.kdl @@ -0,0 +1,28 @@ +// Contract-first, model-free acceptance for the field-change rules proposed by merged st2 PR #102. +// The run step is intentionally RED while the exact expected G01-G08 product gaps remain. +eval { + copy "./fixture" + max-timeout "180s" + + run "matrix" { + command "bash ./matrix.sh" + } + + judges { + judge "COVERAGE - every F01-F16 field rule plus dry-run and moved intent has one closed-set case" { + exec "bash ./judges/contract.sh coverage $RUNS_DIR/matrix.out ./fixture/cases.tsv" + } + judge "GAPS - every expected red maps only to a named G01-G08 implementation gap" { + exec "bash ./judges/contract.sh gaps $RUNS_DIR/matrix.out ./fixture/cases.tsv" + } + judge "CLASSIFICATION - exact product observations match the frozen pass/red map" { + exec "bash ./judges/contract.sh classification $RUNS_DIR/matrix.out ./fixture/cases.tsv" + } + judge "HONESTY - no expected product red is counted as a conformance pass" { + exec "bash ./judges/contract.sh honesty $RUNS_DIR/matrix.out ./fixture/cases.tsv" + } + judge "CLEANUP - isolated exec and PTY state has zero live residue" { + exec "bash ./judges/contract.sh cleanup $RUNS_DIR/matrix.out ./fixture/cases.tsv" + } + } +} diff --git a/cells/agent-spec-field-change-matrix/fixture/cases.tsv b/cells/agent-spec-field-change-matrix/fixture/cases.tsv new file mode 100644 index 0000000..80a6384 --- /dev/null +++ b/cells/agent-spec-field-change-matrix/fixture/cases.tsv @@ -0,0 +1,19 @@ +case_id fields expected gaps contract +source-noop-heals F01 RED G01 semantic no-op writes and reports nothing while independently dead work heals +identity-remove-add F02 RED G03 exact old identity is removed before the new identity boots +host-projection F03 RED G05 old and new hosts converge through independent local projections +invalid-type-refuses F04 PASS - unsupported type refuses before launch write or teardown +role-metadata-adopts F05 PASS - role-only change preserves the healthy generation without notification +workspace-survivor-event F06 RED G02,G04,G06 workspace survivor keeps its generation and receives one post-commit event +resource-survivor-event F07 RED G02,G06 Resource-only change preserves the generation and emits one post-commit event +render-survivor-event F08 RED G02,G06 changed render commits before one event while unchanged bytes emit none +task-set-remove F09 RED G03 removed child is cleaned by exact ownership without changing its sibling +task-id-change F10 RED G03 exact old task id is removed before the new id boots +spawn-drift-visible F11 RED G03,G04 healthy spawn mismatch stays live and reports desired and observed fingerprints +invalid-policy-refuses F12 RED G03 invalid restart or lifecycle policy refuses instead of defaulting +retire-removed-child F13 RED G03 retirement stops every owned child including one removed in the same revision +compact-lowering-equal F14 PASS - compact command and ding lower to the same public task contract as explicit tasks +provider-fields-core-noop F15 PASS - provider-only fields do not change core equality launch or wake behavior +invalid-agent-isolated F16 RED G01,G07 invalid related work refuses while an independent valid agent progresses +plan-dry-run-order R05 RED G07 true dry-run reports the fixed action order and writes nothing +moved-intent-refusal MOVED RED G08 unsupported moved intent rejects cycles conflicts and host changes diff --git a/cells/agent-spec-field-change-matrix/fixture/matrix.sh b/cells/agent-spec-field-change-matrix/fixture/matrix.sh new file mode 100755 index 0000000..7470754 --- /dev/null +++ b/cells/agent-spec-field-change-matrix/fixture/matrix.sh @@ -0,0 +1,615 @@ +#!/usr/bin/env bash +set -uo pipefail + +root="${CATALOG:?CATALOG must be set}" +manifest="$root/cases.tsv" +cases_root="$root/cases" +mkdir -p "$cases_root" + +case_dir() { + printf '%s/%s\n' "$cases_root" "$1" +} + +case_env() { + id="$1" + shift + dir="$(case_dir "$id")" + CATALOG="$dir/catalog" \ + XDG_STATE_HOME="$dir/state" \ + PTY_ROOT="$dir/pty" \ + "$@" +} + +initialize_case() { + id="$1" + dir="$(case_dir "$id")" + mkdir -p "$dir/catalog" "$dir/state" "$dir/pty" "$dir/work-a" "$dir/work-b" + cat >"$dir/task.sh" <<'SH' +#!/usr/bin/env bash +set -euo pipefail +label="${1:?label required}" +count_file="${2:?count file required}" +count=0 +test ! -f "$count_file" || count="$(cat "$count_file")" +count=$((count + 1)) +printf '%s\n' "$count" >"$count_file" +printf '%s\n' "$$" >"${count_file}.pid" +printf '%s\n' "$PWD" >"${count_file}.cwd" +trap 'exit 0' TERM INT +while :; do sleep 1; done +SH + chmod +x "$dir/task.sh" +} + +write_single_exec() { + id="$1" + host="$2" + identity="$3" + role="$4" + type="$5" + workspace="$6" + task_name="$7" + task_id="$8" + label="$9" + agent_extra="${10:-}" + task_extra="${11:-}" + dir="$(case_dir "$id")" + spec="$dir/catalog/agents/$host/$identity/agent.kdl" + mkdir -p "$(dirname "$spec")" + { + printf 'agent "%s" {\n' "$identity" + printf ' host "%s"\n' "$host" + printf ' role "%s"\n' "$role" + printf ' type "%s"\n' "$type" + printf ' workspace "%s"\n' "$workspace" + test -z "$agent_extra" || printf '%s\n' "$agent_extra" + printf ' exec "%s" {\n' "$task_name" + test "$task_id" = "-" || printf ' id "%s"\n' "$task_id" + printf ' command "exec bash \\"%s/task.sh\\" \\"%s\\" \\"%s/%s-count\\""\n' \ + "$dir" "$label" "$dir" "$label" + test -z "$task_extra" || printf '%s\n' "$task_extra" + printf ' }\n' + printf '}\n' + } >"$spec" + printf '%s\n' "$spec" +} + +write_two_exec() { + id="$1" + host="$2" + identity="$3" + workspace="$4" + first_name="$5" + first_id="$6" + first_label="$7" + second_name="$8" + second_id="$9" + second_label="${10}" + retired="${11:-false}" + dir="$(case_dir "$id")" + spec="$dir/catalog/agents/$host/$identity/agent.kdl" + mkdir -p "$(dirname "$spec")" + { + printf 'agent "%s" {\n' "$identity" + printf ' host "%s"\n' "$host" + printf ' role "worker"\n' + printf ' type "service"\n' + printf ' workspace "%s"\n' "$workspace" + test "$retired" != true || printf ' retired #true\n' + for task in first second; do + if test "$task" = first; then + name="$first_name" task_id="$first_id" label="$first_label" + else + name="$second_name" task_id="$second_id" label="$second_label" + fi + test "$name" = "-" && continue + printf ' exec "%s" {\n' "$name" + test "$task_id" = "-" || printf ' id "%s"\n' "$task_id" + printf ' command "exec bash \\"%s/task.sh\\" \\"%s\\" \\"%s/%s-count\\""\n' \ + "$dir" "$label" "$dir" "$label" + printf ' }\n' + done + printf '}\n' + } >"$spec" + printf '%s\n' "$spec" +} + +run_once() { + id="$1" + host="$2" + label="$3" + dir="$(case_dir "$id")" + case_env "$id" st2 up --once --catalog "$dir/catalog" --host "$host" \ + >"$dir/$label.out" 2>"$dir/$label.err" +} + +validate_case() { + id="$1" + host="$2" + label="$3" + dir="$(case_dir "$id")" + case_env "$id" st2 validate --catalog "$dir/catalog" --host "$host" \ + >"$dir/$label.out" 2>"$dir/$label.err" +} + +tasks_json() { + id="$1" + host="$2" + case_env "$id" st2 tasks --json --catalog "$(case_dir "$id")/catalog" --host "$host" +} + +task_pid() { + id="$1" + host="$2" + runtime_id="$3" + tasks_json "$id" "$host" 2>/dev/null | + jq -r --arg runtime_id "$runtime_id" \ + '.tasks[]? | select(.runtimeId == $runtime_id) | .runtime.pid // empty' +} + +wait_pid() { + id="$1" + host="$2" + runtime_id="$3" + for _ in $(seq 1 100); do + pid="$(task_pid "$id" "$host" "$runtime_id")" + if test -n "$pid" && kill -0 "$pid" 2>/dev/null; then + printf '%s\n' "$pid" + return 0 + fi + sleep 0.05 + done + return 1 +} + +wait_dead() { + pid="$1" + for _ in $(seq 1 100); do + kill -0 "$pid" 2>/dev/null || return 0 + sleep 0.05 + done + return 1 +} + +stop_exact_pid() { + pid="$1" + case "$pid" in + ''|*[!0-9]*) return 1 ;; + esac + kill -TERM -- "-$pid" 2>/dev/null || kill -TERM "$pid" 2>/dev/null || true + wait_dead "$pid" +} + +message_count() { + id="$1" + identity="$2" + case_env "$id" st2 message ls "$identity" --json 2>/dev/null | + jq 'length' 2>/dev/null || printf '0\n' +} + +probe_source_noop_heals() { + id="source-noop-heals" host="fm01" identity="agent" runtime_id="$host.$identity.work" + initialize_case "$id" + dir="$(case_dir "$id")" + spec="$(write_single_exec "$id" "$host" "$identity" worker service "$dir/work-a" work "$runtime_id" noop)" + case_env "$id" st2 up --catalog "$dir/catalog" --host "$host" --interval 30 \ + >"$dir/supervisor.out" 2>"$dir/supervisor.err" & + supervisor_pid="$!" + trap 'kill "$supervisor_pid" >/dev/null 2>&1 || true; wait "$supervisor_pid" >/dev/null 2>&1 || true' EXIT + pid_before="$(wait_pid "$id" "$host" "$runtime_id")" + adopted_before="$(grep -Fc 'adopted' "$dir/supervisor.out" 2>/dev/null || true)" + printf '// live semantic source-only no-op\n' >>"$spec" + sleep 1 + adopted_after="$(grep -Fc 'adopted' "$dir/supervisor.out" 2>/dev/null || true)" + pid_live="$(wait_pid "$id" "$host" "$runtime_id")" + stop_exact_pid "$pid_before" + printf '// independently dead work may heal\n' >>"$spec" + for _ in $(seq 1 100); do + test -f "$dir/noop-count" && test "$(cat "$dir/noop-count")" -eq 2 && break + sleep 0.05 + done + pid_after="$(wait_pid "$id" "$host" "$runtime_id")" + kill "$supervisor_pid" >/dev/null 2>&1 || true + wait "$supervisor_pid" >/dev/null 2>&1 || true + trap - EXIT + test "$pid_live" = "$pid_before" + test "$pid_after" != "$pid_before" + test "$(cat "$dir/noop-count")" -eq 2 + test "$(message_count "$id" "$host.$identity")" -eq 0 + test "$adopted_after" -eq "$adopted_before" +} + +probe_identity_remove_add() { + id="identity-remove-add" host="fm02" + initialize_case "$id" + dir="$(case_dir "$id")" + spec="$(write_single_exec "$id" "$host" old worker service "$dir/work-a" work "$host.old.work" old)" + run_once "$id" "$host" old + old_pid="$(wait_pid "$id" "$host" "$host.old.work")" + rm -f "$spec" + write_single_exec "$id" "$host" new worker service "$dir/work-a" work "$host.new.work" new >/dev/null + run_once "$id" "$host" new + wait_pid "$id" "$host" "$host.new.work" >/dev/null + wait_dead "$old_pid" +} + +probe_host_projection() { + id="host-projection" identity="agent" + initialize_case "$id" + dir="$(case_dir "$id")" + spec="$(write_single_exec "$id" old "$identity" worker service "$dir/work-a" work - oldhost)" + run_once "$id" old old + old_pid="$(wait_pid "$id" old "old.$identity.work")" + rm -f "$spec" + write_single_exec "$id" new "$identity" worker service "$dir/work-a" work - newhost >/dev/null + run_once "$id" old remove-old + run_once "$id" new add-new + wait_pid "$id" new "new.$identity.work" >/dev/null + wait_dead "$old_pid" +} + +probe_invalid_type_refuses() { + id="invalid-type-refuses" host="fm04" + initialize_case "$id" + dir="$(case_dir "$id")" + write_single_exec "$id" "$host" agent worker invalid-f102 "$dir/work-a" work "$host.agent.work" invalid >/dev/null + if validate_case "$id" "$host" validate; then + return 1 + fi + test -z "$(find "$dir/state" -type f -name '*.pid' -print -quit)" +} + +probe_role_metadata_adopts() { + id="role-metadata-adopts" host="fm05" identity="agent" runtime_id="$host.$identity.work" + initialize_case "$id" + dir="$(case_dir "$id")" + spec="$(write_single_exec "$id" "$host" "$identity" worker service "$dir/work-a" work "$runtime_id" role)" + run_once "$id" "$host" launch + pid_before="$(wait_pid "$id" "$host" "$runtime_id")" + sed -i 's/role "worker"/role "reviewer"/' "$spec" + run_once "$id" "$host" role + pid_after="$(wait_pid "$id" "$host" "$runtime_id")" + test "$pid_after" = "$pid_before" + test "$(message_count "$id" "$host.$identity")" -eq 0 +} + +probe_workspace_survivor_event() { + id="workspace-survivor-event" host="fm06" identity="agent" runtime_id="$host.$identity.work" + initialize_case "$id" + dir="$(case_dir "$id")" + spec="$(write_single_exec "$id" "$host" "$identity" worker service "$dir/work-a" work "$runtime_id" workspace)" + run_once "$id" "$host" launch + pid_before="$(wait_pid "$id" "$host" "$runtime_id")" + sed -i "s|workspace \"$dir/work-a\"|workspace \"$dir/work-b\"|" "$spec" + run_once "$id" "$host" workspace + pid_after="$(wait_pid "$id" "$host" "$runtime_id")" + test "$pid_after" = "$pid_before" + test "$(message_count "$id" "$host.$identity")" -eq 1 +} + +probe_resource_survivor_event() { + id="resource-survivor-event" host="fm07" identity="agent" runtime_id="$host.$identity.work" + initialize_case "$id" + dir="$(case_dir "$id")" + spec="$(write_single_exec "$id" "$host" "$identity" worker service "$dir/work-a" work "$runtime_id" resource \ + ' resource "work" _tag="issue" uri="issue://field/old"')" + run_once "$id" "$host" launch + pid_before="$(wait_pid "$id" "$host" "$runtime_id")" + sed -i 's|issue://field/old|issue://field/new|' "$spec" + run_once "$id" "$host" resource + pid_after="$(wait_pid "$id" "$host" "$runtime_id")" + test "$pid_after" = "$pid_before" + case_env "$id" st2 agents --json --catalog "$dir/catalog" | + jq -e '.[] | select(.identity == "fm07.agent") | .resources[] | select(.uri == "issue://field/new")' >/dev/null + test "$(message_count "$id" "$host.$identity")" -eq 1 +} + +probe_render_survivor_event() { + id="render-survivor-event" host="fm08" identity="agent" runtime_id="$host.$identity.work" + initialize_case "$id" + dir="$(case_dir "$id")" + spec="$(write_single_exec "$id" "$host" "$identity" worker service "$dir/work-a" work "$runtime_id" render \ + $' render {\n file "context.txt" "A-f102"\n }')" + run_once "$id" "$host" launch + pid_before="$(wait_pid "$id" "$host" "$runtime_id")" + test "$(cat "$dir/work-a/context.txt")" = A-f102 + sed -i 's/A-f102/B-f102/' "$spec" + run_once "$id" "$host" render + pid_after="$(wait_pid "$id" "$host" "$runtime_id")" + test "$pid_after" = "$pid_before" + test "$(cat "$dir/work-a/context.txt")" = B-f102 + test "$(message_count "$id" "$host.$identity")" -eq 1 + run_once "$id" "$host" unchanged + test "$(message_count "$id" "$host.$identity")" -eq 1 +} + +probe_task_set_remove() { + id="task-set-remove" host="fm09" identity="agent" + initialize_case "$id" + dir="$(case_dir "$id")" + write_two_exec "$id" "$host" "$identity" "$dir/work-a" removed "$host.$identity.removed" removed keep "$host.$identity.keep" keep >/dev/null + run_once "$id" "$host" launch + removed_pid="$(wait_pid "$id" "$host" "$host.$identity.removed")" + keep_pid="$(wait_pid "$id" "$host" "$host.$identity.keep")" + write_two_exec "$id" "$host" "$identity" "$dir/work-a" - - - keep "$host.$identity.keep" keep >/dev/null + run_once "$id" "$host" remove + keep_after="$(wait_pid "$id" "$host" "$host.$identity.keep")" + test "$keep_after" = "$keep_pid" + wait_dead "$removed_pid" +} + +probe_task_id_change() { + id="task-id-change" host="fm10" identity="agent" + initialize_case "$id" + dir="$(case_dir "$id")" + write_single_exec "$id" "$host" "$identity" worker service "$dir/work-a" old "$host.$identity.old" old >/dev/null + run_once "$id" "$host" old + old_pid="$(wait_pid "$id" "$host" "$host.$identity.old")" + write_single_exec "$id" "$host" "$identity" worker service "$dir/work-a" new "$host.$identity.new" new >/dev/null + run_once "$id" "$host" new + wait_pid "$id" "$host" "$host.$identity.new" >/dev/null + wait_dead "$old_pid" +} + +probe_spawn_drift_visible() { + id="spawn-drift-visible" host="fm11" identity="agent" runtime_id="$host.$identity.work" + initialize_case "$id" + dir="$(case_dir "$id")" + spec="$(write_single_exec "$id" "$host" "$identity" worker service "$dir/work-a" work "$runtime_id" generation-a)" + run_once "$id" "$host" launch + pid_before="$(wait_pid "$id" "$host" "$runtime_id")" + sed -i 's/generation-a/generation-b/g' "$spec" + run_once "$id" "$host" drift + pid_after="$(wait_pid "$id" "$host" "$runtime_id")" + test "$pid_after" = "$pid_before" + inventory="$(tasks_json "$id" "$host")" + printf '%s\n' "$inventory" | jq -e --arg runtime_id "$runtime_id" ' + .tasks[] | select(.runtimeId == $runtime_id) | + (.runtime.state == "drifted" or .runtime.state == "unknown") and + (.desiredFingerprint | type == "string" and length > 0) and + (.observedFingerprint | type == "string" and length > 0) and + (.desiredFingerprint != .observedFingerprint) + ' >/dev/null +} + +probe_invalid_policy_refuses() { + id="invalid-policy-refuses" host="fm12" + initialize_case "$id" + dir="$(case_dir "$id")" + write_single_exec "$id" "$host" agent worker service "$dir/work-a" work "$host.agent.work" policy \ + $' restart {\n mode "invalid-f102"\n }' >/dev/null + if validate_case "$id" "$host" validate; then + return 1 + fi + test -z "$(find "$dir/state" -type f -name '*.pid' -print -quit)" +} + +probe_retire_removed_child() { + id="retire-removed-child" host="fm13" identity="agent" + initialize_case "$id" + dir="$(case_dir "$id")" + write_two_exec "$id" "$host" "$identity" "$dir/work-a" removed "$host.$identity.removed" removed keep "$host.$identity.keep" keep >/dev/null + run_once "$id" "$host" launch + removed_pid="$(wait_pid "$id" "$host" "$host.$identity.removed")" + keep_pid="$(wait_pid "$id" "$host" "$host.$identity.keep")" + write_two_exec "$id" "$host" "$identity" "$dir/work-a" - - - keep "$host.$identity.keep" keep true >/dev/null + run_once "$id" "$host" retire + wait_dead "$keep_pid" + wait_dead "$removed_pid" +} + +probe_compact_lowering_equal() { + id="compact-lowering-equal" host="fm14" identity="agent" + initialize_case "$id" + dir="$(case_dir "$id")" + compact="$dir/catalog/compact/agents/$host/$identity/agent.kdl" + explicit="$dir/catalog/explicit/agents/$host/$identity/agent.kdl" + mkdir -p "$(dirname "$compact")" "$(dirname "$explicit")" + cat >"$compact" <"$explicit" <"$dir/compact.json" + CATALOG="$dir/catalog/explicit" XDG_STATE_HOME="$dir/state/explicit" PTY_ROOT="$dir/pty/explicit" \ + st2 tasks --json --catalog "$dir/catalog/explicit" --host "$host" | + jq -S '[.tasks[] | {agent,task,runtimeId,kind,lifecycle,retired,desiredState}]' >"$dir/explicit.json" + test "$(cat "$dir/compact.json")" = "$(cat "$dir/explicit.json")" + test "$(jq 'length' "$dir/compact.json")" -eq 2 +} + +probe_provider_fields_core_noop() { + id="provider-fields-core-noop" host="fm15" identity="agent" + initialize_case "$id" + dir="$(case_dir "$id")" + for arm in base provider; do + spec="$dir/catalog/$arm/agents/$host/$identity/agent.kdl" + mkdir -p "$(dirname "$spec")" + { + printf 'agent "%s" {\n' "$identity" + printf ' host "%s"\n' "$host" + printf ' role "worker"\n' + printf ' type "service"\n' + printf ' workspace "%s"\n' "$dir/work-a" + if test "$arm" = provider; then + printf ' harness "codex"\n model "provider-model"\n persona "provider-persona"\n' + printf ' permissions "provider-policy"\n transport "provider-transport"\n' + printf ' strategy "provider-strategy"\n meta { opaque "value" }\n' + fi + printf ' exec "work" {\n' + printf ' id "%s.%s.work"\n' "$host" "$identity" + printf ' command "true"\n' + printf ' }\n' + printf '}\n' + } >"$spec" + CATALOG="$dir/catalog/$arm" XDG_STATE_HOME="$dir/state/$arm" PTY_ROOT="$dir/pty/$arm" \ + st2 validate --catalog "$dir/catalog/$arm" --host "$host" >"$dir/$arm.validate" 2>&1 + CATALOG="$dir/catalog/$arm" XDG_STATE_HOME="$dir/state/$arm" PTY_ROOT="$dir/pty/$arm" \ + st2 tasks --json --catalog "$dir/catalog/$arm" --host "$host" | + jq -S '[.tasks[] | {agent,task,runtimeId,kind,lifecycle,retired,desiredState}]' >"$dir/$arm.tasks" + CATALOG="$dir/catalog/$arm" XDG_STATE_HOME="$dir/state/$arm" PTY_ROOT="$dir/pty/$arm" \ + st2 agents --json --catalog "$dir/catalog/$arm" | + jq -S '[.[] | {identity,retired,resources}]' >"$dir/$arm.agents" + done + test "$(cat "$dir/base.tasks")" = "$(cat "$dir/provider.tasks")" + test "$(cat "$dir/base.agents")" = "$(cat "$dir/provider.agents")" +} + +probe_invalid_agent_isolated() { + id="invalid-agent-isolated" host="fm16" + initialize_case "$id" + dir="$(case_dir "$id")" + write_single_exec "$id" "$host" valid worker service "$dir/work-a" work "$host.valid.work" valid >/dev/null + write_single_exec "$id" "$host" invalid worker service "$dir/work-b" work "$host.valid.work" invalid >/dev/null + set +e + run_once "$id" "$host" mixed + rc=$? + set -e + test "$rc" -ne 0 + wait_pid "$id" "$host" "$host.valid.work" >/dev/null +} + +probe_plan_dry_run_order() { + id="plan-dry-run-order" host="fm17" + initialize_case "$id" + dir="$(case_dir "$id")" + write_single_exec "$id" "$host" agent worker service "$dir/work-a" work "$host.agent.work" dry >/dev/null + st2 up --help | grep -Fq -- '--dry-run' + case_env "$id" st2 up --dry-run --catalog "$dir/catalog" --host "$host" >"$dir/dry-run.out" + test -z "$(find "$dir/state" -type f -name '*.pid' -print -quit)" + awk ' + /FENCE/ { if (phase < 1) phase=1 } + /REMOVE\/QUIESCE/ { if (phase == 1) phase=2 } + /MATERIALIZE/ { if (phase == 2) phase=3 } + /ADD\/BOOT/ { if (phase == 3) phase=4 } + /NOTIFY/ { if (phase == 4) phase=5 } + /VERIFY\/REPORT/ { if (phase == 5) phase=6 } + END { exit phase == 6 ? 0 : 1 } + ' "$dir/dry-run.out" +} + +probe_moved_intent_refusal() { + id="moved-intent-refusal" host="fm18" + initialize_case "$id" + dir="$(case_dir "$id")" + spec="$(write_single_exec "$id" "$host" agent worker service "$dir/work-a" work "$host.agent.work" moved \ + $' moved {\n from "fm18.agent"\n to "fm18.agent"\n }')" + if validate_case "$id" "$host" validate; then + return 1 + fi + test -f "$spec" + test -z "$(find "$dir/state" -type f -name '*.pid' -print -quit)" +} + +cleanup_all() { + live_exec=0 + live_pty=0 + live_process=0 + + while IFS= read -r pid_file; do + pid="$(cat "$pid_file" 2>/dev/null || true)" + case "$pid" in + ''|*[!0-9]*) continue ;; + esac + if kill -0 "$pid" 2>/dev/null; then + stop_exact_pid "$pid" || true + fi + done < <(find "$cases_root" -type f -path '*/state/st2/*/exec/*.pid' -print 2>/dev/null) + + while IFS= read -r pty_root; do + while IFS= read -r name; do + test -n "$name" || continue + PTY_ROOT="$pty_root" pty kill "$name" >/dev/null 2>&1 || true + PTY_ROOT="$pty_root" pty rm "$name" >/dev/null 2>&1 || true + done < <(PTY_ROOT="$pty_root" pty list --json 2>/dev/null | jq -r '.[].name' 2>/dev/null || true) + done < <(find "$cases_root" -type d -name pty -print 2>/dev/null) + + while IFS= read -r pid_file; do + pid="$(cat "$pid_file" 2>/dev/null || true)" + case "$pid" in + ''|*[!0-9]*) continue ;; + esac + if kill -0 "$pid" 2>/dev/null; then + live_exec=$((live_exec + 1)) + live_process=$((live_process + 1)) + fi + done < <(find "$cases_root" -type f -path '*/state/st2/*/exec/*.pid' -print 2>/dev/null) + + while IFS= read -r pty_root; do + count="$(PTY_ROOT="$pty_root" pty list --json 2>/dev/null | jq 'length' 2>/dev/null || printf '0\n')" + live_pty=$((live_pty + count)) + live_process=$((live_process + count)) + done < <(find "$cases_root" -type d -name pty -print 2>/dev/null) + + if test "$live_exec" -eq 0 && test "$live_pty" -eq 0 && test "$live_process" -eq 0; then + printf 'ZERO-RESIDUE\texec=0\tpty=0\tprocess=0\tcatalogs=18\n' + return 0 + fi + printf 'RESIDUE\texec=%s\tpty=%s\tprocess=%s\n' "$live_exec" "$live_pty" "$live_process" + return 1 +} + +probe_name() { + printf 'probe_%s\n' "$(printf '%s' "$1" | tr '-' '_')" +} + +pass_count=0 +red_count=0 +unexpected=0 + +while IFS=$'\t' read -r case_id fields expected gaps contract; do + test "$case_id" != case_id || continue + probe="$(probe_name "$case_id")" + log="$(case_dir "$case_id")/probe.log" + mkdir -p "$(dirname "$log")" + ( + set -euo pipefail + "$probe" + ) >"$log" 2>&1 + rc=$? + if test "$rc" -eq 0; then + observed=PASS + pass_count=$((pass_count + 1)) + else + observed=RED + red_count=$((red_count + 1)) + fi + test "$observed" = "$expected" || unexpected=$((unexpected + 1)) + printf 'RESULT\t%s\t%s\t%s\t%s\t%s\trc=%s log=cases/%s/probe.log\n' \ + "$case_id" "$fields" "$expected" "$gaps" "$observed" "$rc" "$case_id" +done <"$manifest" + +printf 'SUMMARY\tconformance-pass=%s\texpected-red=%s\tunexpected=%s\n' \ + "$pass_count" "$red_count" "$unexpected" + +cleanup_rc=0 +cleanup_all || cleanup_rc=$? + +# This is a contract conformance step, not a gap-classification step. Keep it +# red while any product case is red, even when every red is expected and mapped. +test "$unexpected" -eq 0 || exit 2 +test "$cleanup_rc" -eq 0 || exit 3 +test "$red_count" -eq 0 diff --git a/cells/agent-spec-field-change-matrix/judges/contract.sh b/cells/agent-spec-field-change-matrix/judges/contract.sh new file mode 100755 index 0000000..7a81164 --- /dev/null +++ b/cells/agent-spec-field-change-matrix/judges/contract.sh @@ -0,0 +1,125 @@ +#!/usr/bin/env bash +set -euo pipefail + +mode="${1:?mode required}" +output="${2:?matrix output required}" +manifest="${3:?case manifest required}" + +fail() { + printf 'FAIL: %s\n' "$*" >&2 + exit 1 +} + +test -f "$manifest" || fail "missing manifest $manifest" +test -f "$output" || fail "missing matrix output $output" + +coverage() { + awk -F '\t' ' + NR == 1 { + if ($0 != "case_id\tfields\texpected\tgaps\tcontract") exit 10 + next + } + NF != 5 || $1 == "" || $5 == "" { exit 11 } + { seen_case[$1]++; fields = fields (fields == "" ? "" : ",") $2 } + END { + if (NR != 19) exit 12 + for (case_id in seen_case) if (seen_case[case_id] != 1) exit 13 + if (fields != "F01,F02,F03,F04,F05,F06,F07,F08,F09,F10,F11,F12,F13,F14,F15,F16,R05,MOVED") exit 14 + } + ' "$manifest" || fail "manifest is not the exact F01-F16 plus R05/MOVED closed set" + echo "PASS: 18 unique cases cover F01-F16 plus dry-run ordering and moved intent" +} + +gaps() { + awk -F '\t' ' + NR == 1 { next } + $3 == "PASS" { + if ($4 != "-") exit 20 + pass++ + next + } + $3 != "RED" { exit 21 } + { + red++ + count = split($4, parts, ",") + if (count < 1) exit 22 + for (i = 1; i <= count; i++) { + if (parts[i] !~ /^G0[1-8]$/) exit 23 + seen[parts[i]] = 1 + } + } + END { + if (pass != 4 || red != 14) exit 24 + for (i = 1; i <= 8; i++) { + gap = sprintf("G%02d", i) + if (!(gap in seen)) exit 25 + } + } + ' "$manifest" || fail "expected-red map is not limited to and complete across G01-G08" + echo "PASS: 14 expected reds map only to named gaps G01-G08; 4 cases are existing passes" +} + +classification() { + awk -F '\t' ' + NR == FNR { + if (FNR == 1) next + fields[$1] = $2 + expected[$1] = $3 + gaps[$1] = $4 + cases[$1] = 1 + manifest_count++ + next + } + $1 != "RESULT" { next } + { + case_id = $2 + if (!(case_id in cases) || seen[case_id]++) exit 30 + if ($3 != fields[case_id] || $4 != expected[case_id] || $5 != gaps[case_id]) exit 31 + if ($6 != expected[case_id]) exit 32 + result_count++ + } + END { + if (manifest_count != 18 || result_count != manifest_count) exit 33 + for (case_id in cases) if (!(case_id in seen)) exit 34 + } + ' "$manifest" "$output" || fail "product results drift from the exact closed-set pass/red map" + echo "PASS: all 18 product observations match the frozen classification" +} + +honesty() { + awk -F '\t' ' + NR == FNR { + if (FNR > 1) expected[$1] = $3 + next + } + $1 == "RESULT" { + if (expected[$2] == "RED" && $6 != "RED") exit 40 + if ($6 == "PASS") pass++ + if ($6 == "RED") red++ + next + } + $1 == "SUMMARY" { + summary++ + if ($2 != "conformance-pass=4" || $3 != "expected-red=14" || $4 != "unexpected=0") exit 41 + } + END { + if (summary != 1 || pass != 4 || red != 14) exit 42 + } + ' "$manifest" "$output" || fail "an expected red was counted as conformance or the summary drifted" + echo "PASS: no expected product red is counted as a conformance pass" +} + +cleanup() { + count="$(grep -Fxc $'ZERO-RESIDUE\texec=0\tpty=0\tprocess=0\tcatalogs=18' "$output" || true)" + test "$count" -eq 1 || fail "missing exact zero-residue receipt" + echo "PASS: exact cleanup receipt proves zero exec, PTY, and process residue" +} + +case "$mode" in + coverage) coverage ;; + gaps) gaps ;; + classification) classification ;; + honesty) honesty ;; + cleanup) cleanup ;; + *) fail "unknown contract mode $mode" ;; +esac diff --git a/cells/agent-spec-field-change-matrix/judges/self-test.sh b/cells/agent-spec-field-change-matrix/judges/self-test.sh new file mode 100755 index 0000000..efe84e8 --- /dev/null +++ b/cells/agent-spec-field-change-matrix/judges/self-test.sh @@ -0,0 +1,62 @@ +#!/usr/bin/env bash +set -euo pipefail + +here="$(cd "$(dirname "$0")" && pwd)" +manifest="$here/../fixture/cases.tsv" +oracle="$here/contract.sh" +tmp="$(mktemp -d "${TMPDIR:-/tmp}/agent-spec-field-change-self-test.XXXXXX")" +trap 'rm -rf "$tmp"' EXIT + +good="$tmp/good.out" +awk -F '\t' 'BEGIN { OFS="\t" } NR > 1 { print "RESULT", $1, $2, $3, $4, $3, "planted-observation" }' \ + "$manifest" >"$good" +printf 'SUMMARY\tconformance-pass=4\texpected-red=14\tunexpected=0\n' >>"$good" +printf 'ZERO-RESIDUE\texec=0\tpty=0\tprocess=0\tcatalogs=18\n' >>"$good" + +for mode in coverage gaps classification honesty cleanup; do + bash "$oracle" "$mode" "$good" "$manifest" >/dev/null +done +echo "PASS: canonical static receipt satisfies all five oracle modes" + +missing="$tmp/missing.tsv" +awk -F '\t' '$2 != "F16"' "$manifest" >"$missing" +if bash "$oracle" coverage "$good" "$missing" >/dev/null 2>&1; then + echo "FAIL: missing F16 case passed coverage" >&2 + exit 1 +fi +echo "PASS: missing-field mutation fails coverage" + +unknown_gap="$tmp/unknown-gap.tsv" +sed 's/F01\tRED\tG01/F01\tRED\tG09/' "$manifest" >"$unknown_gap" +if bash "$oracle" gaps "$good" "$unknown_gap" >/dev/null 2>&1; then + echo "FAIL: unknown G09 gap passed mapping" >&2 + exit 1 +fi +echo "PASS: unknown-gap mutation fails mapping" + +red_as_pass="$tmp/red-as-pass.out" +sed $'s/RESULT\tidentity-remove-add\tF02\tRED\tG03\tRED/RESULT\tidentity-remove-add\tF02\tRED\tG03\tPASS/' \ + "$good" >"$red_as_pass" +if bash "$oracle" honesty "$red_as_pass" "$manifest" >/dev/null 2>&1; then + echo "FAIL: expected red counted as pass" >&2 + exit 1 +fi +echo "PASS: expected-red-as-pass mutation fails honesty" + +drift="$tmp/drift.out" +sed $'s/RESULT\thost-projection\tF03/RESULT\thost-projection\tF99/' "$good" >"$drift" +if bash "$oracle" classification "$drift" "$manifest" >/dev/null 2>&1; then + echo "FAIL: result field drift passed classification" >&2 + exit 1 +fi +echo "PASS: result-drift mutation fails classification" + +no_cleanup="$tmp/no-cleanup.out" +grep -v '^ZERO-RESIDUE' "$good" >"$no_cleanup" +if bash "$oracle" cleanup "$no_cleanup" "$manifest" >/dev/null 2>&1; then + echo "FAIL: missing cleanup receipt passed" >&2 + exit 1 +fi +echo "PASS: missing-residue-receipt mutation fails cleanup" + +echo "STATIC-ORACLE-SELF-TEST-GREEN-f102" diff --git a/evidence/harness-exclusions.tsv b/evidence/harness-exclusions.tsv index 2ac5f66..a2440bf 100644 --- a/evidence/harness-exclusions.tsv +++ b/evidence/harness-exclusions.tsv @@ -1,5 +1,6 @@ cell surface reason adopt-only-migration cell deterministic native adoption/replacement lifecycle probe; no harness seat +agent-spec-field-change-matrix cell deterministic native Agent Spec field-change contract probe; no harness seat agent-spec-resource-bindings cell deterministic native Agent Spec Resource-envelope probe; no harness seat context-resource-continuity cell deterministic native context/resource restart probe; no harness seat docs judge:cold-reader one-shot offline Claude print grader; no bus identity, DING, or hook surface From 5a5b6633336317f07680a846d0e7991e722937f6 Mon Sep 17 00:00:00 2001 From: Nathan Date: Fri, 31 Jul 2026 16:11:21 +0200 Subject: [PATCH 2/4] test: strengthen Agent Spec field-change proofs --- .../agent-spec-field-change-matrix/README.md | 20 +- .../fixture/cases.tsv | 20 +- .../fixture/matrix.sh | 623 +++++++++++++++--- .../judges/self-test.sh | 13 +- 4 files changed, 549 insertions(+), 127 deletions(-) diff --git a/cells/agent-spec-field-change-matrix/README.md b/cells/agent-spec-field-change-matrix/README.md index d374548..caf15bd 100644 --- a/cells/agent-spec-field-change-matrix/README.md +++ b/cells/agent-spec-field-change-matrix/README.md @@ -9,28 +9,32 @@ implementation gaps. A red case is accepted by the classification judges only wh matches the exact frozen map below; it is never counted as a conformance pass. The run step remains red until all contract cases pass and the manifest is deliberately advanced with matching product evidence. -**Capabilities required:** `st2,pty,jq,awk,grep,sed`. No model and no provider. Every catalog, workspace, exec state, +**Capabilities required:** `st2,pty,jq,awk,grep,sed,coreutils`. No model and no provider. Every catalog, workspace, exec state, message, PTY root, process, and receipt is created below the eval-owned temporary root. ## Closed-set map | Outcome | Cases | Field / gap | | --- | --- | --- | -| Existing pass | `invalid-type-refuses`, `role-metadata-adopts`, `compact-lowering-equal`, `provider-fields-core-noop` | F04, F05, F14, F15 | -| Expected red | `source-noop-heals` | F01 / G01 | +| Existing pass | `source-noop-heals`, `compact-lowering-equal`, `provider-fields-core-noop`, `invalid-agent-isolated` | F01, F14-F16 | +| Expected red | `invalid-type-refuses` | F04 / G01 | | Expected red | `workspace-survivor-event`, `resource-survivor-event`, `render-survivor-event` | F06-F08 / G02, G04, G06 | | Expected red | `identity-remove-add`, `task-set-remove`, `task-id-change`, `spawn-drift-visible`, `invalid-policy-refuses`, `retire-removed-child` | F02, F09-F13 / G03, G04 | | Expected red | `host-projection` | F03 / G05 | -| Expected red | `invalid-agent-isolated`, `plan-dry-run-order` | F16, shared R05 / G01, G07 | +| Expected red | `role-metadata-adopts`, `plan-dry-run-order` | F05, shared R05 / G07 | | Expected red | `moved-intent-refusal` | unsupported moved intent / G08 | -The probes use only public `st2 validate`, `agents`, `up --once`, message, and task observation surfaces plus -isolated exec and PTY state. They do not import st2 source modules or treat unit tests as conformance evidence. +The probes use only public `st2 validate`, `ls`, `agents`, `tasks`, `up --once`, message, and task observation +surfaces plus isolated exec, process-generation, and PTY state. The F14 arms compare compact argv lowering with +an explicit PTY and send a real inbox message through both native DING sidecars; the F06-F08 oracle requires one durable stable-ID event with class, +affected paths or Resource descriptors, post-commit visibility, coalescing, replay idempotence, unchanged +silence, and no file or secret bytes. The probes do not import st2 source modules or treat unit tests as +conformance evidence. ## Gates -- `judges/self-test.sh` plants missing-field, unknown-gap, expected-red-as-pass, result-drift, and residue - mutations against the static oracle. +- `judges/self-test.sh` plants missing-field, unknown-gap, expected-red-as-pass, stale-map-with-equal-counts, + result-drift, and residue mutations against the static oracle. - The focused eval runs all public-behavior probes, prints one receipt row per closed-set case, cleans every exact eval-owned process/session, and exits nonzero while any contract case is red. - `bin/check-corpus.sh` remains the model-free repository preflight and does not execute the product-red cell. diff --git a/cells/agent-spec-field-change-matrix/fixture/cases.tsv b/cells/agent-spec-field-change-matrix/fixture/cases.tsv index 80a6384..58bbe61 100644 --- a/cells/agent-spec-field-change-matrix/fixture/cases.tsv +++ b/cells/agent-spec-field-change-matrix/fixture/cases.tsv @@ -1,19 +1,19 @@ case_id fields expected gaps contract -source-noop-heals F01 RED G01 semantic no-op writes and reports nothing while independently dead work heals +source-noop-heals F01 PASS - real KDL-to-JSON path/form move preserves normalized effects while independently dead work heals identity-remove-add F02 RED G03 exact old identity is removed before the new identity boots -host-projection F03 RED G05 old and new hosts converge through independent local projections -invalid-type-refuses F04 PASS - unsupported type refuses before launch write or teardown -role-metadata-adopts F05 PASS - role-only change preserves the healthy generation without notification -workspace-survivor-event F06 RED G02,G04,G06 workspace survivor keeps its generation and receives one post-commit event -resource-survivor-event F07 RED G02,G06 Resource-only change preserves the generation and emits one post-commit event -render-survivor-event F08 RED G02,G06 changed render commits before one event while unchanged bytes emit none +host-projection F03 RED G05 independent old/new projections converge after both overlap-first and absence-first orders without a shared receipt +invalid-type-refuses F04 RED G01 unsupported type refuses validation and reconciliation before launch write or teardown +role-metadata-adopts F05 RED G07 role-only change preserves the generation and becomes observable declaration metadata without notification +workspace-survivor-event F06 RED G02,G04,G06 workspace survivor keeps its generation and gets one durable coalesced privacy-safe post-commit event +resource-survivor-event F07 RED G02,G06 Resource field changes commit before one durable coalesced privacy-safe event and unchanged replay stays quiet +render-survivor-event F08 RED G02,G06 two changed render targets commit before one durable coalesced privacy-safe event and unchanged bytes stay quiet task-set-remove F09 RED G03 removed child is cleaned by exact ownership without changing its sibling task-id-change F10 RED G03 exact old task id is removed before the new id boots spawn-drift-visible F11 RED G03,G04 healthy spawn mismatch stays live and reports desired and observed fingerprints invalid-policy-refuses F12 RED G03 invalid restart or lifecycle policy refuses instead of defaulting retire-removed-child F13 RED G03 retirement stops every owned child including one removed in the same revision -compact-lowering-equal F14 PASS - compact command and ding lower to the same public task contract as explicit tasks -provider-fields-core-noop F15 PASS - provider-only fields do not change core equality launch or wake behavior -invalid-agent-isolated F16 RED G01,G07 invalid related work refuses while an independent valid agent progresses +compact-lowering-equal F14 PASS - compact argv env cwd lifecycle equal the explicit PTY while native DING lowering delivers a real message +provider-fields-core-noop F15 PASS - live provider-only mutation preserves generation and launch fingerprint with zero wake event or action +invalid-agent-isolated F16 PASS - distinct invalid last-known-good generation is held while an unrelated valid runtime id boots plan-dry-run-order R05 RED G07 true dry-run reports the fixed action order and writes nothing moved-intent-refusal MOVED RED G08 unsupported moved intent rejects cycles conflicts and host changes diff --git a/cells/agent-spec-field-change-matrix/fixture/matrix.sh b/cells/agent-spec-field-change-matrix/fixture/matrix.sh index 7470754..8d33a41 100755 --- a/cells/agent-spec-field-change-matrix/fixture/matrix.sh +++ b/cells/agent-spec-field-change-matrix/fixture/matrix.sh @@ -147,6 +147,15 @@ task_pid() { '.tasks[]? | select(.runtimeId == $runtime_id) | .runtime.pid // empty' } +task_generation() { + id="$1" + host="$2" + runtime_id="$3" + tasks_json "$id" "$host" 2>/dev/null | + jq -r --arg runtime_id "$runtime_id" \ + '.tasks[]? | select(.runtimeId == $runtime_id) | .runtime.generationId // empty' +} + wait_pid() { id="$1" host="$2" @@ -171,6 +180,15 @@ wait_dead() { return 1 } +wait_file() { + path="$1" + for _ in $(seq 1 100); do + test -s "$path" && return 0 + sleep 0.05 + done + return 1 +} + stop_exact_pid() { pid="$1" case "$pid" in @@ -180,6 +198,20 @@ stop_exact_pid() { wait_dead "$pid" } +process_generation_token() { + pid="$1" + start_ticks="$(awk '{print $22}' "/proc/$pid/stat")" + printf '%s:%s\n' "$pid" "$start_ticks" +} + +same_file_contents() { + first="$1" + second="$2" + first_sha="$(sha256sum "$first" | awk '{print $1}')" + second_sha="$(sha256sum "$second" | awk '{print $1}')" + test "$first_sha" = "$second_sha" +} + message_count() { id="$1" identity="$2" @@ -187,36 +219,145 @@ message_count() { jq 'length' 2>/dev/null || printf '0\n' } +assert_no_reconcile_action() { + output="$1" + ! grep -Eq '^ (launched|torn down|gc|held|flapping) \(' "$output" +} + +capture_change_event() { + id="$1" + host="$2" + identity="$3" + change_class="$4" + output="$5" + shift 5 + dir="$(case_dir "$id")" + rows="$(case_env "$id" st2 message ls "$host.$identity" --json)" || return 1 + test "$(printf '%s\n' "$rows" | jq 'length')" -eq 1 || { + printf 'expected exactly one %s event for %s, observed: %s\n' \ + "$change_class" "$host.$identity" "$rows" >&2 + return 1 + } + filename="$(printf '%s\n' "$rows" | jq -r '.[0].filename')" + [[ "$filename" =~ ^[0-9]{13}-[0-9a-z]{6}\.md$ ]] || return 1 + inbox="$dir/catalog/agents/$host/$identity/resources/inbox/$filename" + test -s "$inbox" || return 1 + case_env "$id" st2 message read "$host.$identity" "$filename" --json >"$output" || return 1 + jq -r '[.filename, .from, .subject, (.tags[]?), .body] | + map(select(type == "string" and length > 0)) | join("\n")' "$output" >"$output.text" || return 1 + tr '[:upper:]' '[:lower:]' <"$output.text" >"$output.lower" || return 1 + grep -Fq "$change_class" "$output.lower" || return 1 + grep -Fq "$host.$identity" "$output.text" || return 1 + for expected in "$@"; do + grep -Fq "$expected" "$output.text" || return 1 + done + printf '%s\n' "$filename" +} + +assert_event_omits() { + output="$1" + shift + for forbidden in "$@"; do + if grep -Fq "$forbidden" "$output.text"; then + return 1 + fi + done +} + +archive_and_replay_event() { + id="$1" + host="$2" + identity="$3" + filename="$4" + dir="$(case_dir "$id")" + agent_dir="$dir/catalog/agents/$host/$identity" + case_env "$id" st2 message archive "$host.$identity" "$filename" >/dev/null || return 1 + test -s "$agent_dir/resources/archive/$filename" || return 1 + cp "$agent_dir/resources/archive/$filename" "$agent_dir/resources/inbox/$filename" || return 1 + test "$(message_count "$id" "$host.$identity")" -eq 0 || return 1 + case_env "$id" st2 message archive "$host.$identity" "$filename" >/dev/null || return 1 + test ! -e "$agent_dir/resources/inbox/$filename" +} + +cleanup_case() { + id="$1" + dir="$(case_dir "$id")" + while IFS= read -r pid_file; do + pid="$(cat "$pid_file" 2>/dev/null || true)" + case "$pid" in + ''|*[!0-9]*) continue ;; + esac + kill -0 "$pid" 2>/dev/null && stop_exact_pid "$pid" || true + done < <(find "$dir" -type f -path '*/state/st2/*/exec/*.pid' -print 2>/dev/null) + while IFS= read -r pty_root; do + while IFS= read -r name; do + test -n "$name" || continue + PTY_ROOT="$pty_root" pty kill "$name" >/dev/null 2>&1 || true + PTY_ROOT="$pty_root" pty rm "$name" >/dev/null 2>&1 || true + done < <(PTY_ROOT="$pty_root" pty list --json 2>/dev/null | jq -r '.[].name' 2>/dev/null || true) + test "$(PTY_ROOT="$pty_root" pty list --json 2>/dev/null | jq 'length')" -eq 0 + done < <(find "$dir" -type d -name pty -print 2>/dev/null) +} + probe_source_noop_heals() { id="source-noop-heals" host="fm01" identity="agent" runtime_id="$host.$identity.work" initialize_case "$id" dir="$(case_dir "$id")" spec="$(write_single_exec "$id" "$host" "$identity" worker service "$dir/work-a" work "$runtime_id" noop)" - case_env "$id" st2 up --catalog "$dir/catalog" --host "$host" --interval 30 \ - >"$dir/supervisor.out" 2>"$dir/supervisor.err" & - supervisor_pid="$!" - trap 'kill "$supervisor_pid" >/dev/null 2>&1 || true; wait "$supervisor_pid" >/dev/null 2>&1 || true' EXIT + run_once "$id" "$host" initial pid_before="$(wait_pid "$id" "$host" "$runtime_id")" - adopted_before="$(grep -Fc 'adopted' "$dir/supervisor.out" 2>/dev/null || true)" - printf '// live semantic source-only no-op\n' >>"$spec" - sleep 1 - adopted_after="$(grep -Fc 'adopted' "$dir/supervisor.out" 2>/dev/null || true)" + generation_before="$(task_generation "$id" "$host" "$runtime_id")" + test -n "$generation_before" + + # Move both source path and source form while keeping every normalized effect equal. Explicit + # workspace and cwd values ensure the old KDL path is not an implicit launch input. + moved="$dir/moved.json.tmp" + jq -n \ + --arg identity "$identity" \ + --arg host "$host" \ + --arg workspace "$dir/work-a" \ + --arg runtime_id "$runtime_id" \ + --arg command "exec bash \"$dir/task.sh\" \"noop\" \"$dir/noop-count\"" \ + '{ + identity: $identity, + host: $host, + role: "worker", + type: "service", + workspace: $workspace, + exec: { + work: { + id: $runtime_id, + command: $command, + cwd: $workspace + } + } + }' >"$moved" + rm -f "$spec" + mkdir -p "$dir/catalog/moved-source" + mv "$moved" "$dir/catalog/moved-source/agent.json" + + run_once "$id" "$host" source-move pid_live="$(wait_pid "$id" "$host" "$runtime_id")" + generation_live="$(task_generation "$id" "$host" "$runtime_id")" + case_env "$id" st2 ls --catalog "$dir/catalog" >"$dir/source-move.ls" + grep -Fq "$dir/catalog/moved-source/agent.json" "$dir/source-move.ls" + grep -Fq "command exec bash \"$dir/task.sh\" \"noop\" \"$dir/noop-count\"" "$dir/source-move.ls" + grep -Fq "adopted (1): $identity" "$dir/source-move.out" + assert_no_reconcile_action "$dir/source-move.out" + test "$pid_live" = "$pid_before" + test "$generation_live" = "$generation_before" + test "$(cat "$dir/noop-count")" -eq 1 + test "$(message_count "$id" "$host.$identity")" -eq 0 + + # A source-only semantic no-op must not suppress independent healing of the exact dead task. stop_exact_pid "$pid_before" - printf '// independently dead work may heal\n' >>"$spec" - for _ in $(seq 1 100); do - test -f "$dir/noop-count" && test "$(cat "$dir/noop-count")" -eq 2 && break - sleep 0.05 - done + run_once "$id" "$host" heal pid_after="$(wait_pid "$id" "$host" "$runtime_id")" - kill "$supervisor_pid" >/dev/null 2>&1 || true - wait "$supervisor_pid" >/dev/null 2>&1 || true - trap - EXIT - test "$pid_live" = "$pid_before" + generation_after="$(task_generation "$id" "$host" "$runtime_id")" test "$pid_after" != "$pid_before" + test "$generation_after" != "$generation_before" test "$(cat "$dir/noop-count")" -eq 2 test "$(message_count "$id" "$host.$identity")" -eq 0 - test "$adopted_after" -eq "$adopted_before" } probe_identity_remove_add() { @@ -233,19 +374,80 @@ probe_identity_remove_add() { wait_dead "$old_pid" } +host_projection_overlap() { + old_id="host-projection/overlap/old" + new_id="host-projection/overlap/new" + identity="agent" + initialize_case "$old_id" + initialize_case "$new_id" + old_dir="$(case_dir "$old_id")" + new_dir="$(case_dir "$new_id")" + old_spec="$(write_single_exec "$old_id" old "$identity" worker service "$old_dir/work-a" work - oldhost)" + run_once "$old_id" old initial-old + old_pid="$(wait_pid "$old_id" old "old.$identity.work")" + test "$(PTY_ROOT="$new_dir/pty" pty list --json | jq 'length')" -eq 0 + + # New projection arrives first: overlap is allowed, and neither host consults a shared receipt. + write_single_exec "$new_id" new "$identity" worker service "$new_dir/work-a" work - newhost >/dev/null + run_once "$new_id" new add-new + new_pid="$(wait_pid "$new_id" new "new.$identity.work")" + new_generation="$(task_generation "$new_id" new "new.$identity.work")" + kill -0 "$old_pid" + kill -0 "$new_pid" + test "$old_dir/state" != "$new_dir/state" + test "$old_dir/pty" != "$new_dir/pty" + + rm -f "$old_spec" + run_once "$old_id" old converge-remove-old + removed=0 + wait_dead "$old_pid" || removed=1 + new_after="$(wait_pid "$new_id" new "new.$identity.work")" + new_generation_after="$(task_generation "$new_id" new "new.$identity.work")" + test "$new_after" = "$new_pid" + test "$new_generation_after" = "$new_generation" + test "$removed" -eq 0 +} + +host_projection_absence() { + old_id="host-projection/absence/old" + new_id="host-projection/absence/new" + identity="agent" + initialize_case "$old_id" + initialize_case "$new_id" + old_dir="$(case_dir "$old_id")" + new_dir="$(case_dir "$new_id")" + old_spec="$(write_single_exec "$old_id" old "$identity" worker service "$old_dir/work-a" work - oldhost)" + run_once "$old_id" old initial-old + old_pid="$(wait_pid "$old_id" old "old.$identity.work")" + + # Old projection leaves first: temporary global absence is allowed before the new host catches up. + rm -f "$old_spec" + run_once "$old_id" old remove-old-first + removed=0 + wait_dead "$old_pid" || removed=1 + new_absent="$(PTY_ROOT="$new_dir/pty" pty list --json | jq 'length')" + + write_single_exec "$new_id" new "$identity" worker service "$new_dir/work-a" work - newhost >/dev/null + run_once "$new_id" new converge-add-new + new_pid="$(wait_pid "$new_id" new "new.$identity.work")" + test "$new_absent" -eq 0 + kill -0 "$new_pid" + test "$old_dir/state" != "$new_dir/state" + test "$old_dir/pty" != "$new_dir/pty" + test "$removed" -eq 0 +} + probe_host_projection() { - id="host-projection" identity="agent" - initialize_case "$id" + id="host-projection" dir="$(case_dir "$id")" - spec="$(write_single_exec "$id" old "$identity" worker service "$dir/work-a" work - oldhost)" - run_once "$id" old old - old_pid="$(wait_pid "$id" old "old.$identity.work")" - rm -f "$spec" - write_single_exec "$id" new "$identity" worker service "$dir/work-a" work - newhost >/dev/null - run_once "$id" old remove-old - run_once "$id" new add-new - wait_pid "$id" new "new.$identity.work" >/dev/null - wait_dead "$old_pid" + mkdir -p "$dir" + overlap_rc=0 + absence_rc=0 + (set -euo pipefail; host_projection_overlap) >"$dir/overlap.log" 2>&1 || overlap_rc=$? + (set -euo pipefail; host_projection_absence) >"$dir/absence.log" 2>&1 || absence_rc=$? + printf 'overlap_rc=%s absence_rc=%s\n' "$overlap_rc" "$absence_rc" + test "$overlap_rc" -eq 0 + test "$absence_rc" -eq 0 } probe_invalid_type_refuses() { @@ -253,10 +455,14 @@ probe_invalid_type_refuses() { initialize_case "$id" dir="$(case_dir "$id")" write_single_exec "$id" "$host" agent worker invalid-f102 "$dir/work-a" work "$host.agent.work" invalid >/dev/null - if validate_case "$id" "$host" validate; then - return 1 - fi + validate_rc=0 + validate_case "$id" "$host" validate || validate_rc=$? + run_rc=0 + run_once "$id" "$host" reconcile || run_rc=$? + test "$validate_rc" -ne 0 + test "$run_rc" -ne 0 test -z "$(find "$dir/state" -type f -name '*.pid' -print -quit)" + test ! -e "$dir/invalid-count" } probe_role_metadata_adopts() { @@ -266,25 +472,61 @@ probe_role_metadata_adopts() { spec="$(write_single_exec "$id" "$host" "$identity" worker service "$dir/work-a" work "$runtime_id" role)" run_once "$id" "$host" launch pid_before="$(wait_pid "$id" "$host" "$runtime_id")" + generation_before="$(task_generation "$id" "$host" "$runtime_id")" + role_before="$(case_env "$id" st2 agents --json --catalog "$dir/catalog" | + jq -r --arg identity "$host.$identity" '.[] | select(.identity == $identity) | .role // "__missing__"')" sed -i 's/role "worker"/role "reviewer"/' "$spec" run_once "$id" "$host" role pid_after="$(wait_pid "$id" "$host" "$runtime_id")" + generation_after="$(task_generation "$id" "$host" "$runtime_id")" + role_after="$(case_env "$id" st2 agents --json --catalog "$dir/catalog" | + jq -r --arg identity "$host.$identity" '.[] | select(.identity == $identity) | .role // "__missing__"')" test "$pid_after" = "$pid_before" + test "$generation_after" = "$generation_before" + test "$(cat "$dir/role-count")" -eq 1 test "$(message_count "$id" "$host.$identity")" -eq 0 + grep -Fq "adopted (1): $identity" "$dir/role.out" + assert_no_reconcile_action "$dir/role.out" + test "$role_before" = worker + test "$role_after" = reviewer } probe_workspace_survivor_event() { id="workspace-survivor-event" host="fm06" identity="agent" runtime_id="$host.$identity.work" initialize_case "$id" dir="$(case_dir "$id")" + printf 'WORKSPACE_SECRET_OLD_F102\n' >"$dir/work-a/private.txt" + printf 'WORKSPACE_SECRET_NEW_F102\n' >"$dir/work-b/private.txt" spec="$(write_single_exec "$id" "$host" "$identity" worker service "$dir/work-a" work "$runtime_id" workspace)" run_once "$id" "$host" launch pid_before="$(wait_pid "$id" "$host" "$runtime_id")" + generation_before="$(task_generation "$id" "$host" "$runtime_id")" sed -i "s|workspace \"$dir/work-a\"|workspace \"$dir/work-b\"|" "$spec" run_once "$id" "$host" workspace pid_after="$(wait_pid "$id" "$host" "$runtime_id")" + generation_after="$(task_generation "$id" "$host" "$runtime_id")" test "$pid_after" = "$pid_before" - test "$(message_count "$id" "$host.$identity")" -eq 1 + test "$generation_after" = "$generation_before" + test "$(cat "$dir/workspace-count")" -eq 1 + case_env "$id" st2 ls --catalog "$dir/catalog" >"$dir/workspace.ls" + visibility_rc=0 + grep -Fq "$dir/work-b" "$dir/workspace.ls" || visibility_rc=$? + event_rc=0 + filename="$(capture_change_event "$id" "$host" "$identity" workspace "$dir/event.json" \ + "$dir/work-a" "$dir/work-b")" || event_rc=$? + if test "$event_rc" -eq 0; then + assert_event_omits "$dir/event.json" \ + WORKSPACE_SECRET_OLD_F102 WORKSPACE_SECRET_NEW_F102 || event_rc=$? + if test "$event_rc" -eq 0; then + archive_and_replay_event "$id" "$host" "$identity" "$filename" || event_rc=$? + fi + fi + run_once "$id" "$host" unchanged + test "$(message_count "$id" "$host.$identity")" -eq 0 + test "$(wait_pid "$id" "$host" "$runtime_id")" = "$pid_before" + printf 'post-commit-visibility-rc=%s event-oracle-rc=%s\n' "$visibility_rc" "$event_rc" + test "$visibility_rc" -eq 0 + test "$event_rc" -eq 0 } probe_resource_survivor_event() { @@ -292,35 +534,89 @@ probe_resource_survivor_event() { initialize_case "$id" dir="$(case_dir "$id")" spec="$(write_single_exec "$id" "$host" "$identity" worker service "$dir/work-a" work "$runtime_id" resource \ - ' resource "work" _tag="issue" uri="issue://field/old"')" + ' resource "old-work" _tag="issue" uri="issue://field/old"')" run_once "$id" "$host" launch pid_before="$(wait_pid "$id" "$host" "$runtime_id")" - sed -i 's|issue://field/old|issue://field/new|' "$spec" + generation_before="$(task_generation "$id" "$host" "$runtime_id")" + sed -i 's|resource "old-work" _tag="issue" uri="issue://field/old"|resource "new-work" _tag="task" uri="task://field/new"|' "$spec" run_once "$id" "$host" resource pid_after="$(wait_pid "$id" "$host" "$runtime_id")" + generation_after="$(task_generation "$id" "$host" "$runtime_id")" test "$pid_after" = "$pid_before" + test "$generation_after" = "$generation_before" + test "$(cat "$dir/resource-count")" -eq 1 case_env "$id" st2 agents --json --catalog "$dir/catalog" | - jq -e '.[] | select(.identity == "fm07.agent") | .resources[] | select(.uri == "issue://field/new")' >/dev/null - test "$(message_count "$id" "$host.$identity")" -eq 1 + jq -e '.[] | select(.identity == "fm07.agent") | + (.resources | length == 1) and + (.resources[0] == {"name":"new-work","_tag":"task","uri":"task://field/new"})' >/dev/null + event_rc=0 + filename="$(capture_change_event "$id" "$host" "$identity" resource "$dir/event.json" \ + old-work new-work issue://field/old task://field/new)" || event_rc=$? + if test "$event_rc" -eq 0; then + assert_event_omits "$dir/event.json" RESOURCE_SECRET_F102 || event_rc=$? + if test "$event_rc" -eq 0; then + archive_and_replay_event "$id" "$host" "$identity" "$filename" || event_rc=$? + fi + fi + run_once "$id" "$host" unchanged + test "$(message_count "$id" "$host.$identity")" -eq 0 + test "$(wait_pid "$id" "$host" "$runtime_id")" = "$pid_before" + printf 'event-oracle-rc=%s\n' "$event_rc" + test "$event_rc" -eq 0 } probe_render_survivor_event() { id="render-survivor-event" host="fm08" identity="agent" runtime_id="$host.$identity.work" initialize_case "$id" dir="$(case_dir "$id")" - spec="$(write_single_exec "$id" "$host" "$identity" worker service "$dir/work-a" work "$runtime_id" render \ - $' render {\n file "context.txt" "A-f102"\n }')" + spec="$dir/catalog/agents/$host/$identity/agent.kdl" + mkdir -p "$(dirname "$spec")" + cat >"$spec" <"$compact" <"$spec" <"$explicit" <"$spec" <"$dir/compact.json" - CATALOG="$dir/catalog/explicit" XDG_STATE_HOME="$dir/state/explicit" PTY_ROOT="$dir/pty/explicit" \ - st2 tasks --json --catalog "$dir/catalog/explicit" --host "$host" | - jq -S '[.tasks[] | {agent,task,runtimeId,kind,lifecycle,retired,desiredState}]' >"$dir/explicit.json" - test "$(cat "$dir/compact.json")" = "$(cat "$dir/explicit.json")" - test "$(jq 'length' "$dir/compact.json")" -eq 2 + fi + + rm -f "$workspace/launch.tsv" "$workspace/ding.txt" + run_once "$id" "$host" launch + wait_file "$workspace/launch.tsv" + agent_pid="$(wait_pid "$id" "$host" "$host.$identity")" + ding_pid="$(wait_pid "$id" "$host" "$host.$identity.ding")" + kill -0 "$agent_pid" + kill -0 "$ding_pid" + cp "$workspace/launch.tsv" "$base/$arm.launch.tsv" + tasks_json "$id" "$host" | + jq -S '[.tasks[] | {agent,task,runtimeId,kind,lifecycle,retired,desiredState}]' >"$base/$arm.tasks.json" + case_env "$id" st2 ls --catalog "$dir/catalog" | + grep '^ - ' >"$base/$arm.lowering.txt" + PTY_ROOT="$dir/pty" pty list --json | + jq -S --arg runtime_id "$host.$identity" \ + '[.[] | select(.name == $runtime_id) | {command,cwd}]' >"$base/$arm.pty-launch.json" + tr '\0' '\n' <"/proc/$ding_pid/environ" | grep '^F14_VALUE=' >"$base/$arm.ding-env.txt" + + case_env "$id" st2 message send "$host.$identity" --as "$host.sender" \ + --subject "F14 lowering" -m "same lowering payload" >/dev/null + wait_file "$workspace/ding.txt" + sed -E 's/\[id:[0-9a-z]{6}\]/[id:STABLE]/' "$workspace/ding.txt" >"$base/$arm.ding.txt" + cleanup_case "$id" +} + +probe_compact_lowering_equal() { + id="compact-lowering-equal" + dir="$(case_dir "$id")" + mkdir -p "$dir/work-a" + cat >"$dir/f14-agent.sh" <<'SH' +#!/usr/bin/env bash +set -euo pipefail +{ + printf 'script=%s\n' "$0" + printf 'arg=%s\n' "${1:-}" + printf 'env=%s\n' "${F14_VALUE:-}" + printf 'cwd=%s\n' "$PWD" +} >"$PWD/launch.tsv" +while IFS= read -r line; do + printf '%s\n' "$line" >>"$PWD/ding.txt" +done +SH + chmod +x "$dir/f14-agent.sh" + + run_compact_lowering_arm compact + run_compact_lowering_arm explicit + id="compact-lowering-equal" + dir="$(case_dir "$id")" + for surface in launch.tsv tasks.json lowering.txt pty-launch.json ding-env.txt ding.txt; do + same_file_contents "$dir/compact.$surface" "$dir/explicit.$surface" + done + test "$(jq 'length' "$dir/compact.tasks.json")" -eq 2 + grep -Fq $'arg=argument with space' "$dir/compact.launch.tsv" + grep -Fq $'env=equal-f102' "$dir/compact.launch.tsv" + grep -Fq "cwd=$dir/work-a" "$dir/compact.launch.tsv" + grep -Fq '[DING] new st2 message: [id:STABLE] F14 lowering (from fm14.sender); check your inbox' \ + "$dir/compact.ding.txt" } probe_provider_fields_core_noop() { - id="provider-fields-core-noop" host="fm15" identity="agent" + id="provider-fields-core-noop" host="fm15" identity="agent" runtime_id="$host.$identity.work" initialize_case "$id" dir="$(case_dir "$id")" - for arm in base provider; do - spec="$dir/catalog/$arm/agents/$host/$identity/agent.kdl" - mkdir -p "$(dirname "$spec")" - { - printf 'agent "%s" {\n' "$identity" - printf ' host "%s"\n' "$host" - printf ' role "worker"\n' - printf ' type "service"\n' - printf ' workspace "%s"\n' "$dir/work-a" - if test "$arm" = provider; then - printf ' harness "codex"\n model "provider-model"\n persona "provider-persona"\n' - printf ' permissions "provider-policy"\n transport "provider-transport"\n' - printf ' strategy "provider-strategy"\n meta { opaque "value" }\n' - fi - printf ' exec "work" {\n' - printf ' id "%s.%s.work"\n' "$host" "$identity" - printf ' command "true"\n' - printf ' }\n' - printf '}\n' - } >"$spec" - CATALOG="$dir/catalog/$arm" XDG_STATE_HOME="$dir/state/$arm" PTY_ROOT="$dir/pty/$arm" \ - st2 validate --catalog "$dir/catalog/$arm" --host "$host" >"$dir/$arm.validate" 2>&1 - CATALOG="$dir/catalog/$arm" XDG_STATE_HOME="$dir/state/$arm" PTY_ROOT="$dir/pty/$arm" \ - st2 tasks --json --catalog "$dir/catalog/$arm" --host "$host" | - jq -S '[.tasks[] | {agent,task,runtimeId,kind,lifecycle,retired,desiredState}]' >"$dir/$arm.tasks" - CATALOG="$dir/catalog/$arm" XDG_STATE_HOME="$dir/state/$arm" PTY_ROOT="$dir/pty/$arm" \ - st2 agents --json --catalog "$dir/catalog/$arm" | - jq -S '[.[] | {identity,retired,resources}]' >"$dir/$arm.agents" - done - test "$(cat "$dir/base.tasks")" = "$(cat "$dir/provider.tasks")" - test "$(cat "$dir/base.agents")" = "$(cat "$dir/provider.agents")" + cat >"$dir/f15-task.sh" <<'SH' +#!/usr/bin/env bash +set -euo pipefail +{ + printf 'script=%s\n' "$0" + printf 'arg=%s\n' "${1:-}" + printf 'env=%s\n' "${F15_VALUE:-}" + printf 'cwd=%s\n' "$PWD" +} >"$PWD/launch.tsv" +trap 'exit 0' TERM INT +while :; do sleep 1; done +SH + chmod +x "$dir/f15-task.sh" + spec="$dir/catalog/agents/$host/$identity/agent.kdl" + mkdir -p "$(dirname "$spec")" + cat >"$spec" <"$dir/before.tasks.json" + + sed -i 's/provider-a/provider-b/g' "$spec" + run_once "$id" "$host" provider-fields + pid_after="$(wait_pid "$id" "$host" "$runtime_id")" + generation_after="$(task_generation "$id" "$host" "$runtime_id")" + process_after="$(process_generation_token "$pid_after")" + fingerprint_after="$(sha256sum "$dir/work-a/launch.tsv" | awk '{print $1}')" + tasks_json "$id" "$host" | jq -S . >"$dir/after.tasks.json" + + test "$pid_after" = "$pid_before" + test "$generation_after" = "$generation_before" + test "$process_after" = "$process_before" + test "$fingerprint_after" = "$fingerprint_before" + same_file_contents "$dir/before.tasks.json" "$dir/after.tasks.json" + grep -Fq "adopted (1): $identity" "$dir/provider-fields.out" + assert_no_reconcile_action "$dir/provider-fields.out" + test "$(message_count "$id" "$host.$identity")" -eq 0 } probe_invalid_agent_isolated() { id="invalid-agent-isolated" host="fm16" initialize_case "$id" dir="$(case_dir "$id")" - write_single_exec "$id" "$host" valid worker service "$dir/work-a" work "$host.valid.work" valid >/dev/null - write_single_exec "$id" "$host" invalid worker service "$dir/work-b" work "$host.valid.work" invalid >/dev/null - set +e - run_once "$id" "$host" mixed - rc=$? - set -e - test "$rc" -ne 0 - wait_pid "$id" "$host" "$host.valid.work" >/dev/null + held_spec="$(write_single_exec "$id" "$host" held worker service "$dir/work-a" work \ + "$host.held.work" held '' ' lifecycle "service"')" + run_once "$id" "$host" initial-held + held_pid="$(wait_pid "$id" "$host" "$host.held.work")" + held_generation="$(task_generation "$id" "$host" "$host.held.work")" + held_process="$(process_generation_token "$held_pid")" + + # Corrupt only the held agent and add one independently valid agent with a distinct runtime id. + sed -i 's/lifecycle "service"/lifecycle "invalid-f102"/' "$held_spec" + write_single_exec "$id" "$host" valid worker service "$dir/work-b" work \ + "$host.valid.work" valid >/dev/null + mixed_rc=0 + run_once "$id" "$host" mixed || mixed_rc=$? + valid_pid="$(wait_pid "$id" "$host" "$host.valid.work")" + + kill -0 "$held_pid" + test "$(process_generation_token "$held_pid")" = "$held_process" + test "$(cat "$dir/held-count")" -eq 1 + test "$(cat "$dir/valid-count")" -eq 1 + test "$host.held.work" != "$host.valid.work" + kill -0 "$valid_pid" + grep -Fq "unknown lifecycle 'invalid-f102'" "$dir/mixed.err" + grep -Fq "$host.valid.work" "$dir/mixed.out" + test "$(message_count "$id" "$host.held")" -eq 0 + test "$(message_count "$id" "$host.valid")" -eq 0 + printf 'partial-run-exit=%s held-generation=%s\n' "$mixed_rc" "$held_generation" >"$dir/isolation-receipt.txt" } probe_plan_dry_run_order() { diff --git a/cells/agent-spec-field-change-matrix/judges/self-test.sh b/cells/agent-spec-field-change-matrix/judges/self-test.sh index efe84e8..73c12d5 100755 --- a/cells/agent-spec-field-change-matrix/judges/self-test.sh +++ b/cells/agent-spec-field-change-matrix/judges/self-test.sh @@ -27,13 +27,24 @@ fi echo "PASS: missing-field mutation fails coverage" unknown_gap="$tmp/unknown-gap.tsv" -sed 's/F01\tRED\tG01/F01\tRED\tG09/' "$manifest" >"$unknown_gap" +sed 's/F04\tRED\tG01/F04\tRED\tG09/' "$manifest" >"$unknown_gap" if bash "$oracle" gaps "$good" "$unknown_gap" >/dev/null 2>&1; then echo "FAIL: unknown G09 gap passed mapping" >&2 exit 1 fi echo "PASS: unknown-gap mutation fails mapping" +stale_map="$tmp/stale-map.out" +sed \ + -e $'s/RESULT\tsource-noop-heals\tF01\tPASS\t-\tPASS/RESULT\tsource-noop-heals\tF01\tRED\tG01\tRED/' \ + -e $'s/RESULT\tinvalid-type-refuses\tF04\tRED\tG01\tRED/RESULT\tinvalid-type-refuses\tF04\tPASS\t-\tPASS/' \ + "$good" >"$stale_map" +if bash "$oracle" classification "$stale_map" "$manifest" >/dev/null 2>&1; then + echo "FAIL: stale pass/red membership with unchanged 4/14 totals passed classification" >&2 + exit 1 +fi +echo "PASS: stale pass/red membership fails even when aggregate counts stay 4/14" + red_as_pass="$tmp/red-as-pass.out" sed $'s/RESULT\tidentity-remove-add\tF02\tRED\tG03\tRED/RESULT\tidentity-remove-add\tF02\tRED\tG03\tPASS/' \ "$good" >"$red_as_pass" From cfd8007549ad78c74236580b1b7e0cf74eaa267f Mon Sep 17 00:00:00 2001 From: Nathan Date: Fri, 31 Jul 2026 16:18:55 +0200 Subject: [PATCH 3/4] test: close field-change review gaps --- cells/agent-spec-field-change-matrix/fixture/matrix.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/cells/agent-spec-field-change-matrix/fixture/matrix.sh b/cells/agent-spec-field-change-matrix/fixture/matrix.sh index 8d33a41..1c9af82 100755 --- a/cells/agent-spec-field-change-matrix/fixture/matrix.sh +++ b/cells/agent-spec-field-change-matrix/fixture/matrix.sh @@ -221,7 +221,7 @@ message_count() { assert_no_reconcile_action() { output="$1" - ! grep -Eq '^ (launched|torn down|gc|held|flapping) \(' "$output" + ! grep -Eq '^ (launched|restarted|torn down|gc|held|flapping) \(' "$output" } capture_change_event() { @@ -533,6 +533,7 @@ probe_resource_survivor_event() { id="resource-survivor-event" host="fm07" identity="agent" runtime_id="$host.$identity.work" initialize_case "$id" dir="$(case_dir "$id")" + printf 'RESOURCE_SECRET_F102\n' >"$dir/work-a/private.txt" spec="$(write_single_exec "$id" "$host" "$identity" worker service "$dir/work-a" work "$runtime_id" resource \ ' resource "old-work" _tag="issue" uri="issue://field/old"')" run_once "$id" "$host" launch From 04171e212d61a27e54df2581b35b01ca2b5f0d3a Mon Sep 17 00:00:00 2001 From: Nathan Herald Date: Sat, 1 Aug 2026 11:07:20 +0200 Subject: [PATCH 4/4] docs: explain expected product failures plainly --- cells/agent-spec-field-change-matrix/README.md | 14 ++++++++------ .../agent-spec-field-change-matrix.kdl | 2 +- 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/cells/agent-spec-field-change-matrix/README.md b/cells/agent-spec-field-change-matrix/README.md index caf15bd..b1d97fd 100644 --- a/cells/agent-spec-field-change-matrix/README.md +++ b/cells/agent-spec-field-change-matrix/README.md @@ -4,10 +4,11 @@ Contract-first, model-free acceptance matrix for the field-change rules in merge [`compoundingtech/st2#102`](https://github.com/compoundingtech/st2/pull/102) at exact merge [`e54a04a3ee8af6fe0b0bce4cd961f8188ac90525`](https://github.com/compoundingtech/st2/commit/e54a04a3ee8af6fe0b0bce4cd961f8188ac90525). -The cell is intentionally product-red. PR #102 changes documentation, not runtime behavior, and names eight -implementation gaps. A red case is accepted by the classification judges only when its public-CLI observation -matches the exact frozen map below; it is never counted as a conformance pass. The run step remains red until -all contract cases pass and the manifest is deliberately advanced with matching product evidence. +This eval is expected to fail until st2 implements the documented field-change rules. PR #102 changes +documentation, not runtime behavior, and names eight implementation gaps. The classification judges accept a +known failure only when its public CLI result matches the exact map below. They never count a known failure as +a conformance pass. The run step continues to fail until all contract cases pass and matching product evidence +updates the manifest. **Capabilities required:** `st2,pty,jq,awk,grep,sed,coreutils`. No model and no provider. Every catalog, workspace, exec state, message, PTY root, process, and receipt is created below the eval-owned temporary root. @@ -37,9 +38,10 @@ conformance evidence. result-drift, and residue mutations against the static oracle. - The focused eval runs all public-behavior probes, prints one receipt row per closed-set case, cleans every exact eval-owned process/session, and exits nonzero while any contract case is red. -- `bin/check-corpus.sh` remains the model-free repository preflight and does not execute the product-red cell. +- `bin/check-corpus.sh` remains the model-free repository preflight and does not execute this expected-failure + cell. -Run the static oracle and the expected-red product matrix with: +Run the static oracle and the expected-failure product matrix with: ```sh bash ./cells/agent-spec-field-change-matrix/judges/self-test.sh diff --git a/cells/agent-spec-field-change-matrix/agent-spec-field-change-matrix.kdl b/cells/agent-spec-field-change-matrix/agent-spec-field-change-matrix.kdl index bd4c59e..e4abf99 100644 --- a/cells/agent-spec-field-change-matrix/agent-spec-field-change-matrix.kdl +++ b/cells/agent-spec-field-change-matrix/agent-spec-field-change-matrix.kdl @@ -1,5 +1,5 @@ // Contract-first, model-free acceptance for the field-change rules proposed by merged st2 PR #102. -// The run step is intentionally RED while the exact expected G01-G08 product gaps remain. +// The run step is expected to fail while the exact G01-G08 product gaps remain. eval { copy "./fixture" max-timeout "180s"