From 2cc93cc2f3b432d5f5a93b00629240584a54f27b Mon Sep 17 00:00:00 2001 From: schickling-assistant <261620128+schickling-assistant@users.noreply.github.com> Date: Mon, 3 Aug 2026 23:27:09 +0200 Subject: [PATCH] eval: cover launch method selection agent-tool: Codex agent-tool-version: 0.145.0 agent-runtime: Codex 0.145.0 agent-session-lookup: unavailable tooling-profile: dotfiles@unknown-dirty --- .gitignore | 1 + CATALOG.md | 6 +- cells/launch-method-selection/README.md | 52 +++ .../fixture/adapter.sh | 42 +++ .../launch-method-selection/fixture/cases.tsv | 6 + .../launch-method-selection/fixture/probe.sh | 338 ++++++++++++++++++ .../judges/contract.sh | 53 +++ .../judges/self-test.sh | 78 ++++ .../launch-method-selection.kdl | 28 ++ evidence/harness-exclusions.tsv | 1 + 10 files changed, 603 insertions(+), 2 deletions(-) create mode 100644 cells/launch-method-selection/README.md create mode 100755 cells/launch-method-selection/fixture/adapter.sh create mode 100644 cells/launch-method-selection/fixture/cases.tsv create mode 100755 cells/launch-method-selection/fixture/probe.sh create mode 100755 cells/launch-method-selection/judges/contract.sh create mode 100755 cells/launch-method-selection/judges/self-test.sh create mode 100644 cells/launch-method-selection/launch-method-selection.kdl diff --git a/.gitignore b/.gitignore index 41545a56..b2595245 100644 --- a/.gitignore +++ b/.gitignore @@ -3,6 +3,7 @@ .sandbox/ .personas/ .eval-runs/ +tmp/ # noise .DS_Store diff --git a/CATALOG.md b/CATALOG.md index a4f12707..14f0566c 100644 --- a/CATALOG.md +++ b/CATALOG.md @@ -1,7 +1,7 @@ # Current eval catalog and evidence -This is the sole current corpus surface: **59 included cells** (19 model-free, +This is the sole current corpus surface: **60 included cells** (20 model-free, 40 model-backed) and **4 retired exclusions**. Inventory fields are derived from the active KDL and executable judge scripts. Latest accepted PASS evidence remains distinct from the append-only run history: a failed last run is visible without being advertised as accepted evidence, and cells with no @@ -42,6 +42,7 @@ harness-native loader plus canonical hook file. | `host-lock-health-negatives` | model-free | — | 0 | none | `30s` | 5 | — | **NO STRUCTURED RUN** | | `inbox-hygiene` | Claude | `claude-sonnet-5` / medium | 1 | low | `900s` | 4 | — | **NO STRUCTURED RUN** | | `incident-response` | Claude | `claude-sonnet-5` / medium | 2 | medium | `1200s` | 5 | — | **NO STRUCTURED RUN** | +| `launch-method-selection` | model-free | — | 0 | none | `120s` | 5 | — | **NO STRUCTURED RUN** | | `license-mit` | mixed | `claude-sonnet-5+gpt-5.6-sol` / medium | 3 | high | `1200s` | 6 | — | **FAIL** 2026-07-28T13:27:18Z, 5/6, source [`b3cd5fb`](https://github.com/compoundingtech/evals/commit/b3cd5fbd98c11179a4555d0f9bbccfe98351a734), [receipt](evidence/stable-main-phase4-runs-20260728.json) | | `license-mit-codex` | Codex | `gpt-5.6-sol` / medium | 3 | high | `1200s` | 6 | **PASS** 2026-07-26, st2 `9d26245`, 1m39s, [`40ec23d`](https://github.com/compoundingtech/evals/commit/40ec23d3e068e5bd70b6d4b9a92a92d291868dce) | **NO STRUCTURED RUN** | | `managed-agent-color-env` | model-free | — | 0 | none | `90s` | 9 | — | **NO STRUCTURED RUN** | @@ -102,6 +103,7 @@ while the last-run column makes a recorded failure distinct from a cell with no - `exec-lifecycle-logging` / `cell` — deterministic native exec lifecycle probe; no harness seat - `hook-integrity` / `cell` — deterministic native hook-installation probe; no live harness seat - `host-lock-health-negatives` / `cell` — deterministic native doctor negative probe; no harness seat +- `launch-method-selection` / `cell` — deterministic proposed launch-method selection probe; no harness seat - `pty-attach-machine-stream` / `cell` — deterministic installed PTY attach composition probe; no harness seat - `managed-agent-color-env` / `cell` — deterministic managed-agent environment policy probe; no harness seat - `pty-attach-only` / `cell` — deterministic PTY dead-attach policy probe; no harness seat @@ -116,7 +118,7 @@ while the last-run column makes a recorded failure distinct from a cell with no - `targeted-reconcile-isolation` / `cell` — deterministic selected reconcile isolation probe; no bus-connected agent - `two-networks-coexist` / `cell` — deterministic catalog-isolation probe; no harness seat -These **20** rows are generated from structured exclusions. A cell-level row is required for +These **21** rows are generated from structured exclusions. A cell-level row is required for every derived model-free cell. The one-shot docs cold reader is a model-backed grader, but not a bus-connected agent: it has no st2 identity, DING sidecar, or lifecycle hooks. diff --git a/cells/launch-method-selection/README.md b/cells/launch-method-selection/README.md new file mode 100644 index 00000000..3d86e442 --- /dev/null +++ b/cells/launch-method-selection/README.md @@ -0,0 +1,52 @@ +# launch-method-selection + +Contract-first, model-free acceptance coverage for the proposed start/resume launch-method shape discussed in +[`compoundingtech/st2#124`](https://github.com/compoundingtech/st2/issues/124). The field names and defaults are +not ratified. This cell therefore remains proposal-tracking evidence and does not update `AGENT-SPEC.md`. + +The four proposed cases are expected product reds while the current st2 grammar exposes one top-level `argv`. +The legacy case is a positive control and must keep passing. Expected reds are classified evidence, never +conformance passes. + +**Capabilities required:** `st2,pty,jq,awk,grep,sed,coreutils`. No model or provider. Every catalog, synthetic +session record, workspace, durable marker, process, and PTY root lives below the eval-owned temporary catalog. + +## Closed-set cases + +| Case | Current result | Discriminator | +| --- | --- | --- | +| `explicit-start-new-session` | `RED/P01` | selects `start`, creates one new synthetic native ID, and preserves durable work | +| `explicit-resume-exact` | `RED/P01` | appends parsed conversation records only to the declared ID, never a newer neighbor | +| `unavailable-refuse` | `RED/P01` | refuses before either method launches or any session record changes | +| `unavailable-start` | `RED/P01` | selects `start` only when that unavailable fallback is declared | +| `legacy-single-argv` | `PASS` | preserves the existing single top-level `argv` launch behavior | + +`P01` means the proposed grammar is unavailable. Once a candidate parses it, behavioral failures are classified +as `P02` wrong method, `P03` wrong session, `P04` wrong unavailable policy, `P05` lost durable continuity, or +`P06` leaked eval-owned runtime state. + +The checked-in `AGENT-SPEC.md` remains pinned to an older portable runner and documents compact `command`. +Current st2 declarations use one top-level `argv`; this proposal replaces that cardinality invariant with a +successor that accepts either one legacy `argv` or launch methods, never an ambiguous mixture. + +## Evidence boundary + +The adapter writes synthetic JSONL conversation records and checks them by parsed `sessionId` and record type. +It never infers continuity from file bytes or mtime. It does not call Claude or Codex and does not touch their +homes or transcript stores. Provider-native facts remain the version-scoped evidence recorded on st2#124; +this cell covers only provider-neutral selection and refusal. + +The candidate preserves the optional `resume.session` pin in its public Rust `agent-spec` model, but the +current versioned `agents`/`tasks` CLI and launch receipts do not project it. This external behavior cell can +therefore prove that the selected resume argv continues the exact adversarially chosen session, but it cannot +independently prove parser-level pin preservation. The upstream model tests own that assertion until a public +wire surface is deliberately versioned; this cell does not invent one. + +`judges/self-test.sh` plants missing-case, duplicate-case, false-promotion, unknown-gap, false-honesty, and +cleanup-receipt mutations. The focused run remains red until every proposed case passes and the closed-set +manifest is updated. + +```sh +bash ./cells/launch-method-selection/judges/self-test.sh +st2 eval ./cells/launch-method-selection/ --keep +``` diff --git a/cells/launch-method-selection/fixture/adapter.sh b/cells/launch-method-selection/fixture/adapter.sh new file mode 100755 index 00000000..de54ebe1 --- /dev/null +++ b/cells/launch-method-selection/fixture/adapter.sh @@ -0,0 +1,42 @@ +#!/usr/bin/env bash +set -euo pipefail + +mode="${1:?mode is required}" +store="${2:?session store is required}" +durable="${3:?durable marker is required}" +receipt="${4:?launch receipt is required}" +requested_session="${5:-}" + +test "$(<"$durable")" = "DURABLE-WORK-CONTINUES-8d31" +mkdir -p "$store" + +fresh_uuid() { + seed="$(date +%s%N)-$$-$RANDOM-$store" + hex="$(printf '%s' "$seed" | sha256sum | cut -c1-32)" + printf '%s-%s-4%s-a%s-%s\n' \ + "${hex:0:8}" "${hex:8:4}" "${hex:13:3}" "${hex:17:3}" "${hex:20:12}" +} + +case "$mode" in + start) + session="$(fresh_uuid)" + printf '{"type":"conversation","sessionId":"%s","turn":"created"}\n' "$session" \ + > "$store/$session.jsonl" + ;; + resume) + session="${requested_session:?resume session is required}" + test -f "$store/$session.jsonl" + printf '{"type":"conversation","sessionId":"%s","turn":"resumed"}\n' "$session" \ + >> "$store/$session.jsonl" + ;; + legacy) + session="-" + ;; + *) + printf 'unknown adapter mode: %s\n' "$mode" >&2 + exit 2 + ;; +esac + +printf '%s\t%s\t%s\t%s\n' "$mode" "$session" "$(<"$durable")" "$$" > "$receipt" +exec tail -f /dev/null diff --git a/cells/launch-method-selection/fixture/cases.tsv b/cells/launch-method-selection/fixture/cases.tsv new file mode 100644 index 00000000..68d9b906 --- /dev/null +++ b/cells/launch-method-selection/fixture/cases.tsv @@ -0,0 +1,6 @@ +case_id expected gaps contract +explicit-start-new-session RED P01 explicit start creates a new synthetic native session while durable work remains intact +explicit-resume-exact RED P01 explicit resume appends only to the declared session and never to a newer neighbor +unavailable-refuse RED P01 an unavailable resume refuses without launching either method or mutating session state +unavailable-start RED P01 an unavailable resume degrades to start only when that fallback is declared +legacy-single-argv PASS - the legacy single top-level argv form retains its existing launch behavior diff --git a/cells/launch-method-selection/fixture/probe.sh b/cells/launch-method-selection/fixture/probe.sh new file mode 100755 index 00000000..fe5f6308 --- /dev/null +++ b/cells/launch-method-selection/fixture/probe.sh @@ -0,0 +1,338 @@ +#!/usr/bin/env bash +set -euo pipefail + +root="${CATALOG:?CATALOG must be set}" +template="$root/adapter.sh" +red_count=0 +pass_count=0 +declare -a scenario_roots=() +declare -a launched_pids=() + +uuid_for() { + value="$1" + hex="$(printf '%s' "$value" | sha256sum | cut -c1-32)" + printf '%s-%s-4%s-a%s-%s\n' \ + "${hex:0:8}" "${hex:8:4}" "${hex:13:3}" "${hex:17:3}" "${hex:20:12}" +} + +emit_pass() { + printf 'CASE\t%s\tPASS\t-\t%s\n' "$1" "$2" + ((pass_count += 1)) +} + +emit_red() { + printf 'CASE\t%s\tRED\t%s\t%s\n' "$1" "$2" "$3" + ((red_count += 1)) +} + +pty_at() { + scenario="$1" + shift + env -u PTY_SESSION PTY_ROOT="$scenario/pty" pty "$@" +} + +conversation_count() { + file="$1" + if [ ! -f "$file" ]; then + printf '0\n' + return + fi + jq -r 'select(.type == "conversation") | .sessionId' "$file" | wc -l +} + +conversation_ids_are() { + file="$1" + expected="$2" + test -f "$file" + test "$(jq -r 'select(.type == "conversation") | .sessionId' "$file" | sort -u)" = "$expected" +} + +wait_for_receipt() { + receipt="$1" + for _ in $(seq 1 100); do + test -s "$receipt" && return 0 + sleep 0.02 + done + return 1 +} + +setup_scenario() { + case_id="$1" + scenario="$root/scenarios/$case_id" + scenario_roots+=("$scenario") + mkdir -p "$scenario/agents/lm/worker" "$scenario/workspace" "$scenario/sessions" "$scenario/state" + cp "$template" "$scenario/adapter.sh" + chmod +x "$scenario/adapter.sh" + printf '%s\n' 'DURABLE-WORK-CONTINUES-8d31' > "$scenario/durable" + + exact="$(uuid_for "$scenario/exact")" + neighbor="$(uuid_for "$scenario/neighbor")" + printf '%s\n' "$exact" > "$scenario/exact-id" + printf '%s\n' "$neighbor" > "$scenario/neighbor-id" + printf '{"type":"conversation","sessionId":"%s","turn":"seed-exact"}\n' "$exact" \ + > "$scenario/sessions/$exact.jsonl" + printf '{"type":"conversation","sessionId":"%s","turn":"seed-neighbor"}\n' "$neighbor" \ + > "$scenario/sessions/$neighbor.jsonl" +} + +write_specs() { + start="$root/scenarios/explicit-start-new-session" + start_exact="$(<"$start/exact-id")" + cat > "$start/agents/lm/worker/agent.kdl" < "$resume/agents/lm/worker/agent.kdl" < "$refuse/agents/lm/worker/agent.kdl" < "$fallback/agents/lm/worker/agent.kdl" < "$legacy/agents/lm/worker/agent.kdl" < "$scenario/up.out" 2>&1 +} + +record_pid() { + receipt="$1" + pid="$(cut -f4 "$receipt")" + test -n "$pid" + launched_pids+=("$pid") +} + +run_explicit_start() { + scenario="$root/scenarios/explicit-start-new-session" + exact="$(<"$scenario/exact-id")" + neighbor="$(<"$scenario/neighbor-id")" + if ! run_launch "$scenario" || ! wait_for_receipt "$scenario/launch.tsv"; then + emit_red explicit-start-new-session P02 "start method was not selected" + return + fi + record_pid "$scenario/launch.tsv" + IFS=$'\t' read -r mode session durable _ < "$scenario/launch.tsv" + if [ "$mode" != start ]; then + emit_red explicit-start-new-session P02 "selected $mode instead of start" + elif [ "$session" = "$exact" ] || [ "$session" = "$neighbor" ] || \ + ! conversation_ids_are "$scenario/sessions/$session.jsonl" "$session"; then + emit_red explicit-start-new-session P03 "start did not create one new session" + elif [ "$durable" != 'DURABLE-WORK-CONTINUES-8d31' ] || \ + [ "$(<"$scenario/durable")" != 'DURABLE-WORK-CONTINUES-8d31' ]; then + emit_red explicit-start-new-session P05 "durable work did not continue" + else + emit_pass explicit-start-new-session "start created $session and preserved durable work" + fi +} + +run_explicit_resume() { + scenario="$root/scenarios/explicit-resume-exact" + exact="$(<"$scenario/exact-id")" + neighbor="$(<"$scenario/neighbor-id")" + exact_before="$(conversation_count "$scenario/sessions/$exact.jsonl")" + neighbor_before="$(conversation_count "$scenario/sessions/$neighbor.jsonl")" + if ! run_launch "$scenario" || ! wait_for_receipt "$scenario/launch.tsv"; then + emit_red explicit-resume-exact P02 "resume method was not selected" + return + fi + record_pid "$scenario/launch.tsv" + IFS=$'\t' read -r mode session durable _ < "$scenario/launch.tsv" + if [ "$mode" != resume ]; then + emit_red explicit-resume-exact P02 "selected $mode instead of resume" + elif [ "$session" != "$exact" ] || ! conversation_ids_are "$scenario/sessions/$exact.jsonl" "$exact" || \ + [ "$(conversation_count "$scenario/sessions/$exact.jsonl")" -ne $((exact_before + 1)) ] || \ + [ "$(conversation_count "$scenario/sessions/$neighbor.jsonl")" -ne "$neighbor_before" ] || \ + ! conversation_ids_are "$scenario/sessions/$neighbor.jsonl" "$neighbor"; then + emit_red explicit-resume-exact P03 "resume did not continue only the declared session" + elif [ "$durable" != 'DURABLE-WORK-CONTINUES-8d31' ]; then + emit_red explicit-resume-exact P05 "durable work did not continue" + else + emit_pass explicit-resume-exact "resume appended only to $exact" + fi +} + +run_unavailable_refuse() { + scenario="$root/scenarios/unavailable-refuse" + exact="$(<"$scenario/exact-id")" + neighbor="$(<"$scenario/neighbor-id")" + exact_before="$(conversation_count "$scenario/sessions/$exact.jsonl")" + neighbor_before="$(conversation_count "$scenario/sessions/$neighbor.jsonl")" + if run_launch "$scenario"; then + run_status=0 + else + run_status=$? + fi + if [ -f "$scenario/launch.tsv" ] || \ + [ "$(conversation_count "$scenario/sessions/$exact.jsonl")" -ne "$exact_before" ] || \ + [ "$(conversation_count "$scenario/sessions/$neighbor.jsonl")" -ne "$neighbor_before" ]; then + emit_red unavailable-refuse P04 "unavailable resume launched or mutated session state" + elif [ "$run_status" -ne 0 ] && \ + grep -Fq "agent 'worker' default launch method 'resume' is unavailable and no declared \`on-unavailable\` method can be selected" \ + "$scenario/up.out"; then + emit_pass unavailable-refuse "missing resume refused before launch" + elif [ "$run_status" -eq 0 ] && \ + grep -Fq "agent 'worker' default launch method 'resume' is unavailable and no declared \`on-unavailable\` method can be selected" \ + "$scenario/up.out"; then + emit_red unavailable-refuse P04 "missing resume reported an error but exited zero" + else + emit_red unavailable-refuse P04 "missing resume did not fail closed with the exact diagnostic" + fi +} + +run_unavailable_start() { + scenario="$root/scenarios/unavailable-start" + exact="$(<"$scenario/exact-id")" + neighbor="$(<"$scenario/neighbor-id")" + if ! run_launch "$scenario" || ! wait_for_receipt "$scenario/launch.tsv"; then + emit_red unavailable-start P04 "declared start fallback was not selected" + return + fi + record_pid "$scenario/launch.tsv" + IFS=$'\t' read -r mode session durable _ < "$scenario/launch.tsv" + if [ "$mode" != start ]; then + emit_red unavailable-start P04 "selected $mode instead of start fallback" + elif [ "$session" = "$exact" ] || [ "$session" = "$neighbor" ] || \ + ! conversation_ids_are "$scenario/sessions/$session.jsonl" "$session"; then + emit_red unavailable-start P03 "fallback did not create a new session" + elif [ "$durable" != 'DURABLE-WORK-CONTINUES-8d31' ]; then + emit_red unavailable-start P05 "durable work did not continue" + else + emit_pass unavailable-start "unavailable resume selected start and created $session" + fi +} + +run_legacy() { + scenario="$root/scenarios/legacy-single-argv" + if ! st2 validate --catalog "$scenario" --host lm --strict > "$scenario/validate.out" 2>&1 || \ + ! run_launch "$scenario" || ! wait_for_receipt "$scenario/launch.tsv"; then + emit_red legacy-single-argv P02 "legacy single argv no longer launches" + return + fi + record_pid "$scenario/launch.tsv" + IFS=$'\t' read -r mode session durable _ < "$scenario/launch.tsv" + if [ "$mode" = legacy ] && [ "$session" = - ] && \ + [ "$durable" = 'DURABLE-WORK-CONTINUES-8d31' ]; then + emit_pass legacy-single-argv "legacy single argv launched unchanged" + else + emit_red legacy-single-argv P02 "legacy argv selected an unexpected payload" + fi +} + +cleanup() { + for scenario in "${scenario_roots[@]}"; do + XDG_STATE_HOME="$scenario/state" PTY_ROOT="$scenario/pty" \ + st2 down --catalog "$scenario" --host lm >/dev/null 2>&1 || true + while IFS= read -r id; do + [ -n "$id" ] || continue + pty_at "$scenario" kill "$id" >/dev/null 2>&1 || true + pty_at "$scenario" rm "$id" >/dev/null 2>&1 || true + done < <(pty_at "$scenario" list --json 2>/dev/null | jq -r '.[].name' || true) + done +} +trap cleanup EXIT + +for case_id in explicit-start-new-session explicit-resume-exact unavailable-refuse unavailable-start legacy-single-argv; do + setup_scenario "$case_id" +done +write_specs + +grammar="$root/scenarios/explicit-start-new-session" +if st2 validate --catalog "$grammar" --host lm --strict > "$grammar/validate.out" 2>&1; then + run_explicit_start + run_explicit_resume + run_unavailable_refuse + run_unavailable_start +else + emit_red explicit-start-new-session P01 "proposed launch-method grammar is unavailable" + emit_red explicit-resume-exact P01 "proposed launch-method grammar is unavailable" + emit_red unavailable-refuse P01 "proposed launch-method grammar is unavailable" + emit_red unavailable-start P01 "proposed launch-method grammar is unavailable" +fi +run_legacy + +cleanup +trap - EXIT + +live_pty=0 +for scenario in "${scenario_roots[@]}"; do + count="$(pty_at "$scenario" list --json 2>/dev/null | jq 'length' || printf '0')" + live_pty=$((live_pty + count)) +done + +live_process=0 +for pid in "${launched_pids[@]}"; do + if kill -0 "$pid" 2>/dev/null; then + live_process=$((live_process + 1)) + fi +done + +printf 'SUMMARY pass=%d red=%d\n' "$pass_count" "$red_count" +if [ "$red_count" -gt 0 ]; then + printf '%s\n' 'PRODUCT-RED launch-method-selection' +else + printf '%s\n' 'PRODUCT-GREEN launch-method-selection' +fi +if [ "$live_pty" -eq 0 ] && [ "$live_process" -eq 0 ]; then + printf '%s\n' 'ZERO-RESIDUE pty=0 process=0 catalogs=5' +fi + +test "$red_count" -eq 0 diff --git a/cells/launch-method-selection/judges/contract.sh b/cells/launch-method-selection/judges/contract.sh new file mode 100755 index 00000000..d3d37c22 --- /dev/null +++ b/cells/launch-method-selection/judges/contract.sh @@ -0,0 +1,53 @@ +#!/usr/bin/env bash +set -euo pipefail + +mode="${1:?mode is required}" +observed="${2:?observed output is required}" +cases="${3:?case manifest is required}" + +case_rows() { + awk -F '\t' '$1 == "CASE" { print $2 "\t" $3 "\t" $4 }' "$observed" +} + +expected_rows() { + awk -F '\t' 'NR > 1 { print $1 "\t" $2 "\t" $3 }' "$cases" +} + +case "$mode" in + coverage) + expected_ids="$(awk -F '\t' 'NR > 1 { print $1 }' "$cases")" + observed_ids="$(case_rows | cut -f1)" + test "$observed_ids" = "$expected_ids" + test "$(printf '%s\n' "$observed_ids" | sort -u | wc -l)" -eq "$(printf '%s\n' "$observed_ids" | wc -l)" + ;; + gaps) + while IFS=$'\t' read -r _ outcome gaps; do + if [ "$outcome" = "PASS" ]; then + test "$gaps" = "-" + else + printf '%s\n' "$gaps" | grep -Eq '^P0[1-6](,P0[1-6])*$' + fi + done < <(case_rows) + ;; + classification) + test "$(case_rows)" = "$(expected_rows)" + ;; + honesty) + red_count="$(case_rows | awk -F '\t' '$2 == "RED" { count += 1 } END { print count + 0 }')" + pass_count="$(case_rows | awk -F '\t' '$2 == "PASS" { count += 1 } END { print count + 0 }')" + grep -Fqx "SUMMARY pass=$pass_count red=$red_count" "$observed" + test "$(grep -Ec '^PRODUCT-(RED|GREEN) launch-method-selection$' "$observed")" -eq 1 + if [ "$red_count" -gt 0 ]; then + grep -Fqx "PRODUCT-RED launch-method-selection" "$observed" + else + grep -Fqx "PRODUCT-GREEN launch-method-selection" "$observed" + fi + ;; + cleanup) + grep -Fqx "ZERO-RESIDUE pty=0 process=0 catalogs=5" "$observed" + ;; + *) + printf 'unknown judge mode: %s\n' "$mode" >&2 + exit 2 + ;; +esac diff --git a/cells/launch-method-selection/judges/self-test.sh b/cells/launch-method-selection/judges/self-test.sh new file mode 100755 index 00000000..932a4d69 --- /dev/null +++ b/cells/launch-method-selection/judges/self-test.sh @@ -0,0 +1,78 @@ +#!/usr/bin/env bash +set -euo pipefail + +root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +judge="$root/judges/contract.sh" +cases="$root/fixture/cases.tsv" +tmp="$(mktemp -d)" +trap 'rm -rf -- "$tmp"' EXIT + +write_baseline() { + output="$1" + pass_count=0 + red_count=0 + { + while IFS=$'\t' read -r case_id expected gaps _; do + [ "$case_id" = "case_id" ] && continue + printf 'CASE\t%s\t%s\t%s\n' "$case_id" "$expected" "$gaps" + if [ "$expected" = PASS ]; then + ((pass_count += 1)) + else + ((red_count += 1)) + fi + done < "$cases" + printf 'SUMMARY pass=%d red=%d\n' "$pass_count" "$red_count" + if [ "$red_count" -gt 0 ]; then + printf '%s\n' 'PRODUCT-RED launch-method-selection' + else + printf '%s\n' 'PRODUCT-GREEN launch-method-selection' + fi + printf '%s\n' 'ZERO-RESIDUE pty=0 process=0 catalogs=5' + } > "$output" +} + +baseline="$tmp/baseline.out" +write_baseline "$baseline" +for mode in coverage gaps classification honesty cleanup; do + "$judge" "$mode" "$baseline" "$cases" +done + +mutations=0 + +missing="$tmp/missing.out" +sed '/explicit-resume-exact/d' "$baseline" > "$missing" +! "$judge" coverage "$missing" "$cases" +((mutations += 1)) + +duplicate="$tmp/duplicate.out" +awk '1; /explicit-resume-exact/ { print }' "$baseline" > "$duplicate" +! "$judge" coverage "$duplicate" "$cases" +((mutations += 1)) + +misclassified="$tmp/misclassified.out" +awk -F '\t' 'BEGIN { OFS = "\t" } $1 == "CASE" && $2 == "explicit-start-new-session" { $3 = ($3 == "PASS" ? "RED" : "PASS"); $4 = ($3 == "PASS" ? "-" : "P01") } { print }' \ + "$baseline" > "$misclassified" +! "$judge" classification "$misclassified" "$cases" +((mutations += 1)) + +unknown_gap="$tmp/unknown-gap.out" +awk -F '\t' 'BEGIN { OFS = "\t" } $1 == "CASE" && $2 == "explicit-resume-exact" { $3 = "RED"; $4 = "P99" } { print }' \ + "$baseline" > "$unknown_gap" +! "$judge" gaps "$unknown_gap" "$cases" +((mutations += 1)) + +missing_cleanup="$tmp/missing-cleanup.out" +sed '/^ZERO-RESIDUE /d' "$baseline" > "$missing_cleanup" +! "$judge" cleanup "$missing_cleanup" "$cases" +((mutations += 1)) + +false_honesty="$tmp/false-honesty.out" +if grep -Fq 'PRODUCT-RED launch-method-selection' "$baseline"; then + sed 's/^PRODUCT-RED /PRODUCT-GREEN /' "$baseline" > "$false_honesty" +else + sed 's/^PRODUCT-GREEN /PRODUCT-RED /' "$baseline" > "$false_honesty" +fi +! "$judge" honesty "$false_honesty" "$cases" +((mutations += 1)) + +printf 'PASS: %d launch-method contract mutations rejected\n' "$mutations" diff --git a/cells/launch-method-selection/launch-method-selection.kdl b/cells/launch-method-selection/launch-method-selection.kdl new file mode 100644 index 00000000..082b8787 --- /dev/null +++ b/cells/launch-method-selection/launch-method-selection.kdl @@ -0,0 +1,28 @@ +// Proposal-tracking launch-method selection over isolated synthetic session records. +// The run step remains red while the proposed Agent Spec grammar is unavailable. +eval { + copy "./fixture" + max-timeout "120s" + + run "probe" { + command "bash ./probe.sh" + } + + judges { + judge "COVERAGE - every proposed branch and the legacy positive control has one closed-set case" { + exec "bash ./judges/contract.sh coverage $RUNS_DIR/probe.out ./fixture/cases.tsv" + } + judge "GAPS - every red maps only to a named P01-P06 proposal gap" { + exec "bash ./judges/contract.sh gaps $RUNS_DIR/probe.out ./fixture/cases.tsv" + } + judge "CLASSIFICATION - observations match the frozen proposal-red and legacy-pass map" { + exec "bash ./judges/contract.sh classification $RUNS_DIR/probe.out ./fixture/cases.tsv" + } + judge "HONESTY - expected product reds are never counted as conformance passes" { + exec "bash ./judges/contract.sh honesty $RUNS_DIR/probe.out ./fixture/cases.tsv" + } + judge "CLEANUP - all eval-owned catalogs and processes leave zero live residue" { + exec "bash ./judges/contract.sh cleanup $RUNS_DIR/probe.out ./fixture/cases.tsv" + } + } +} diff --git a/evidence/harness-exclusions.tsv b/evidence/harness-exclusions.tsv index 020ab448..b776f7bb 100644 --- a/evidence/harness-exclusions.tsv +++ b/evidence/harness-exclusions.tsv @@ -6,6 +6,7 @@ docs judge:cold-reader one-shot offline Claude print grader; no bus identity, DI exec-lifecycle-logging cell deterministic native exec lifecycle probe; no harness seat hook-integrity cell deterministic native hook-installation probe; no live harness seat host-lock-health-negatives cell deterministic native doctor negative probe; no harness seat +launch-method-selection cell deterministic proposed launch-method selection probe; no harness seat pty-attach-machine-stream cell deterministic installed PTY attach composition probe; no harness seat managed-agent-color-env cell deterministic managed-agent environment policy probe; no harness seat pty-attach-only cell deterministic PTY dead-attach policy probe; no harness seat