Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
.sandbox/
.personas/
.eval-runs/
tmp/

# noise
.DS_Store
Expand Down
6 changes: 4 additions & 2 deletions CATALOG.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
<!-- Generated by bin/generate-catalog.sh from cell KDL and evidence/*.tsv. Do not edit. -->
# Current eval catalog and evidence

This is the sole current corpus surface: **59 included cells** (19 model-free,
This is the sole current corpus surface: **60 included cells** (20 model-free,
40 model-backed) and **4 retired exclusions**. Inventory fields are derived from the active
KDL and executable judge scripts. Latest accepted PASS evidence remains distinct from the append-only run
history: a failed last run is visible without being advertised as accepted evidence, and cells with no
Expand Down Expand Up @@ -42,6 +42,7 @@ harness-native loader plus canonical hook file.
| `host-lock-health-negatives` | model-free | — | 0 | none | `30s` | 5 | — | **NO STRUCTURED RUN** |
| `inbox-hygiene` | Claude | `claude-sonnet-5` / medium | 1 | low | `900s` | 4 | — | **NO STRUCTURED RUN** |
| `incident-response` | Claude | `claude-sonnet-5` / medium | 2 | medium | `1200s` | 5 | — | **NO STRUCTURED RUN** |
| `launch-method-selection` | model-free | — | 0 | none | `120s` | 5 | — | **NO STRUCTURED RUN** |
| `license-mit` | mixed | `claude-sonnet-5+gpt-5.6-sol` / medium | 3 | high | `1200s` | 6 | — | **FAIL** 2026-07-28T13:27:18Z, 5/6, source [`b3cd5fb`](https://github.com/compoundingtech/evals/commit/b3cd5fbd98c11179a4555d0f9bbccfe98351a734), [receipt](evidence/stable-main-phase4-runs-20260728.json) |
| `license-mit-codex` | Codex | `gpt-5.6-sol` / medium | 3 | high | `1200s` | 6 | **PASS** 2026-07-26, st2 `9d26245`, 1m39s, [`40ec23d`](https://github.com/compoundingtech/evals/commit/40ec23d3e068e5bd70b6d4b9a92a92d291868dce) | **NO STRUCTURED RUN** |
| `managed-agent-color-env` | model-free | — | 0 | none | `90s` | 9 | — | **NO STRUCTURED RUN** |
Expand Down Expand Up @@ -102,6 +103,7 @@ while the last-run column makes a recorded failure distinct from a cell with no
- `exec-lifecycle-logging` / `cell` — deterministic native exec lifecycle probe; no harness seat
- `hook-integrity` / `cell` — deterministic native hook-installation probe; no live harness seat
- `host-lock-health-negatives` / `cell` — deterministic native doctor negative probe; no harness seat
- `launch-method-selection` / `cell` — deterministic proposed launch-method selection probe; no harness seat
- `pty-attach-machine-stream` / `cell` — deterministic installed PTY attach composition probe; no harness seat
- `managed-agent-color-env` / `cell` — deterministic managed-agent environment policy probe; no harness seat
- `pty-attach-only` / `cell` — deterministic PTY dead-attach policy probe; no harness seat
Expand All @@ -116,7 +118,7 @@ while the last-run column makes a recorded failure distinct from a cell with no
- `targeted-reconcile-isolation` / `cell` — deterministic selected reconcile isolation probe; no bus-connected agent
- `two-networks-coexist` / `cell` — deterministic catalog-isolation probe; no harness seat

These **20** rows are generated from structured exclusions. A cell-level row is required for
These **21** rows are generated from structured exclusions. A cell-level row is required for
every derived model-free cell. The one-shot docs cold reader is a model-backed grader, but not a bus-connected
agent: it has no st2 identity, DING sidecar, or lifecycle hooks.

Expand Down
52 changes: 52 additions & 0 deletions cells/launch-method-selection/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# launch-method-selection

Contract-first, model-free acceptance coverage for the proposed start/resume launch-method shape discussed in
[`compoundingtech/st2#124`](https://github.com/compoundingtech/st2/issues/124). The field names and defaults are
not ratified. This cell therefore remains proposal-tracking evidence and does not update `AGENT-SPEC.md`.

The four proposed cases are expected product reds while the current st2 grammar exposes one top-level `argv`.
The legacy case is a positive control and must keep passing. Expected reds are classified evidence, never
conformance passes.

**Capabilities required:** `st2,pty,jq,awk,grep,sed,coreutils`. No model or provider. Every catalog, synthetic
session record, workspace, durable marker, process, and PTY root lives below the eval-owned temporary catalog.

## Closed-set cases

| Case | Current result | Discriminator |
| --- | --- | --- |
| `explicit-start-new-session` | `RED/P01` | selects `start`, creates one new synthetic native ID, and preserves durable work |
| `explicit-resume-exact` | `RED/P01` | appends parsed conversation records only to the declared ID, never a newer neighbor |
| `unavailable-refuse` | `RED/P01` | refuses before either method launches or any session record changes |
| `unavailable-start` | `RED/P01` | selects `start` only when that unavailable fallback is declared |
| `legacy-single-argv` | `PASS` | preserves the existing single top-level `argv` launch behavior |

`P01` means the proposed grammar is unavailable. Once a candidate parses it, behavioral failures are classified
as `P02` wrong method, `P03` wrong session, `P04` wrong unavailable policy, `P05` lost durable continuity, or
`P06` leaked eval-owned runtime state.

The checked-in `AGENT-SPEC.md` remains pinned to an older portable runner and documents compact `command`.
Current st2 declarations use one top-level `argv`; this proposal replaces that cardinality invariant with a
successor that accepts either one legacy `argv` or launch methods, never an ambiguous mixture.

## Evidence boundary

The adapter writes synthetic JSONL conversation records and checks them by parsed `sessionId` and record type.
It never infers continuity from file bytes or mtime. It does not call Claude or Codex and does not touch their
homes or transcript stores. Provider-native facts remain the version-scoped evidence recorded on st2#124;
this cell covers only provider-neutral selection and refusal.

The candidate preserves the optional `resume.session` pin in its public Rust `agent-spec` model, but the
current versioned `agents`/`tasks` CLI and launch receipts do not project it. This external behavior cell can
therefore prove that the selected resume argv continues the exact adversarially chosen session, but it cannot
independently prove parser-level pin preservation. The upstream model tests own that assertion until a public
wire surface is deliberately versioned; this cell does not invent one.

`judges/self-test.sh` plants missing-case, duplicate-case, false-promotion, unknown-gap, false-honesty, and
cleanup-receipt mutations. The focused run remains red until every proposed case passes and the closed-set
manifest is updated.

```sh
bash ./cells/launch-method-selection/judges/self-test.sh
st2 eval ./cells/launch-method-selection/ --keep
```
42 changes: 42 additions & 0 deletions cells/launch-method-selection/fixture/adapter.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
#!/usr/bin/env bash
set -euo pipefail

mode="${1:?mode is required}"
store="${2:?session store is required}"
durable="${3:?durable marker is required}"
receipt="${4:?launch receipt is required}"
requested_session="${5:-}"

test "$(<"$durable")" = "DURABLE-WORK-CONTINUES-8d31"
mkdir -p "$store"

fresh_uuid() {
seed="$(date +%s%N)-$$-$RANDOM-$store"
hex="$(printf '%s' "$seed" | sha256sum | cut -c1-32)"
printf '%s-%s-4%s-a%s-%s\n' \
"${hex:0:8}" "${hex:8:4}" "${hex:13:3}" "${hex:17:3}" "${hex:20:12}"
}

case "$mode" in
start)
session="$(fresh_uuid)"
printf '{"type":"conversation","sessionId":"%s","turn":"created"}\n' "$session" \
> "$store/$session.jsonl"
;;
resume)
session="${requested_session:?resume session is required}"
test -f "$store/$session.jsonl"
printf '{"type":"conversation","sessionId":"%s","turn":"resumed"}\n' "$session" \
>> "$store/$session.jsonl"
;;
legacy)
session="-"
;;
*)
printf 'unknown adapter mode: %s\n' "$mode" >&2
exit 2
;;
esac

printf '%s\t%s\t%s\t%s\n' "$mode" "$session" "$(<"$durable")" "$$" > "$receipt"
exec tail -f /dev/null
6 changes: 6 additions & 0 deletions cells/launch-method-selection/fixture/cases.tsv
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
case_id expected gaps contract
explicit-start-new-session RED P01 explicit start creates a new synthetic native session while durable work remains intact
explicit-resume-exact RED P01 explicit resume appends only to the declared session and never to a newer neighbor
unavailable-refuse RED P01 an unavailable resume refuses without launching either method or mutating session state
unavailable-start RED P01 an unavailable resume degrades to start only when that fallback is declared
legacy-single-argv PASS - the legacy single top-level argv form retains its existing launch behavior
Loading