Skip to content

eval: cover Agent Spec launch method selection - #71

Draft
schickling-assistant wants to merge 1 commit into
compoundingtech:mainfrom
schickling-assistant:schickling-assistant/2026-08-03-resume-lifecycle-evals
Draft

eval: cover Agent Spec launch method selection#71
schickling-assistant wants to merge 1 commit into
compoundingtech:mainfrom
schickling-assistant:schickling-assistant/2026-08-03-resume-lifecycle-evals

Conversation

@schickling-assistant

Copy link
Copy Markdown
Contributor

Problem

The proposed Agent Spec start/resume lifecycle has no model-free behavior cell covering method selection, exact-session continuation, unavailable-method policy, and compatibility with the existing single argv form.

Goal

Add contract-first expected-red coverage for the proposed syntax while keeping the current portable runner and canonical AGENT-SPEC.md unchanged.

Decisions

  • Cover five closed-set cases: explicit start, exact resume, unavailable refusal, declared start fallback, and legacy argv.
  • Treat start/resume availability as declaration-level behavior; the probe never inspects provider transcript stores.
  • Require unavailable-without-fallback to emit the exact resolved-identity diagnostic, exit nonzero, launch nothing, and leave session records unchanged.
  • Keep the four proposal cases at RED/P01 against the published runner. Expected reds are classified evidence, not conformance passes.
  • Preserve the legacy case as a positive control.
  • Mark the syntax as proposed and unratified; do not update AGENT-SPEC.md.

Verification

  • Published runner 0fed14b: four proposal cases RED/P01, legacy case PASS, PRODUCT-RED, zero residue; 5/6 judges pass and the run step fails as intended.
  • Held-out st2 candidate add61394c335092eaed6d85e00d7befe8b05c023: all five behavior cases pass, PRODUCT-GREEN, zero residue. The outer eval fails only the deliberately frozen current-runner classification map.
  • bash cells/launch-method-selection/judges/self-test.sh: 6/6 contract mutations rejected.
  • 229 shell scripts parse.
  • Preflight safety, model policy and mutations, run history, retired surfaces, event-first, harness contract, closed-set mutations, overnight policy, PII/history, catalog freshness, and git diff --check pass.
  • bin/check-corpus.sh remains red on the unchanged weird-git-setup fixture: its intentional alternate commit identity is rejected by the globally installed assistant-identity commit hook. The fixture and check are unchanged from main, and the failure predated this change. The hook was not bypassed.

Complexity

Medium. The cell is model-free, but it exercises five isolated catalogs and adversarial synthetic session records with explicit cleanup.

Concerns

Friction & bottlenecks

The repository did not ignore the tmp/ directory used by the required Axe worklog. This PR adds the missing ignore rule so local tooling metadata cannot enter the publishable PII surface.

Follow-ups

When a ratified implementation becomes the published eval runner, update the frozen case manifest from the four P01 expected reds to the accepted behavior map.

References

Refs compoundingtech/st2#124.

agent-tool: Codex
agent-tool-version: 0.145.0
agent-runtime: Codex 0.145.0
agent-session-lookup: unavailable
tooling-profile: dotfiles@unknown-dirty
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant