Skip to content

evals: prove agent-new interview handoff boundary - #56

Draft
schickling-assistant wants to merge 12 commits into
mainfrom
schickling-assistant/2026-07-30-agent-new-interview-evals
Draft

evals: prove agent-new interview handoff boundary#56
schickling-assistant wants to merge 12 commits into
mainfrom
schickling-assistant/2026-07-30-agent-new-interview-evals

Conversation

@schickling-assistant

@schickling-assistant schickling-assistant commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Summary

Pins the corpus to merged st2 c6846f6239329f0803142afc06c15a07b93937c1 and adds the B2 eval pair for the proposed axe agent new interview handoff:

  • agent-new-bundle-contract is model-free: typed semantic intent deterministically lowers to canonical Agent Spec KDL plus an initial inbox Resource.
  • canonical-agent-runtime-smoke is model-free: an explicitly identified local declaration at an arbitrary path owns a custom PTY and exec task, while a valid remote-host declaration remains inert. The same admitted vector drives native kickoff/reply, custom-task logging, and teardown.
  • agent-new-interview is one held paid Claude Sonnet agent. Deterministic pre-admission creates a clean Git worktree, then projects the selected runtime profile's exact immutable Axe adapter, absolute profile path, and canonical persona source into one canonical Agent Spec. st2 owns strict admission, launch, routing, completion, and teardown.
  • One structured model-agent inventory covers compact eval declarations and canonical paid templates, including source kind/path/line. Corpus cost classification, event-first policy, and harness-overlay checks consume it.

The checked AGENT-SPEC.md now matches merged st2 task semantics: direct argv is the preferred launch form, command is intentional sh -c source, the forms are mutually exclusive, and lifecycle "adopt-only" is an explicit migration fence for the compact task or each declared explicit task.

Immutable runner provenance

The corpus no longer trusts command -v st2 or a checkout-sensitive debug executable hash. It verifies and runs exactly:

  • source OID c6846f6239329f0803142afc06c15a07b93937c1;
  • source-store flake.lock byte SHA256 aa547f85b21a8a8787adaa9f2a3ad37d55246d355248388e2ec38bd85a830141;
  • derivation /nix/store/nrhfzsarya6ny7wdmy64i9vwgax965a8-st2-0.1.0.drv;
  • output /nix/store/z4wj1y20wq00n02gpknnz45fdi2kyc7h-st2-0.1.0;
  • NAR sha256-w3OSHKt96U0aWMzW2FHom6Ii1NvikdXqmJ6wJ1xGJNA= / 5,147,208 bytes;
  • version prefix st2 0.1.0+c6846f6;
  • executable SHA256 25276f30a9cfb287e1a9f72318fada2936402343e958160d46a0a7270ab2db2b.

A non-vacuous mutation accepts that package and rejects a different executable reporting the same packaged version. evidence/st2-package-provenance-20260730.md records why: two debug builds from distinct clean checkouts at the same source OID reported the same source version yet had distinct SHA256 values and embedded checkout paths. Those ephemeral debug outputs are evidence, not pinned inputs.

Model-free evidence

Green at successor source commit c40cb7643688a249f615c660fb134ddf19af0962 using the packaged st2 above:

  • Complete free preflight: 184 shell parses; 73 KDL parses; semantic loading of all 48 evals; strict validation of 9 nested catalogs; model policy, mutation, fixture reset, harness, generated catalog, PII, history, and append-only gates.
  • bin/check-canonical-agent-runtime-smoke.sh: 9 PASS / 0 FAIL / 9 gating judges, followed by direct zero-task/zero-PTY teardown verification.
  • st2 eval ./cells/agent-new-bundle-contract/: 7 PASS / 0 FAIL / 7 gating judges, including closed-input and malicious KDL-injection negatives.
  • The public renderer invocation supplies an explicit output root and produces both the canonical Agent Spec and initial inbox Resource; renderer-security and template-mutation negatives remain closed.

The host's global commit-message hook initially rejected the synthetic fixture author seed@local. The unchanged full gate passed with hooks disabled only for its disposable Git seed repositories; this is host policy, not an eval or st2 failure.

Paid-attempt and lifecycle boundary

One authorized attempt at source 44b75cfc052d3d21530a3d5a366441162efb2feb stopped before Axe, account selection, provider readiness, trust behavior, judges, or verdict. Strict materialization made the required git-exclude warning fatal because the copied interviewer workspace was not a Git worktree. Cleanup left neither catalog nor process residue. It remains pre-run evidence, not provider evidence.

No paid/provider run was made for this successor.

The corrected future R04 lifecycle discriminator is recorded but not faked here: mutate absence of only the temporary interviewer's main PTY; treat DING-sidecar state as indeterminate at that observation boundary; retain the durable retired tombstone as anti-resurrection authority; and prove exact idempotent transaction completion with a pinned future st2 successor. This PR deliberately adds no shell stand-in cell.

This PR remains draft.

agent-session-id: dev3.dotfiles-cos-misc-agent-runtime-simplification
agent-tool: Codex
agent-tool-version: 0.145.0
agent-model: gpt-5.6-sol
agent-runtime-profile: /home/schickling/.config/coding-agents/profile.json
agent-skills-manifest: /nix/store/nk9iml2841l1yjjg0f6f0d3y60zkg1nn-agent-skills-corpus/share/agent-skills/manifest.json
tooling-profile: dotfiles@4a0515f
agent-session-id: dev3.dotfiles-cos-misc-agent-runtime-simplification
agent-tool: Codex
agent-tool-version: 0.145.0
agent-model: gpt-5.6-sol
agent-runtime-profile: /home/schickling/.config/coding-agents/profile.json
agent-skills-manifest: /nix/store/nk9iml2841l1yjjg0f6f0d3y60zkg1nn-agent-skills-corpus/share/agent-skills/manifest.json
tooling-profile: dotfiles@4a0515f
agent-session-id: dev3.dotfiles-cos-misc-agent-runtime-simplification
agent-tool: Codex
agent-tool-version: 0.145.0
agent-model: gpt-5.6-sol
agent-runtime-profile: /home/schickling/.config/coding-agents/profile.json
agent-skills-manifest: /nix/store/nk9iml2841l1yjjg0f6f0d3y60zkg1nn-agent-skills-corpus/share/agent-skills/manifest.json
tooling-profile: dotfiles@4a0515f
agent-session-id: dev3.dotfiles-cos-misc-agent-runtime-simplification
agent-tool: Codex
agent-tool-version: 0.145.0
agent-model: gpt-5.6-sol
agent-runtime-profile: /home/schickling/.config/coding-agents/profile.json
agent-skills-manifest: /nix/store/nk9iml2841l1yjjg0f6f0d3y60zkg1nn-agent-skills-corpus/share/agent-skills/manifest.json
tooling-profile: dotfiles@4a0515f
@schickling-assistant
schickling-assistant force-pushed the schickling-assistant/2026-07-30-agent-new-interview-evals branch from e64f6ba to 901d9f2 Compare July 30, 2026 19:44
schickling-assistant and others added 8 commits July 30, 2026 21:59
agent-session-id: dev3.dotfiles-cos-misc-agent-runtime-simplification
agent-tool: Codex
agent-tool-version: 0.145.0
agent-model: gpt-5.6-sol
agent-runtime-profile: /home/schickling/.config/coding-agents/profile.json
agent-skills-manifest: /nix/store/nk9iml2841l1yjjg0f6f0d3y60zkg1nn-agent-skills-corpus/share/agent-skills/manifest.json
tooling-profile: dotfiles@4a0515f
agent-session-id: dev3.dotfiles-cos-misc-agent-runtime-simplification
agent-tool: Codex
agent-tool-version: 0.145.0
agent-model: gpt-5.6-sol
agent-runtime-profile: /home/schickling/.config/coding-agents/profile.json
agent-skills-manifest: /nix/store/nk9iml2841l1yjjg0f6f0d3y60zkg1nn-agent-skills-corpus/share/agent-skills/manifest.json
tooling-profile: dotfiles@4a0515f
agent-session-id: dev3.dotfiles-cos-misc-agent-runtime-simplification
agent-tool: Codex
agent-tool-version: 0.145.0
agent-model: gpt-5.6-sol
agent-runtime-profile: /home/schickling/.config/coding-agents/profile.json
agent-skills-manifest: /nix/store/nk9iml2841l1yjjg0f6f0d3y60zkg1nn-agent-skills-corpus/share/agent-skills/manifest.json
tooling-profile: dotfiles@4a0515f
agent-session-id: dev3.dotfiles-cos-misc-agent-runtime-simplification
agent-tool: Codex
agent-tool-version: 0.145.0
agent-model: gpt-5.6-sol
agent-runtime-profile: /home/schickling/.config/coding-agents/profile.json
agent-skills-manifest: /nix/store/nk9iml2841l1yjjg0f6f0d3y60zkg1nn-agent-skills-corpus/share/agent-skills/manifest.json
tooling-profile: dotfiles@4a0515f
agent-session-id: dev3.dotfiles-cos-misc-agent-runtime-simplification
agent-tool: Codex
agent-tool-version: 0.145.0
agent-model: gpt-5.6-sol
agent-runtime-profile: /home/schickling/.config/coding-agents/profile.json
agent-skills-manifest: /nix/store/nk9iml2841l1yjjg0f6f0d3y60zkg1nn-agent-skills-corpus/share/agent-skills/manifest.json
tooling-profile: dotfiles@4a0515f
agent-session-id: dev3.dotfiles-cos-misc-agent-runtime-simplification
agent-tool: Codex
agent-tool-version: 0.145.0
agent-model: gpt-5.6-sol
agent-runtime-profile: /home/schickling/.config/coding-agents/profile.json
agent-skills-manifest: /nix/store/nk9iml2841l1yjjg0f6f0d3y60zkg1nn-agent-skills-corpus/share/agent-skills/manifest.json
tooling-profile: dotfiles@4a0515f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants