evals: prove agent-new interview handoff boundary - #56
Draft
schickling-assistant wants to merge 12 commits into
Draft
evals: prove agent-new interview handoff boundary#56schickling-assistant wants to merge 12 commits into
schickling-assistant wants to merge 12 commits into
Conversation
This was referenced Jul 30, 2026
agent-session-id: dev3.dotfiles-cos-misc-agent-runtime-simplification agent-tool: Codex agent-tool-version: 0.145.0 agent-model: gpt-5.6-sol agent-runtime-profile: /home/schickling/.config/coding-agents/profile.json agent-skills-manifest: /nix/store/nk9iml2841l1yjjg0f6f0d3y60zkg1nn-agent-skills-corpus/share/agent-skills/manifest.json tooling-profile: dotfiles@4a0515f
agent-session-id: dev3.dotfiles-cos-misc-agent-runtime-simplification agent-tool: Codex agent-tool-version: 0.145.0 agent-model: gpt-5.6-sol agent-runtime-profile: /home/schickling/.config/coding-agents/profile.json agent-skills-manifest: /nix/store/nk9iml2841l1yjjg0f6f0d3y60zkg1nn-agent-skills-corpus/share/agent-skills/manifest.json tooling-profile: dotfiles@4a0515f
agent-session-id: dev3.dotfiles-cos-misc-agent-runtime-simplification agent-tool: Codex agent-tool-version: 0.145.0 agent-model: gpt-5.6-sol agent-runtime-profile: /home/schickling/.config/coding-agents/profile.json agent-skills-manifest: /nix/store/nk9iml2841l1yjjg0f6f0d3y60zkg1nn-agent-skills-corpus/share/agent-skills/manifest.json tooling-profile: dotfiles@4a0515f
agent-session-id: dev3.dotfiles-cos-misc-agent-runtime-simplification agent-tool: Codex agent-tool-version: 0.145.0 agent-model: gpt-5.6-sol agent-runtime-profile: /home/schickling/.config/coding-agents/profile.json agent-skills-manifest: /nix/store/nk9iml2841l1yjjg0f6f0d3y60zkg1nn-agent-skills-corpus/share/agent-skills/manifest.json tooling-profile: dotfiles@4a0515f
schickling-assistant
force-pushed
the
schickling-assistant/2026-07-30-agent-new-interview-evals
branch
from
July 30, 2026 19:44
e64f6ba to
901d9f2
Compare
agent-session-id: dev3.dotfiles-cos-misc-agent-runtime-simplification agent-tool: Codex agent-tool-version: 0.145.0 agent-model: gpt-5.6-sol agent-runtime-profile: /home/schickling/.config/coding-agents/profile.json agent-skills-manifest: /nix/store/nk9iml2841l1yjjg0f6f0d3y60zkg1nn-agent-skills-corpus/share/agent-skills/manifest.json tooling-profile: dotfiles@4a0515f
agent-session-id: dev3.dotfiles-cos-misc-agent-runtime-simplification agent-tool: Codex agent-tool-version: 0.145.0 agent-model: gpt-5.6-sol agent-runtime-profile: /home/schickling/.config/coding-agents/profile.json agent-skills-manifest: /nix/store/nk9iml2841l1yjjg0f6f0d3y60zkg1nn-agent-skills-corpus/share/agent-skills/manifest.json tooling-profile: dotfiles@4a0515f
agent-session-id: dev3.dotfiles-cos-misc-agent-runtime-simplification agent-tool: Codex agent-tool-version: 0.145.0 agent-model: gpt-5.6-sol agent-runtime-profile: /home/schickling/.config/coding-agents/profile.json agent-skills-manifest: /nix/store/nk9iml2841l1yjjg0f6f0d3y60zkg1nn-agent-skills-corpus/share/agent-skills/manifest.json tooling-profile: dotfiles@4a0515f
agent-session-id: dev3.dotfiles-cos-misc-agent-runtime-simplification agent-tool: Codex agent-tool-version: 0.145.0 agent-model: gpt-5.6-sol agent-runtime-profile: /home/schickling/.config/coding-agents/profile.json agent-skills-manifest: /nix/store/nk9iml2841l1yjjg0f6f0d3y60zkg1nn-agent-skills-corpus/share/agent-skills/manifest.json tooling-profile: dotfiles@4a0515f
agent-session-id: dev3.dotfiles-cos-misc-agent-runtime-simplification agent-tool: Codex agent-tool-version: 0.145.0 agent-model: gpt-5.6-sol agent-runtime-profile: /home/schickling/.config/coding-agents/profile.json agent-skills-manifest: /nix/store/nk9iml2841l1yjjg0f6f0d3y60zkg1nn-agent-skills-corpus/share/agent-skills/manifest.json tooling-profile: dotfiles@4a0515f
agent-session-id: dev3.dotfiles-cos-misc-agent-runtime-simplification agent-tool: Codex agent-tool-version: 0.145.0 agent-model: gpt-5.6-sol agent-runtime-profile: /home/schickling/.config/coding-agents/profile.json agent-skills-manifest: /nix/store/nk9iml2841l1yjjg0f6f0d3y60zkg1nn-agent-skills-corpus/share/agent-skills/manifest.json tooling-profile: dotfiles@4a0515f
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Pins the corpus to merged st2
c6846f6239329f0803142afc06c15a07b93937c1and adds the B2 eval pair for the proposedaxe agent newinterview handoff:agent-new-bundle-contractis model-free: typed semantic intent deterministically lowers to canonical Agent Spec KDL plus an initial inbox Resource.canonical-agent-runtime-smokeis model-free: an explicitly identified local declaration at an arbitrary path owns a custom PTY and exec task, while a valid remote-host declaration remains inert. The same admitted vector drives native kickoff/reply, custom-task logging, and teardown.agent-new-interviewis one held paid Claude Sonnet agent. Deterministic pre-admission creates a clean Git worktree, then projects the selected runtime profile's exact immutable Axe adapter, absolute profile path, and canonical persona source into one canonical Agent Spec. st2 owns strict admission, launch, routing, completion, and teardown.The checked
AGENT-SPEC.mdnow matches merged st2 task semantics: directargvis the preferred launch form,commandis intentionalsh -csource, the forms are mutually exclusive, andlifecycle "adopt-only"is an explicit migration fence for the compact task or each declared explicit task.Immutable runner provenance
The corpus no longer trusts
command -v st2or a checkout-sensitive debug executable hash. It verifies and runs exactly:c6846f6239329f0803142afc06c15a07b93937c1;flake.lockbyte SHA256aa547f85b21a8a8787adaa9f2a3ad37d55246d355248388e2ec38bd85a830141;/nix/store/nrhfzsarya6ny7wdmy64i9vwgax965a8-st2-0.1.0.drv;/nix/store/z4wj1y20wq00n02gpknnz45fdi2kyc7h-st2-0.1.0;sha256-w3OSHKt96U0aWMzW2FHom6Ii1NvikdXqmJ6wJ1xGJNA=/ 5,147,208 bytes;st2 0.1.0+c6846f6;25276f30a9cfb287e1a9f72318fada2936402343e958160d46a0a7270ab2db2b.A non-vacuous mutation accepts that package and rejects a different executable reporting the same packaged version.
evidence/st2-package-provenance-20260730.mdrecords why: two debug builds from distinct clean checkouts at the same source OID reported the same source version yet had distinct SHA256 values and embedded checkout paths. Those ephemeral debug outputs are evidence, not pinned inputs.Model-free evidence
Green at successor source commit
c40cb7643688a249f615c660fb134ddf19af0962using the packaged st2 above:bin/check-canonical-agent-runtime-smoke.sh:9 PASS / 0 FAIL / 9 gating judges, followed by direct zero-task/zero-PTY teardown verification.st2 eval ./cells/agent-new-bundle-contract/:7 PASS / 0 FAIL / 7 gating judges, including closed-input and malicious KDL-injection negatives.The host's global commit-message hook initially rejected the synthetic fixture author
seed@local. The unchanged full gate passed with hooks disabled only for its disposable Git seed repositories; this is host policy, not an eval or st2 failure.Paid-attempt and lifecycle boundary
One authorized attempt at source
44b75cfc052d3d21530a3d5a366441162efb2febstopped before Axe, account selection, provider readiness, trust behavior, judges, or verdict. Strict materialization made the requiredgit-excludewarning fatal because the copied interviewer workspace was not a Git worktree. Cleanup left neither catalog nor process residue. It remains pre-run evidence, not provider evidence.No paid/provider run was made for this successor.
The corrected future R04 lifecycle discriminator is recorded but not faked here: mutate absence of only the temporary interviewer's main PTY; treat DING-sidecar state as indeterminate at that observation boundary; retain the durable retired tombstone as anti-resurrection authority; and prove exact idempotent transaction completion with a pinned future st2 successor. This PR deliberately adds no shell stand-in cell.
This PR remains draft.