Skip to content

Let eval teams boot canonical Agent Specs from the hermetic catalog #104

Description

@schickling-assistant

Problem

st2 eval currently lowers team { agent { command ... } } into in-memory AgentSpec values whose shared path is <catalog>/spec.kdl; it deliberately performs no catalog discovery. A managed eval seat launched through account-neutral typed axe agent launch therefore cannot pass the same admission contract as production.

The failure progresses through three independent contradictions:

  1. With only the eval bus identity/root supplied, ST_ROOT <eval-catalog> contradicts the ambient catalog-derived root.
  2. After selecting the eval catalog, ambient PTY_ROOT <live-root> contradicts <eval-catalog>/pty.
  3. After all three roots agree, ST_AGENT new.interviewer resolves to zero canonical Agent Specs.

The exact paid cell is tracked in compoundingtech/evals#56. It uses explicit harness/model/effort/persona/mode/boot axes and intentionally omits an account pin.

Desired seam

Provide an explicit eval opt-in that boots canonical Agent Specs already present in the eval's copied, hermetic catalog. The runtime should use the same discovered declarations for materialization, launch, kickoff/message routing, supervision, logging, and teardown.

Constraints:

  • keep the existing temp catalog and PTY isolation;
  • do not synthesize a shell-backed compatibility declaration or wrapper;
  • do not teach the eval DSL a second partial Agent Spec grammar;
  • do not fabricate an identity outside the canonical catalog or pin an account;
  • keep legacy command-team cells unchanged unless they explicitly opt in;
  • fail closed on missing, duplicate, malformed, nonlocal, or unrunnable canonical seats.

This makes the declaration under evaluation the same declaration the real-work runtime executes, matching the st2 vision rather than adding an Axe-specific bypass.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:agent-specAgent Spec format, parsing, and the agent-spec crate · Set: manualarea:catalogCatalog structure, declarations, transactions, and admission · Set: manualarea:evalst2 eval harness and fixtures · Set: manualorigin:agentFiled or primarily produced by an AI agent · Set: manualtype:featureNew user-visible or system capability · Set: manual

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions