Problem
st2 eval currently lowers team { agent { command ... } } into in-memory AgentSpec values whose shared path is <catalog>/spec.kdl; it deliberately performs no catalog discovery. A managed eval seat launched through account-neutral typed axe agent launch therefore cannot pass the same admission contract as production.
The failure progresses through three independent contradictions:
- With only the eval bus identity/root supplied,
ST_ROOT <eval-catalog> contradicts the ambient catalog-derived root.
- After selecting the eval catalog, ambient
PTY_ROOT <live-root> contradicts <eval-catalog>/pty.
- After all three roots agree,
ST_AGENT new.interviewer resolves to zero canonical Agent Specs.
The exact paid cell is tracked in compoundingtech/evals#56. It uses explicit harness/model/effort/persona/mode/boot axes and intentionally omits an account pin.
Desired seam
Provide an explicit eval opt-in that boots canonical Agent Specs already present in the eval's copied, hermetic catalog. The runtime should use the same discovered declarations for materialization, launch, kickoff/message routing, supervision, logging, and teardown.
Constraints:
- keep the existing temp catalog and PTY isolation;
- do not synthesize a shell-backed compatibility declaration or wrapper;
- do not teach the eval DSL a second partial Agent Spec grammar;
- do not fabricate an identity outside the canonical catalog or pin an account;
- keep legacy command-team cells unchanged unless they explicitly opt in;
- fail closed on missing, duplicate, malformed, nonlocal, or unrunnable canonical seats.
This makes the declaration under evaluation the same declaration the real-work runtime executes, matching the st2 vision rather than adding an Axe-specific bypass.
Problem
st2 evalcurrently lowersteam { agent { command ... } }into in-memoryAgentSpecvalues whose shared path is<catalog>/spec.kdl; it deliberately performs no catalog discovery. A managed eval seat launched through account-neutral typedaxe agent launchtherefore cannot pass the same admission contract as production.The failure progresses through three independent contradictions:
ST_ROOT <eval-catalog>contradicts the ambient catalog-derived root.PTY_ROOT <live-root>contradicts<eval-catalog>/pty.ST_AGENT new.interviewerresolves to zero canonical Agent Specs.The exact paid cell is tracked in compoundingtech/evals#56. It uses explicit harness/model/effort/persona/mode/boot axes and intentionally omits an account pin.
Desired seam
Provide an explicit eval opt-in that boots canonical Agent Specs already present in the eval's copied, hermetic catalog. The runtime should use the same discovered declarations for materialization, launch, kickoff/message routing, supervision, logging, and teardown.
Constraints:
This makes the declaration under evaluation the same declaration the real-work runtime executes, matching the st2 vision rather than adding an Axe-specific bypass.