Eval catalog plan versus direct brief A/B scaffold

[myobie]

Outcome

This draft preserves the realistic model-free catalog-plan versus direct-brief comparison while adopting the accepted Resource-linked PR115 discovery contract.

• Merged current evals main normally at merge commit 17bc728; no force-push or history rewrite.
• Arm A discovers plan truth only through childless Agent Spec Resources such as resource "receipt-report" _tag="plan" uri="file:plans/receipt-report/plan.kdl".
• The referenced plan.kdl owns identity, owner, versions, parents, reasons, and either external Markdown content or inline intent; the Agent Resource owns only its local role/link.
• The existing A/B scenario remains external Markdown: initial and steered intent bytes are unchanged and still match the direct-message arm exactly.
• One focused second plan proves inline intent through the same Resource-link boundary without changing the comparison.
• Arm B still delivers the byte-identical briefs through isolated durable st2 message send operations with inReplyTo lineage, then recovers them through message ls|read.
• Both arms lose volatile state and recover from local durable folders while their remote sources are offline.
• Legacy plan-ref and agent-owned inline plan truth are absent.

Model-free comparison result

Endpoint	Catalog plan	Direct brief	Verdict
Cold resume	pass	pass	tie
Exact intent recovery	pass	pass	tie
Human steering	pass	pass	tie
Worker acceptance evidence	evaluator-only	evaluator-only	tie

The conclusion remains no-measured-advantage for cold resume, intent recovery, or acceptance evidence. The plan adds static schema/graph validation and resolved-content provenance. Neither read-only plan inspection nor direct-message delivery reports worker acceptance.

Plain copied folders remain sufficient and casRequired is false.

Exact pairing

• evals head: 9fae14a192a1f11074de0ef2044e3235017bd476
• st2 draft PR: #115
• exact Resource-link source: 8a76b6e71355140e5b89cd9313fcfd88c82b5cad
• Linux x86-64 artifact SHA256: 214e08874720bc546d4adf7d7977e614237baf7989cc09f6932cd991f497a753
• hosted st2 validation: run 30835684680, job 91760161352, pass
• experiment-shape source: gist revision 5c1d142
• deterministic comparison receipt SHA256: 5491c28cded9172ad63afaf1665ecfe22b61aeca76160c4cf98544682562b57c

The contract stores no content digest or historical record, so it cannot prove that an earlier declaration, parent link, or content file stayed unchanged. The repository-wide released runner remains 0fed14b; the experimental 8a76b6e artifact is isolated to this cell.

Validation

• Exact experimental runner: st2 eval ./cells/catalog-plan-vs-direct-brief/ --keep — 24 PASS / 0 FAIL / 24 gating judges.
• Exact target forms: external Markdown content and inline intent both pass validate|list|show|inspect, with catalog bytes unchanged.
• Released 0fed14b runner: bin/check-corpus.sh — PASS, 60 maintained cells, 119 KDL files, 12 nested catalogs.
• Shell syntax and git diff --check: pass.
• Model/provider calls: 0.

Honest limits

The reference implementation proves the frozen task and arm-neutral judges are valid; it is not substituted for two agent executions. Correctness, coordination messages/bytes, model input/output tokens, cost, and wall duration remain unresolved provider-backed endpoints. A live A/B requires separate authorization naming the model, effort, per-arm budgets, order/counterbalancing, spend ceiling, cleanup, and rollback.

This PR remains draft and unmerged pending Nathan review of the exact pair.

[myobie]

Moved this shared PR from draft to ready at the unchanged accepted head a8e223f99c8f4b4e67e3390dec507fcbe992edf6 after Nathan's review, and requested @schickling for the normal review path. No merge is authorized until that review returns.

[myobie]

Returning this shared PR to draft at unchanged head a8e223f99c8f4b4e67e3390dec507fcbe992edf6 because the accepted PR115 discovery contract changed. The eval shape and model-free conclusion will be preserved; plan discovery will move to an Agent KDL Resource with _tag="plan" whose referenced plan.kdl owns plan truth. No branch rewrite or product-cell run will occur until the exact corrected PR115 head and interface arrive.