Skip to content

plan: preserve declared evidence from measured failures - #101

Open
tlrmchlsmth wants to merge 3 commits into
neuralmagic:mainfrom
tlrmchlsmth:failure-evidence-files
Open

plan: preserve declared evidence from measured failures#101
tlrmchlsmth wants to merge 3 commits into
neuralmagic:mainfrom
tlrmchlsmth:failure-evidence-files

Conversation

@tlrmchlsmth

@tlrmchlsmth tlrmchlsmth commented Sep 3, 2026

Copy link
Copy Markdown
Member

Summary

  • add capture_on_failure to isolated, non-mapped command and evaluate tasks that declare emits_files
  • atomically retain the complete declared file set after a measured failure while discarding the failed workspace
  • make captured failure evidence available only to epilogue tasks under inputs/<producer>/<declared-path>
  • project the option onto plan_admitted, defaulting older events to false
  • amend the playbook and wire RFCs and regenerate the DSL reference

Why this is useful

A certain backport workflow needs two things to happen when a measurement fails:

  1. the failure must block later work, especially delivery; and
  2. the final report must retain the evidence explaining the failure.

Today those requirements conflict with the playbook workspace contract: a failing task does not contribute its workspace writes. The pack therefore makes producers such as repro, build, and confirm always settle successfully so their evidence/*.json files survive, then adds a separate repro_ok, build_ok, or confirm_ok task to fail the graph. Reporting has to correlate three things: the producer, its gate, and the shared evidence file.

This change gives the workflow a direct expression of the intended behavior:

probe fails
   |-- downstream delivery is blocked
   |-- undeclared and partial workspace changes are discarded
   `-- the complete declared evidence set is retained for the epilogue

For example, a probe can fail directly while declaring evidence/repro.json as retained failure evidence. The epilogue receives it at inputs/repro/evidence/repro.json. The producer now owns both the failure and its diagnostic evidence, without laundering a bad measurement through a passing task.

In the current backport graph this can replace the paired gates for ten non-mapped command producers: resolve, spec, rig baseline, repro, build, candidate rig, confirm, regress, deliver, and rig teardown. It intentionally does not cover the mapped skill producers behind review_ok and author_repro_ok; those gates must remain or be folded into a separate non-mapped validator. The domain migration will also need to make reporting a true epilogue and read producer-namespaced inputs.

Contract and safety

  • only isolated, non-mapped command/evaluate tasks may opt in
  • at least one emits_files path is required
  • missing or invalid declared files publish none of the set and augment the original failure diagnostic
  • skipped and transport outcomes publish no failure evidence and remove stale sets
  • captured files do not change task status, satisfy ordinary dependencies, enter git memory, or retain undeclared workspace state
  • epilogues fail closed if an engine-owned captured set is present but incomplete
  • the captured set remains in engine state and survives runner reconstruction

The restrictions on isolation and mapping are deliberate. They avoid retaining a failed shared workspace and avoid using untrusted mapped instance keys as filesystem namespaces.

Verification

  • full cargo test --workspace
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo fmt --all -- --check
  • govctl check
  • focused coverage for measured failures, partial sets, stale skip/transport evidence, epilogue staging, runner reconstruction, plan validation, and current/legacy wire events

The implementation is compliant with the amended candidate contract. RFC-0002 remains in its existing spec phase, so the amendment is not a sealed normative baseline yet.

Refs: WI-2026-09-03-001

Add an isolated failure-file channel for epilogue reporting, project the opt-in onto the admitted-plan wire contract, and document the candidate RFC amendment.

Refs: WI-2026-09-03-001
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@tlrmchlsmth
tlrmchlsmth force-pushed the failure-evidence-files branch from cd6bc8c to de14996 Compare September 3, 2026 21:36
…exit

An epilogue's file producers were only main-graph tasks, so an epilogue
that consumed another epilogue's declared files lost them. Candidates
are now its ancestors plus the main graph.

The executor flagged a failed producer by mutating capture_on_failure
on a staging clone, and the harness probed the filesystem to guess
whether a set was published. TaskRunner gains has_captured_files; the
executor drops a failed producer whose set was withheld before staging,
and the harness answers from its own state.

Both ceiling checks overwrote the halt reason, so an epilogue crossing
the budget after a required failure reported budget-exceeded instead of
the short-circuit. The first halt reason now wins.

capture_on_failure on an epilogue task is refused: its files are never
staged anywhere.

Executor tests cover the epilogue ordering, the failure path, ceiling
blocking, exit labelling, and file staging for ancestors, captured
failures, and withheld sets.

Assisted-by: Claude
Regenerate the RFC-0001 render after both sides amended it, and give
the work item the next free ID: main's neuralmagic#104 took WI-2026-09-03-001.

Assisted-by: Claude
@wseaton

wseaton commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

superceded by #105

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants