Define truthful restart, retry, and interrupted-run recovery semantics

[douglasjarquin]

Problem

Made persists run history, but active execution does not have one complete, documented recovery contract. Queued runs may remain durable without automatically re-entering the executable queue, while running/awaiting-review runs may become generic failures. Consumers need explicit fields and deterministic behavior to decide whether to resume, retry, replace, or escalate.

Durable history must not be described as resumed execution when no WorkFunc or safe stage boundary was restored.

Required implementation

Add explicit fields where applicable:

retryable
retry_of
interrupted_reason
recovered_at

Define and implement restart behavior for every state:

Queued

• Reconstruct the executable job from durable gate/ref/SHA/config identity and requeue it when safe.
• If reconstruction is impossible, transition to a clearly retryable interrupted state with an actionable reason.
• Do not leave a permanently queued record with no executable job.

Running

Choose one honest policy per stage:

• resume from a proven idempotent durable boundary, or
• transition to failed, retryable: true, interrupted_reason: daemon_restart

Do not silently rerun side-effecting stages without proof.

Awaiting review

Preferred:

• restore the exact gate generation, pending findings, and decision waiter so the operator can decide after restart

Acceptable fallback:

• transition to retryable interrupted failure and require an explicit replacement run linked through retry_of

Awaiting merge

• preserve exactly as awaiting merge
• preserve PR URL and output SHA
• allow later verified merge completion

Terminal states

• preserve unchanged

Additional requirements:

1. Expose interruption/retry fields in the status schema.
2. Add an explicit retry/resubmit command or documented high-level submit behavior that links replacement runs using retry_of.
3. Preserve exact run identity and never guess by branch name.
4. Ensure restart recovery cooperates with submission deduplication and supersession.
5. Document which stages are resumable and which are not.
6. Ensure daemon shutdown and crash have distinguishable interruption reasons where useful.

Acceptance criteria

• No restored run is left in a misleading active state without executable work.
• Queued work either resumes or becomes explicitly retryable.
• Running interruption is represented truthfully and does not silently repeat unsafe side effects.
• Awaiting-review behavior after restart is deterministic and tested.
• Awaiting-merge survives unchanged and can later complete successfully.
• Replacement runs link back through retry_of.
• Consigliere can determine retry versus escalation from structured status alone.

Required tests

• restart in queued state
• restart in every running stage, including post-push stages
• restart while awaiting review
• restart while awaiting merge
• terminal-state preservation
• retry/resubmission identity and retry_of
• restart combined with supersession
• restart combined with pending offline submission
• graceful shutdown versus abrupt process crash