Unify durable run and gate-submission storage with interprocess-safe crash recovery

[douglasjarquin]

Problem

Made currently has overlapping persistence surfaces: a checkpoint/WAL run store, an older append-only store implementation, and a separate append-only gate spool. The running daemon keeps an in-memory spool index, while an offline hook can append through another process. This can leave a durable submission invisible until restart and can produce split-brain deduplication/run-ID behavior.

The active file-based store also needs stronger torn-write, generation, symlink, compaction, retention, and repair guarantees.

Required implementation

Strong preference: consolidate runs, stages, decisions, submission identity, pending gate events, cancellation, supersession, retry metadata, and merge completion into one SQLite database using WAL mode, transactions, foreign keys, unique constraints, and a busy timeout.

If file-based storage is retained, all equivalent guarantees below are mandatory.

One persistence owner

1. Identify and remove dead/superseded store implementations and tests.
2. Document one authoritative persistence layer.
3. Version the durable schema.
4. Migrate existing runs.wal, checkpoint, and gate.spool state safely:
   • detect legacy state
   • import once
   • preserve backups
   • record migration completion
   • refuse corrupt/ambiguous input instead of discarding it

Interprocess-safe submissions

1. Use an interprocess lock/transaction around deduplication and enqueue.
2. Same (gate, ref, input SHA) must always return the same run ID across processes.
3. A running daemon must observe an offline hook enqueue without restart.
4. Use a daemon wake mechanism, filesystem notification, database polling, or another bounded reliable mechanism.
5. Drain/claim must be durable and idempotent.
6. Concurrent enqueue/drain must not lose work.
7. Shutdown must account for submissions appended concurrently.
8. Run ID reservation must be durable before success is returned.

Crash consistency

For any retained WAL/log:

1. Detect and truncate a torn final record before later appends.
2. Use symlink refusal (O_NOFOLLOW or equivalent) and owner-only modes.
3. Bound record size.
4. Add sequence/generation metadata and checksums or equivalent validation.
5. Ensure checkpoint installation and WAL reset cannot replay stale records over newer state after a crash.
6. Fsync files and parent directories at required boundaries.
7. Add atomic compaction.
8. Add bounded run/event retention.
9. One corrupt historical record must not make repair impossible; provide an explicit repair/diagnostic command.
10. Do not mutate in-memory authoritative state before the durable write commits, or roll it back on failure.

Acceptance criteria

• An offline hook enqueue becomes visible while the daemon remains running.
• Concurrent duplicate enqueues produce one durable run ID.
• Concurrent enqueue/drain cannot lose an event.
• Torn writes do not brick the next restart or concatenate with later data.
• Crashes at each compaction boundary recover to a consistent generation.
• Symlink attacks on state files are rejected.
• Persistence implementations are not duplicated.
• Storage growth is bounded by documented retention/compaction.
• Doctor reports storage health, migration state, pending submissions, and repair guidance.

Required tests

• two-process duplicate enqueue
• offline append while daemon is running
• enqueue/drain race
• daemon shutdown race with new submission
• torn final record followed by append and restart
• fault injection at every compaction/checkpoint boundary
• stale WAL generation replay prevention
• symlink/refusal tests
• migration from existing state
• corrupt-state repair workflow
• retention/compaction limits
• persistence failure leaves no false in-memory success