Skip to content

A failed audit-floor write blocks the boot purge exactly when the disk is full, making #1115's reclamation unreachable #2486

Description

@dawsontoth

Split out of #2458 as a deliberate trade rather than a defect, so the PR does not have to settle it. Raised there by two review rounds (Chris Barber, and the graded leg of the cross-model round at 1f5ab5b, which carried it forward as a minor).

The mechanism

Every path that prunes audit history raises the retention floor before removing anything, and a throw from raiseAuditFloor is what stops the prune — that ordering is the whole guarantee of #2447, because a floor written after the removal is lost if the process dies in between, and the surviving lower floor then certifies a cursor whose history is gone.

resources/auditStore.tspurgeAgedLogs:

if (isReadOnlyMode()) return [];
const before = Date.now() - auditRetention;
raiseAuditFloor((rootStore as any).auditStore, before);   // throws → the purge below never runs
return rootStore.purgeLogs({ before });

Why that is pointed here specifically

purgeAgedLogs is the one-shot purge wired into replayLogs for #1115: a node that crash-loops during startup replay never reaches the steady-state cleanup loop, so its aged backlog only grows and every recovery attempt gets more expensive. It exists to let such a node shed files before replaying.

The failure mode is therefore self-inflicted at the worst moment: on a full volume, the thing that fails is the 8-byte floor write, so the purge built to reclaim space becomes unreachable in precisely the condition it was built for. The node keeps crash-looping with a backlog it can no longer shed.

Not a boot crash — resources/replayLogs.ts:79-86 already catches and warns, so recovery continues without the purge.

Current call-site split (deliberate, and worth keeping in mind)

Path On a floor-write failure
scheduleAuditCleanup retention loop swallowed and warned — availability wins; nothing is pruned, so the floor stays accurate
purgeAgedLogs (boot/recovery) propagates; replayLogs catches and warns — purge skipped
Table.deleteHistory propagates to the caller — correctness wins
delete_transaction_logs_before (ResourceBridge) propagates to the caller — correctness wins

The constraint that rules out the obvious fix

"Just purge anyway and record that the floor is unknown" does not work, because you cannot record that you pruned without recording if you cannot record. The unknown sentinel is a write to the same store that just refused a write. Any escape has to avoid needing durable state at the moment of failure:

  1. Keep as-is. Correctness-first; documented. Cost is the disk-full recovery path above.
  2. Reserve headroom for the marker so the 8-byte write can always land — the floor record is fixed-size and there is exactly one per database, so this is bounded. Preserves the write-ahead invariant instead of trading it away.
  3. In-memory poison. Let the reclamation paths purge, and mark that database's floor unknown in the process for its lifetime, so oldestRetainedAuditTime() fails closed until a floor write succeeds. Sheds the backlog without lying, but the honesty is not durable — a restart forgets it, and the persisted floor is then stale-low.
  4. Purge and accept an inaccurate floor. Reintroduces the silent truncation Add a supported audit staleness-floor accessor: nothing distinguishes a pruned cursor from a complete one, so MQTT durable resume truncates silently #2447 exists to prevent. Listed for completeness; not recommended.

(2) looks the most promising, since it keeps the invariant rather than choosing which side of it to give up.

Scope

Reclamation paths only — the boot purge, and arguably the retention loop. deleteHistory and the bridge operation should keep propagating: those are user-invoked, the caller can see the error, and there is no availability argument for pruning behind their back.

Reversible per call site, and no consumer reads the floor yet (#2448 is the first), so this can be settled without a migration.

Related: #2447 (the floor primitive), #2458 (the PR that introduced the ordering), #1115 (the boot purge this blocks), #2451, #2448.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions