You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Split out of #2458 as a deliberate trade rather than a defect, so the PR does not have to settle it. Raised there by two review rounds (Chris Barber, and the graded leg of the cross-model round at 1f5ab5b, which carried it forward as a minor).
The mechanism
Every path that prunes audit history raises the retention floor before removing anything, and a throw from raiseAuditFloor is what stops the prune — that ordering is the whole guarantee of #2447, because a floor written after the removal is lost if the process dies in between, and the surviving lower floor then certifies a cursor whose history is gone.
resources/auditStore.ts → purgeAgedLogs:
if(isReadOnlyMode())return[];constbefore=Date.now()-auditRetention;raiseAuditFloor((rootStoreasany).auditStore,before);// throws → the purge below never runsreturnrootStore.purgeLogs({ before });
Why that is pointed here specifically
purgeAgedLogs is the one-shot purge wired into replayLogs for #1115: a node that crash-loops during startup replay never reaches the steady-state cleanup loop, so its aged backlog only grows and every recovery attempt gets more expensive. It exists to let such a node shed files before replaying.
The failure mode is therefore self-inflicted at the worst moment: on a full volume, the thing that fails is the 8-byte floor write, so the purge built to reclaim space becomes unreachable in precisely the condition it was built for. The node keeps crash-looping with a backlog it can no longer shed.
Not a boot crash — resources/replayLogs.ts:79-86 already catches and warns, so recovery continues without the purge.
Current call-site split (deliberate, and worth keeping in mind)
Path
On a floor-write failure
scheduleAuditCleanup retention loop
swallowed and warned — availability wins; nothing is pruned, so the floor stays accurate
purgeAgedLogs (boot/recovery)
propagates; replayLogs catches and warns — purge skipped
Table.deleteHistory
propagates to the caller — correctness wins
delete_transaction_logs_before (ResourceBridge)
propagates to the caller — correctness wins
The constraint that rules out the obvious fix
"Just purge anyway and record that the floor is unknown" does not work, because you cannot record that you pruned without recording if you cannot record. The unknown sentinel is a write to the same store that just refused a write. Any escape has to avoid needing durable state at the moment of failure:
Keep as-is. Correctness-first; documented. Cost is the disk-full recovery path above.
Reserve headroom for the marker so the 8-byte write can always land — the floor record is fixed-size and there is exactly one per database, so this is bounded. Preserves the write-ahead invariant instead of trading it away.
In-memory poison. Let the reclamation paths purge, and mark that database's floor unknown in the process for its lifetime, so oldestRetainedAuditTime() fails closed until a floor write succeeds. Sheds the backlog without lying, but the honesty is not durable — a restart forgets it, and the persisted floor is then stale-low.
(2) looks the most promising, since it keeps the invariant rather than choosing which side of it to give up.
Scope
Reclamation paths only — the boot purge, and arguably the retention loop. deleteHistory and the bridge operation should keep propagating: those are user-invoked, the caller can see the error, and there is no availability argument for pruning behind their back.
Reversible per call site, and no consumer reads the floor yet (#2448 is the first), so this can be settled without a migration.
Related: #2447 (the floor primitive), #2458 (the PR that introduced the ordering), #1115 (the boot purge this blocks), #2451, #2448.
Split out of #2458 as a deliberate trade rather than a defect, so the PR does not have to settle it. Raised there by two review rounds (Chris Barber, and the graded leg of the cross-model round at
1f5ab5b, which carried it forward as a minor).The mechanism
Every path that prunes audit history raises the retention floor before removing anything, and a throw from
raiseAuditFlooris what stops the prune — that ordering is the whole guarantee of #2447, because a floor written after the removal is lost if the process dies in between, and the surviving lower floor then certifies a cursor whose history is gone.resources/auditStore.ts→purgeAgedLogs:Why that is pointed here specifically
purgeAgedLogsis the one-shot purge wired intoreplayLogsfor #1115: a node that crash-loops during startup replay never reaches the steady-state cleanup loop, so its aged backlog only grows and every recovery attempt gets more expensive. It exists to let such a node shed files before replaying.The failure mode is therefore self-inflicted at the worst moment: on a full volume, the thing that fails is the 8-byte floor write, so the purge built to reclaim space becomes unreachable in precisely the condition it was built for. The node keeps crash-looping with a backlog it can no longer shed.
Not a boot crash —
resources/replayLogs.ts:79-86already catches and warns, so recovery continues without the purge.Current call-site split (deliberate, and worth keeping in mind)
scheduleAuditCleanupretention looppurgeAgedLogs(boot/recovery)replayLogscatches and warns — purge skippedTable.deleteHistorydelete_transaction_logs_before(ResourceBridge)The constraint that rules out the obvious fix
"Just purge anyway and record that the floor is unknown" does not work, because you cannot record that you pruned without recording if you cannot record. The unknown sentinel is a write to the same store that just refused a write. Any escape has to avoid needing durable state at the moment of failure:
oldestRetainedAuditTime()fails closed until a floor write succeeds. Sheds the backlog without lying, but the honesty is not durable — a restart forgets it, and the persisted floor is then stale-low.(2) looks the most promising, since it keeps the invariant rather than choosing which side of it to give up.
Scope
Reclamation paths only — the boot purge, and arguably the retention loop.
deleteHistoryand the bridge operation should keep propagating: those are user-invoked, the caller can see the error, and there is no availability argument for pruning behind their back.Reversible per call site, and no consumer reads the floor yet (#2448 is the first), so this can be settled without a migration.
Related: #2447 (the floor primitive), #2458 (the PR that introduced the ordering), #1115 (the boot purge this blocks), #2451, #2448.