Release a purged transaction-log segment's mapping instead of refusing to read it

[kriszyp]

purgeLogs() unlinks a retired segment. That removes one link to an inode whose bytes a
retired segment never changes again; a reader's MemoryMap is the other link, so the entries it
mapped are still exactly the committed history and are still perfectly readable. The bug in
HarperFast/harper#2337 was never that those bytes were served — it was that the mapping was
never released, so the purge reclaimed no space: a 16 MiB mapping of a deleted .txnlog
resident until restart.

This is a rewrite of the branch. The earlier revision published a per-store "purge epoch" that
every reader revalidated against, dropped its caches on, and relocated off deleted segments —
several hundred lines of native + JS coherence machinery whose whole premise (a purged segment is
no longer authoritative) was wrong. All of it is gone; the two commits here are what is left once
the goal is stated as release the mapping rather than stop serving it.

The fix

• TransactionLog._currentLogBuffer is a WeakRef. It is the fast path over the per-segment
  _logBuffers cache — which was already weak — and it is only ever refreshed by query(), so a
  long-lived reader that calls query() once and next() forever (harper's audit subscription)
  froze it on whatever segment was current then, and retention later deleted exactly that segment.
  With both weak, the mapping is released at the next GC once the iterator holding it has moved on:
  no purge-time invalidation, no cross-handle signalling — which matters, because the store is
  process-global and every handle and worker_threads worker has its own JS caches.
• nextReadableLogBuffer() skips a deleted run when an iterator advances. Those segments are
  genuinely gone, and stopping at the hole stopped the iterator permanently — every later poll
  stopped in the same place, so a reader that fell behind the retention floor never saw another
  entry. _findPosition(0) names the oldest survivor, so a purged prefix costs one native call
  rather than a probe per deleted segment. Only a run the store no longer has is skipped: if
  _findPosition(0) does not resolve past the hole, the store still knows that segment and it is
  merely unmappable right now (mid-rotation, 0 bytes at mmap time, transient resource pressure), so
  iteration stops and picks it up on the next poll rather than stepping over durable history. Both
  call sites already know a later segment exists, so the healthy path is unchanged.
• readableExtent() bounds a read of a purged segment by its own mapping. The store forgets a
  purged segment, so getLogFileSize() reports 0 for it — and taking that 0 as the bound dropped
  every entry the reader had not reached yet, including entries appended to the segment after the
  reader last polled, which the writer's in-place overlay extension made visible in that very
  mapping. The fallback walks the frame headers to the zero-timestamp end marker
  (endOfEntries()), cached on the buffer, so iteration reads them all and then advances to the
  next surviving segment instead of stopping on the zero fill. Broken framing yields the whole
  mapping instead, leaving the break for the read path to report with a resync point — this scan
  must not be the thing that decides a corrupt frame ends the log (invariant 11). The same
  reasoning gives an uncommitted read's corruptFrame() a real dataEnd for a purged segment
  instead of 0.
• The native purge stops being able to throw or to lie. removeFile() uses the non-throwing
  std::filesystem::remove overloads on both platforms (a Windows sharing violation used to unwind
  a C++ exception through the N-API purge boundary), a segment that vanished between the purge's
  scan and its unlink is forgotten from sequenceFiles the way the scan forgets an already-missing
  one, and a segment that could not be deleted for a real reason is reported once per purge run via
  log.warn instead of silently stalling the retention floor.

On Windows a live mapping makes the unlink fail outright, so the purge skips that segment and
retention stalls until the buffer is collected; the loop is refused unlink → collection → the next
purge succeeds. That is the one place a human eye is worth more than CI (see below).

Second commit (independent, droppable)

databaseFlushed() keeps its txn.state stream open across flushes, and a stream describes a
descriptor, not a pathname: once the file — or the whole store directory — is unlinked, every write
lands in the orphaned inode while getLastFlushedPosition(), which reads by path, returns the
{0,0} sentinel and retention never advances. The pathname is now checked before the
unchanged-position shortcut (a flush resolving to the already-recorded position must still restore
a missing file), the directory is recreated the way getLogFile() does, isClosing is re-checked
under flushedStateMutex so a concurrent destroy cannot be resurrected, and the reopen is in place
(in | out) rather than truncating — the 8-byte record is overwritten whole, and a truncating
reopen after a failed write would erase the last durable position before a retry that can fail
again. It runs on RocksDB's flush thread, where an escaping exception ends the process, so the
whole rewrite sits behind a catch-all: every failure warns once via log.warn, leaves
lastWrittenFlushedPosition untouched, and is retried on the next flush.

Hardening, not a live bug: only purgeLogs({ destroy: true }) removes the directory today, and
Harper does not call it in production.

For the human reviewer

1. A purged segment stays readable through a mapping that already exists. That is now the
   stated invariant (AGENTS.md 18), not an accident. A reader mid-iteration finishes the segment it
   mapped and then moves on; the test should finish the segment it mapped before the purge exists
   to keep anyone (including a future me) from "fixing" that again.
2. Not covered here: purgeLogs({ destroy: true }) removes the store directory and a fresh
   store restarts segment numbering at 1, so a buffer cached by segment number can answer for a
   different store's file. That is a cache-key identity problem, not a purge-coherence one; it is
   pre-existing, and Harper does not call destroy in production. The previous revision tried to
   solve it with store ids and reader rebinding, and every round of review found another race in
   that machinery.
3. Windows is where CI cannot tell you much. The convergence test
   (should converge on a purge refused by a live mapping) runs only on win32 and only with
   --expose-gc; if the external buffer is not collected promptly there, retention stalls until it
   is. Everything else in the new suite is POSIX-only, since Windows cannot delete a mapped file at
   all.
4. The log.warn for an undeletable segment is once per purge run, not once per process — a
   directory that stays unwritable would otherwise stall retention behind a single log line ever.
5. Open, pre-existing, not fixed here: txn.state is overwritten in place, so a torn write
   could publish a mixed record. The record is 8 bytes (position, sequence) rewritten at offset
   0; a short or torn overwrite could leave one field new and the other old. Both fields only ever
   advance, and a purge reads the file under the same flushedStateMutex the write holds, so
   in-process no reader sees a half-written record and the two safe mixes are conservative; the
   dangerous one is (new sequence, old position) where the old position is further into its
   segment than the new one, which would let a purge treat an unflushed tail of the current segment
   as flushed. It needs a torn 8-byte overwrite inside one sector to happen at all, and main has
   the same write (this branch only makes the reopen non-truncating, which strictly reduces the
   exposure). An atomic publish — temp file plus rename, or a checksummed slot — is the real fix and
   belongs in its own change, since the file has no fsync today either.
6. The extent fallback and the tightened skip came out of the pre-push Codex pass, which
   flagged that a purged segment's getLogFileSize() of 0 truncated a live mapping's read, that an
   unmappable-but-registered segment was indistinguishable from a deleted one, and that the
   original per-segment walk was unbounded. All three are fixed above; the first is the same
   invariant this PR exists to state, so it is a fix rather than a follow-up.

Verification

• Fails on main, passes here: should resume past a purged run rather than stopping at the hole
  and should release the mapping of a purged segment without a restart (Linux, --expose-gc;
  reads /proc/self/maps for 1.txnlog (deleted) after purge + GC). Confirmed by reverting only
  src/transaction-log-reader.ts + src/load-binding.ts to main: 2 failed, 1 passed — the one
  that passes is the "still serves what it mapped" case, which is main's behavior and must stay
  that way.
• should yield entries appended to the segment it mapped before the purge isolates the extent
  fallback: it fails with readableExtent() reduced to the raw getLogFileSize() and passes with
  the fallback in place.
• Item 2: the TransactionLogFlushedState GoogleTests fail on main for both the file-only and
  the whole-directory unlink; the flushSync() restore test in transaction-log.test.ts fails on
  main too.
• Cost of the weak cache, measured interleaved on this machine (1000-entry log, one
  query({start:0}) + one next() per op, 5x100k per run, 3 alternating rounds): strong
  ~2.20-2.34M ops/s, weak ~2.10-2.22M ops/s — roughly 4% on a loop that pays the cache miss on
  every op. A real reader amortizes it over many next() calls per query(), and the WeakRef is
  only re-wrapped when the segment changes, so the steady state costs a deref() rather than an
  allocation.
• pnpm test (Node 26, Linux): 837 passed, 4 skipped, 0 failed. pnpm test:native: 163 passed.
  pnpm check clean. Bun/Deno/Windows not run locally; CI covers them.

Refs HarperFast/harper#2337

Complexity: moderate

_{Review-Coverage: authored=codex; ran=gemini,claude; adjudicated=domain; declined=cursor-grok,cursor-composer; rounds=1 @ def1b22}

_{Human-Review-Need: 4 (decisions: lagging-reader-gap-policy, purged-mapping-lifetime, txn-state-repair-layer) @ def1b22}

[gemini-code-assist]

Code Review

This pull request implements a robust cache revalidation mechanism for JS memory-map caches against the native store's purge epoch, preventing stale reads of purged transaction-log segments. It also hardens databaseFlushed() to verify the physical existence of txn.state on disk rather than relying solely on the stream's open status. The feedback recommends using performance.now() instead of Date.now() in tests to ensure a monotonic clock, and replacing new Uint32Array with readUInt32LE on read buffers to avoid potential alignment errors.

[github-actions]

📊 Benchmark Results

get-sync.bench.ts

getSync() > random keys - small key size (100 records)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 lmdb	1	24.59K ops/sec	40.66	39.30	2,183.807	0.139	122,963
🥈 rocksdb	2	11.20K ops/sec	89.31	86.72	31,441.987	1.24	55,982

getSync() > sequential keys - small key size (100 records)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 lmdb	1	28.53K ops/sec	35.05	33.87	543.881	0.101	142,637
🥈 rocksdb	2	10.92K ops/sec	91.54	87.99	3,634.666	0.154	54,624

ranges.bench.ts

getRange() > small range (100 records, 50 range)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 lmdb	1	25.32K ops/sec	39.50	36.15	1,970.631	0.293	126,590
🥈 rocksdb	2	15.91K ops/sec	62.87	55.16	1,069.884	0.121	79,534

realistic-load.bench.ts

Realistic write load with workers > write variable records with transaction log

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	366.57 ops/sec	2,728.024	86.07	62,593.04	19.12	743
🥈 lmdb	2	26.20 ops/sec	38,164.313	416.526	1,193,390.974	136.66	64.00

transaction-log.bench.ts

Transaction log > read 100 iterators while write log with 100 byte records

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	37.30K ops/sec	26.81	12.18	20,818.892	0.846	186,477
🥈 lmdb	2	445.27 ops/sec	2,245.818	180.018	14,595.213	1.36	2,227

Transaction log > read one entry from random position from log with 1000 100 byte records

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	732.95K ops/sec	1.36	1.20	4,880.697	0.203	3,664,775
🥈 lmdb	2	462.99K ops/sec	2.16	1.12	5,682.291	0.499	2,314,939

worker-put-sync.bench.ts

putSync() > random keys - small key size (100 records, 10 workers)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	840.26 ops/sec	1,190.106	1,030.484	2,470.79	0.359	1,681
🥈 lmdb	2	1.16 ops/sec	860,257.246	797,065.166	921,174.151	2.88	10.00

worker-transaction-log.bench.ts

Transaction log with workers > write log with 100 byte records

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	22.52K ops/sec	44.40	30.12	20,464.151	2.08	45,046
🥈 lmdb	2	825.45 ops/sec	1,211.456	200.709	13,429.13	5.33	1,652

---

Results from commit a69721f