Add secondary (live-follower) open mode via secondaryPath + catchUpWithPrimary; report read-only open races accurately instead of as corruption

[kriszyp]

Opening a database with readOnly: true against a directory another process is actively writing could fail with RocksDB's "The file MANIFEST-… may be corrupted" wording (or a blanket "Database does not exist") when the writer's compaction, blob GC, or flush deleted a file mid-open — a false report against intact data that invites a repair or restore during an incident. This PR fixes the report and adds the mode that makes a live follower actually correct: RocksDB secondary instances.

Part B — honest classification. A Node-free classifier (isMissingSstOpenRace in core/open_status.cpp, GoogleTest-covered) recognizes a status naming a numbered .sst, .blob, or .log at a filename-token boundary, then the open path confirms that file is absent inside the captured database directory without relying on localized OS error text. The read-only branch throws a distinct ERR_CONCURRENT_COMPACTION error that explains the race, preserves the original RocksDB status text, and stops claiming corruption or nonexistence. All three file classes were observed empirically (a parallel writer reclaims compaction inputs, GC'd blob files, and flushed WAL segments). Genuine corruption (checksum mismatch, bad MANIFEST record), a genuinely missing column family, and a genuinely missing database keep their existing failure shapes.

Part A — secondary mode. A new secondaryPath open option switches the open to DB::OpenAsSecondary: path stays the primary's data directory, and secondaryPath names the secondary instance's own workspace (created if missing, required to be outside the primary directory). New db.catchUpWithPrimary() / db.catchUpWithPrimarySync() methods replay the primary's MANIFEST + WAL to advance the follower (ERR_NOT_SECONDARY on other handles), and db.secondaryPath identifies a follower handle. A secondary implies readOnly (an explicit readOnly: false throws; a null secondaryPath is absent, matching the native layer) and forces maxOpenFiles: -1 — the eagerly opened, fd-held table and blob files are what make the primary's deletions safe, so an explicit bounded value is rejected rather than silently ignored.

Three enforcement pieces are load-bearing. Registry identity: DBKey gains the workspace, with descriptorKey() as the single reconstruction point so every close/purge path (CloseDB and the backup/backup-stream/checkpoint purge-retry destructors) finds a secondary's entry instead of leaking it; DestroyDB now claims and closes every descriptor for the path. The database path is resolved to filesystem identity once per open, and that one string is both registry keys, what RocksDB opens, and what destroy deletes — so two spellings of one directory cannot hold two descriptors and open one workspace twice, which is the in-process half of exclusivity that the kernel lock cannot cover where it degrades. Callers still see the spelling they passed: identity keys the registries, it is not what any API hands back. Workspace exclusivity: RocksDB itself accepts a second OpenAsSecondary on the same workspace (proved by a native test) even though the instances corrupt each other's state, so the binding takes a kernel advisory lock on <secondaryPath>/.secondary.lock (the backup-lock utility, with its documented FUSE/NFS degrade caveats) and the registry rejects in-process reuse across primaries. Read-only log access: a read-only or secondary open no longer mutates the primary's transaction logs — discovery skips the retention purge, recoverTail truncation, and segment retirement (reader truncation of a live writer's active segment is how acknowledged writes vanish — invariant 5, the harper#2016 class), log files open with read-only descriptors and never create, repair, or remove append-boundary markers, and a writable open is refused while read-only-loaded stores are live in-process. That guard only meets both handles when they land on one registry entry, so the log-store registry keys on the same resolved identity: two spellings of one directory used to open two entries over one log tree and let a writer truncate a segment a reader had mapped, whose next access is an uncatchable SIGBUS.

For the human reviewer

Judgment calls from the review's decision ledger, with what was chosen and why. Items 1 and 2 were consulted before implementation (needs-input, 2026-09-01) and ruled: proceed with secondary mode over BlobDB backed by the empirical regression net, and spell the API secondaryPath.

1. Shipping over upstream-unsupported BlobDB (ruled): upstream documents secondary+BlobDB as unsupported (facebook/rocksdb#13296) while this library enables blob files unconditionally. Chosen: ship, with test/native/secondary_blob_test.cc as the assertion-only regression net (the pinned build fd-holds blob files exactly like SSTs — verified via /proc/self/fd) and the caveat documented in README/TSDoc/AGENTS. Alternative: disable blob files for secondaries — an on-disk format decision. A RocksDB upgrade that regresses this fails the native suite.

2. API shape (ruled): secondaryPath?: string, presence switches the mode; rejected secondary: {path} (no second sub-option exists) and a mode enum (collides with optimistic/pessimistic).

3. A secondary's log view is what this process has resident, not a frozen snapshot: catchUpWithPrimary() advances the RocksDB view only. A store this process never discovered stays not-found (useLog() throws rather than conjuring or lazily loading one), and a cross-process primary's new stores and appends need a reopen — but a store an in-process writer holds open is the same object, so its appends are visible. That is not unsafe state: the log write completes before the RocksDB commit for every writer, so log-leads-database is the normal direction and every consumer already tolerates it. Alternative: a uniform snapshot contract, which would mean binding each reader to a private store copy; documented in README, TSDoc and AGENTS instead, so changing it later is a docs-and-behavior break.

4. Writable open refused while read-only-loaded stores are live in-process: the alternative (upgrade in place with recovery) truncates under live readers holding maps — the invariant-14 shape. Makes in-process open order load-bearing: open the writer before the follower, or close the follower first. The same rule is enforced per store at resolve time, so a store that appears after a writer is already open cannot be adopted by it either — that rejection reaches JS as an error at both useLog surfaces (a regression test covers it, because an escaping C++ exception there aborts the process rather than failing a call).

5. A retired transaction-log segment ends on a transaction boundary. A Windows zero-fill that cannot land retires the segment instead of leaving it appendable; because a retired segment cannot erase anything, its persisted boundary is its only eraser, so it retires at the last complete transaction rather than at the framing end — otherwise the next segment's first flagged batch closes the orphaned prefix and two source transactions read as one (invariant 14 spans rotations). Erase and retire now share one predicate. Retirement is also decided at load rather than on the first write, which means an unwritable marker fails the open; a reviewer may prefer degrading to read-only there instead.

6. destroy() closes every descriptor for the path, secondaries included (then errors if references remain). Alternative: reject destroy while a follower is open — a behavior change for existing two-handle callers (a read-write + read-only pair already destroys today).

7. Hard rejections over silent normalization: explicit readOnly: false or bounded maxOpenFiles alongside secondaryPath throw. Strictness surfaces config mistakes; the cost is that a shared options object with maxOpenFiles set cannot be reused for secondaries verbatim.

8. Deferred, documented: concurrent catchUpWithPrimary() calls each hold a libuv worker for the whole replay and queue on the per-database mutex while holding it, so overlapping calls can exhaust the default four-thread pool (the shape invariant 16 describes). Coalescing waiters onto one in-flight replay removes it without a new thread and is the obvious follow-up; for now the README and TSDoc say to await one call before starting the next. The registry hazards exposed by this PR are fixed: waits re-find their entries after every wake, keep an explicit closing descriptor visible across spurious wakes, and recheck path-wide workspace ownership; destroy serializes against every registry key for the physical path, so a reader or fresh-workspace secondary cannot open mid-delete. Still left for separate issues (pre-existing): a POSIX truncateFile() failure leaves a torn segment appendable where this PR built the symmetric Windows retirement, the registry mutex is held across slow opens, non-race read-only IOErrors still collapse to "Database does not exist", and catchUpWithPrimary()'s replay is uncancellable, so a close that waits on it blocks until it finishes.

9. Known limitation, documented not fixed: a cross-process follower reads a transaction-log file through a MAP_SHARED overlay covering an unrecovered torn tail, so if the primary restarts and its recovery truncates below a mapped page, the follower's scan of that region takes SIGBUS — an uncatchable kill, and precisely when a follower matters most. Windows is protected by mandatory sections; POSIX is not, and invariant 14's "POSIX needs none" reasoning covered only a same-process writer. The fix is to serve read-only stores through positional reads rather than a shared mapping, which is a change to the read path that deserves its own review rather than a late addition here. Recorded in AGENTS invariant 18.

10. destroy() does not delete a secondary's workspace directory. It closes every follower on the path and releases their locks, but the workspace is a caller-chosen directory outside the database, so removing it is not destroy's to do. A reviewer who wants destroy to clean it up should say so; the alternative is silently deleting a directory the caller named and may reuse.

11. The cross-key open/destroy serialization is inspection-backed. The independent reviewer traced the condition-variable and mutex ordering without finding a lost wake or deadlock, but there is not yet a deterministic worker-thread regression that repeatedly opens read-only or fresh-workspace secondary handles while another thread destroys the same physical path. Adding such a stress seam/test is the strongest follow-up; rejecting this choice would mean delaying the fix while the previously demonstrated open-during-delete hole remains. Separately, test/lock-tracker.test.ts's park test failed twice under Bun on Windows during this work and is green now; this branch touches nothing in the park, verification-table or commit path.

12. Destroy holds the process-global database-registry mutex through RocksDB deletion and workspace removal. This is deliberate conservative serialization: it prevents a different registry key for the same physical path from opening after descriptor claims are erased but before deletion finishes. The cost is that an unusually slow or large destroy stalls unrelated registry operations process-wide; the independent reviewer found no re-entry or deadlock. Narrowing that critical section later requires a path-wide destruction sentinel that survives entry erasure.

Where to look hardest: the read-only transaction-log load paths (transaction_log_store.cpp, transaction_log_file_posix.cpp/_windows.cpp) — they touch invariant-5 machinery, and the Windows side is untested locally (CI covers it).

Verification

• test/secondary.test.ts runs a follower against a primary in a separate process — writing, flushing and compacting while the follower catches up and reads — and asserts the view advances only on catch-up, never backwards, and never fails on a reclaimed file. The race test proves the defect rather than asserting it: a worker-thread writer drives continuous write/flush/compaction traffic while read-only opens race it — measured runs: ~10% of opens fail classified as ERR_CONCURRENT_COMPACTION (never corruption, never "does not exist") plus stale-snapshot read failures — and the same churn through secondary opens, catch-ups, and reads runs clean. Catch-up visibility is asserted in both directions, including >2KB blob-resident values; the test skips (never false-fails) on an environment that cannot reproduce the race.
• test/readonly.test.ts covers the Part B classification end-to-end plus the new no-mutation guarantees: a read-only open leaves a torn transaction-log tail untouched (byte-for-byte) where a writable open truncates it, a writable open is refused while read-only-loaded stores are live, a writer cannot adopt a read-only-loaded store through either useLog surface, and the writable-open refusal holds when the two handles spell the path differently (that last one fails without the identity re-key).
• test/native/secondary_blob_test.cc (assertion-only) is the secondary+BlobDB regression net: blob reads through a secondary, catch-up after primary blob GC, cold reads of GC'd blobs with fd-hold evidence on Linux, and the proof that RocksDB does not reject a shared workspace — the reason the kernel lock exists. test/native/open_status_test.cc covers the classifier over observed status shapes and corruption/missing-CF/missing-DB counterexamples; test/native/transaction_log_erase_tail_test.cc covers the Windows zero-fill fallback, the retire-on-failure path, and that retirement lands on a transaction boundary.
• Full gates on this head: pnpm build passed; pnpm test (859 passed, 3 skipped), pnpm test:native (179 passed), and pnpm check clean. The affected test/secondary.test.ts, test/readonly.test.ts, and test/destroy.test.ts suites also passed after the final race-hardening changes. Cross-process workspace exclusion is verified with a spawned child process.

Refs #812

Complexity: complicated

— Claude Fable 5.1

🤖 Generated with Claude Code

_{Review-Coverage: authored=codex; ran=claude; declined=gemini,cursor-grok,cursor-composer,domain; rounds=6 @ 276d382}

_{Human-Review-Need: 4 @ 276d382}

[gemini-code-assist]

Code Review

This pull request introduces support for opening RocksDB databases as secondary instances (read-only followers of a live primary) via the secondaryPath option, along with catchUpWithPrimary and catchUpWithPrimarySync methods to pull in new writes. It also updates transaction log handling to ensure read-only and secondary opens do not mutate the primary's logs or run tail recovery. The review feedback highlights two main areas for improvement: a bug in the directory nesting check when the primary path ends with a slash, and a potential bypass of the workspace exclusivity check if the secondary path is not canonicalized before comparison in the registry.

[github-actions]

📊 Benchmark Results

get-sync.bench.ts

getSync() > random keys - small key size (100 records)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 lmdb	1	22.53K ops/sec	44.39	42.62	676.109	0.127	112,650
🥈 rocksdb	2	11.35K ops/sec	88.07	85.78	4,688.471	0.187	56,772

getSync() > sequential keys - small key size (100 records)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 lmdb	1	27.00K ops/sec	37.04	35.64	539.21	0.102	134,981
🥈 rocksdb	2	11.79K ops/sec	84.82	82.92	576.948	0.049	58,946

ranges.bench.ts

getRange() > small range (100 records, 50 range)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 lmdb	1	25.91K ops/sec	38.60	35.85	1,885.227	0.303	129,531
🥈 rocksdb	2	15.93K ops/sec	62.79	55.08	1,055.566	0.126	79,627

realistic-load.bench.ts

Realistic write load with workers > write variable records with transaction log

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	343.52 ops/sec	2,911.008	73.21	80,954.364	19.30	688
🥈 lmdb	2	26.06 ops/sec	38,366.546	395.22	1,230,523.147	136.219	64.00

transaction-log.bench.ts

Transaction log > read 100 iterators while write log with 100 byte records

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	39.30K ops/sec	25.45	11.10	20,825.251	0.852	196,479
🥈 lmdb	2	441.24 ops/sec	2,266.34	124.675	9,960.63	1.27	2,207

Transaction log > read one entry from random position from log with 1000 100 byte records

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	728.50K ops/sec	1.37	1.18	4,912.982	0.209	3,642,525
🥈 lmdb	2	440.76K ops/sec	2.27	1.14	9,198.089	0.536	2,203,781

worker-put-sync.bench.ts

putSync() > random keys - small key size (100 records, 10 workers)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	839.76 ops/sec	1,190.812	1,008.03	2,081.617	0.378	1,680
🥈 lmdb	2	1.18 ops/sec	848,627.724	819,872.554	910,114.498	2.22	10.00

worker-transaction-log.bench.ts

Transaction log with workers > write log with 100 byte records

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	22.17K ops/sec	45.11	29.68	20,467.152	2.08	44,333
🥈 lmdb	2	834.46 ops/sec	1,198.376	164.68	15,765.664	5.13	1,669

---

Results from commit ffe77a1

[cb1kenobi]

Cool! This is dope.