Add storage tiering: SST paths and blob file placement across volumes

[kriszyp]

Adds database-wide SST placement through paths, per-column-family blob placement through blobs.dir, and a separate process-wide blob cache for large values. The final validation pass fails closed around persisted blob directories and derives directory creation permission from the same relocation decision, allows a harmless leftover acknowledgement on a read-only no-op, and makes warm paths reuse distinguish omission from an explicit empty list: omission inherits and [] rejects a tiered live database, while the same [] matches an already-untiered one.

Directory isolation remains caller-managed by prior ruling: two databases must never share a paths or blobs.dir directory. The branch keeps the conflict-free origin/main merge requested by @kriszyp; no history was rewritten.

A successful non-default column-family drop now removes that family from every live and retained destroy layout before unregistering its name. This prevents a later destroy() through an old handle from deleting a replacement database that reused the dropped family's blob directory, while preserving same-name recreation and directories still shared by another family; five regressions cover synchronous, asynchronous, live-descriptor, stale-handle, and shared-directory cases.

That drop is also atomic against DBRegistry::OpenDB. It holds databasesMutex across DB::DropColumnFamily itself, both layout erasures, and the columns erase — the same mutex a warm open takes before columnsMutex — so an open sees the family wholly present or wholly gone rather than being handed a ColumnFamilyDescriptor for a family RocksDB has already dropped. The verification-table sweep deliberately stays outside that section: the VT writer mutex is process-global and must never be taken under the registry mutex. A test-only two-phase latch parks the drop mid-section until a worker's warm open has reached the registry mutex, keyed by database path because the counters are process-global and Vitest's thread pool shares them, so the regression asserts the interleaving happened rather than hoping for it.

The retained destroy layout is now canonical rather than last-writer-wins. Registry entries are keyed by path AND read-only, so one database can have two live descriptors, and a read-only handle opens with a paths shorter than the writer's — or none — whenever the files it needs still sit at path index 0. db_paths is serialized nowhere, so recording that shorter list erased the only trace of the writer's other volumes and destroy() reported success with their SST files still on disk (reported by @cb1kenobi). RecordLayout now takes a new list only when it EXTENDS the retained one, and DestroyDB applies the record first and the live descriptor second, so the entry the registry loop happens to pick can add volumes but never remove them. Both halves of that rule are load-bearing in opposite directions, and invariant 18 carries the contract: a shorter list must not shorten the record, and a divergent one must not extend it either — destroy() deletes every SST it finds in each recorded directory, so one mistyped paths naming a neighbouring database's volume would otherwise take that database's files down with this one. Three regressions cover the reader that omits paths, the reader that passes a shorter list and stays live across destroy(), and a reader naming a neighbour's volume — each mutation-checked against the rule it exists for.

Keeping that record is only half of it, and @cb1kenobi found the other half: a writable open the record refused still succeeded, warned about in the info LOG and otherwise ignored. RocksDB then used the requested list while destroy() used the retained one — compaction writing SSTs to a volume destroy() never sweeps, and destroy() still sweeping one this open disowned, which another database may since have been given. Neither is repairable afterwards, so such an open now fails, and it fails before DB::Open — past that point a single compaction placing a single SST is already the divergence. RecordLayout throws on the same condition rather than returning a bool nobody acted on, now purely as the assertion that the record did not move underneath an open database.

The dangerous shape is the quiet one: while every file still sits at path index 0, RocksDB opens happily under a shorter or divergent list, so the regression uses exactly that setup and asserts that no new OPTIONS file appears — RocksDB writes one on every open, so an unchanged set is what proves the refusal preceded the open rather than being unwound after one. Removing the pre-open call makes that assertion fail. One consequence: the check preempts explainOpenFailure's removed/reordered guidance whenever the same process opened the database earlier, so coverage for that message moved to a spawned child, where the record is gone and RocksDB reports the MANIFEST as corrupt — which is what a restart actually produces.

For the human reviewer

1. Require patched-build coverage before merge. The selected policy is to wait for a prebuild carrying Add RocksDB blob_dir patch so blob files can live on a separate volume rocksdb-prebuilds#13, then run the ROCKSDB_JS_REQUIRE_BLOB_DIR=1 suite and specifically confirm external-blob cleanup through DestroyDB; the alternative is merging native storage code that is compiled out by every current prebuild. Waiting is reversible as soon as the prebuild exists, while saying no accepts an unexecuted data-integrity path. One change on this branch sits entirely inside that gap: destroy() now prefers the retained record's blob directory over a live descriptor's, so a read-only handle opened before a relocation can no longer make the sweep visit the directory the blob files left. It is argued from the ordering rather than demonstrated, because every case that would exercise it is capability-skipped.
2. Keep allowDirChange database-scoped even though it is nested under per-family blobs. A relocation moves every family sharing the source directory, so the decision deliberately reaches those families together; the alternative is splitting or renaming the acknowledgement before release. Changing this later is breaking, while saying no now costs an API redesign and its migration tests.
3. Retain tier layouts in the process registry, not a persistent marker. This makes close() followed by destroy() work because the registry keeps a path-keyed layout copy, but a fresh process cannot recover db_paths and may orphan external SSTs. A rocksdb-js-owned marker would cover fresh-process destroy and directory identity at the cost of a new durable format; adding one later is possible but cannot retroactively describe existing tiered databases. Independent review raised the same gap from its other end this round — a process whose first open is read-only seeds an EMPTY path record, so close() + destroy() there reports success with external SSTs still on disk. Declined as the same open question rather than a separate defect: the empty record suppresses nothing, because the fallback it bypasses is LoadLatestOptions, and db_paths sits in RocksDB's "not yet supported" serialization block, so that path recovers exactly as little. Trusting a reader's list instead is the cross-database deletion the retained-record rule exists to prevent. Only a durable marker closes it.
4. Leave the read-only descriptor's column entry registered on a drop. Independent review raised this three rounds running as a leak: a warm read-only descriptor keeps serving the dropped family. It is a separate DB::OpenForReadOnly instance whose handles a writable drop does not invalidate and which by design never observes later changes, so unregistering there would make the next read-only open throw "cannot create column family in read-only mode" for a family its own frozen manifest view still contains; the rationale is now recorded at the call site. The alternative is treating a read-only view as invalidated by a writable drop, which is a broader semantic change than this PR should make.
5. Keep omitted blobs.dir strict on warm opens. Omitting it means “alongside the SSTs,” so a plain second open of a family already using an external directory rejects even though the live handle cannot move. The alternative is an explicitness latch limited to warm reuse; loosening this later is compatible, while saying no now costs that additional state and test surface.

Verification

• pnpm build — production TypeScript bundle and native binding built successfully.
• pnpm check — type-check, lint, and formatting checks passed for 191 files.
• pnpm test — 864 passed, 31 skipped, 0 failed on the pushed head; pnpm check (type-check, lint, format over 192 files) passed.
• pnpm test:native — 197/197 passed; the focused BlobRelocation/BlobDirScanner run passed 31/31, including destination-creation ownership and no-op acknowledgement cases.
• The three destroy-layout regressions were mutation-checked one rule at a time: without the RecordLayout guard the omitted-paths reader orphans the writer's second volume, without the DestroyDB merge the shorter-list reader does, and with either the size-only rule or the previous live-first ordering the neighbour's SST files are deleted.
• The drop/warm-open regression was mutation-checked: restoring the two-lock split makes the worker receive the dropped family and its write is silently discarded, and a second latch use in the same process still works from the baselined counters.
• nm -C -u build/Release/rocksdb-js.node | rg 'rocksdb_js::' and git diff --check produced no output.
• On the origin/main merge (one conflict, the include block in db_descriptor.cpp, both sides kept): pnpm build, pnpm test (875 passed / 32 skipped, 65 files), pnpm test:native (209/209), pnpm check, and nm -C -u clean of undefined rocksdb_js:: symbols — the check that caught the last rebase silently dropping main's ParkTimeoutRegistry hunks.
• The pre-open refusal is mutation-checked in the direction that matters: disabling only AssertDbPathsExtendRetained leaves the post-open throw in place, the message assertion still passes, and the test fails on the OPTIONS-file assertion (OPTIONS-000007, OPTIONS-000015 vs OPTIONS-000007) — i.e. the open had happened.
• The new spawned fixture was run under the repo's actual Deno command (deno run --allow-all --sloppy-imports on Deno 2.7.5): both storage-path regressions pass. Independent review flagged it as needing its own permission flags; measured against pnpm test:deno rather than a bare deno run, the child inherits, as the existing blob-cache fixture already relies on.
• The 28-case blobs.dir integration suite remains capability-skipped because the current RocksDB prebuild lacks the downstream patch; this includes the new read-only relocation/no-op distinction and dropped-layout regressions.

Complexity: complicated

_{Review-Coverage: authored=codex; ran=gemini,claude; blocked=domain(timeout); declined=cursor-grok,cursor-composer; rounds=11 @ a2e39b2}

_{Human-Review-Need: 4 @ a2e39b2}

[gemini-code-assist]

Code Review

This pull request introduces tiered storage support for RocksDB, allowing SST files to be distributed across multiple storage paths and large values to be decoupled into separate blob files. The changes span C++ bindings, TypeScript definitions, documentation, and tests. The review feedback correctly identifies a cross-platform path handling issue on Windows, where using std::filesystem in the C++ layer can corrupt non-ASCII paths due to ANSI code page round-tripping. It is recommended to resolve and normalize all paths in the JavaScript layer using node:path before passing them to the native binding.

[github-actions]

📊 Benchmark Results

get-sync.bench.ts

getSync() > random keys - small key size (100 records)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 lmdb	1	24.38K ops/sec	41.01	39.55	1,949.715	0.133	121,915
🥈 rocksdb	2	11.19K ops/sec	89.34	86.00	24,156.369	0.976	55,965

getSync() > sequential keys - small key size (100 records)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 lmdb	1	28.31K ops/sec	35.32	34.20	520.156	0.099	141,545
🥈 rocksdb	2	11.14K ops/sec	89.78	87.33	2,434.336	0.105	55,692

ranges.bench.ts

getRange() > small range (100 records, 50 range)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 lmdb	1	24.56K ops/sec	40.71	36.51	1,643.372	0.292	122,825
🥈 rocksdb	2	16.27K ops/sec	61.45	54.00	1,070.446	0.119	81,371

realistic-load.bench.ts

Realistic write load with workers > write variable records with transaction log

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	344.94 ops/sec	2,899.072	74.76	70,486.212	16.59	690
🥈 lmdb	2	26.85 ops/sec	37,246.872	432.678	1,185,040.898	136.306	64.00

transaction-log.bench.ts

Transaction log > read 100 iterators while write log with 100 byte records

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	39.48K ops/sec	25.33	11.93	20,914.439	0.849	197,383
🥈 lmdb	2	446.62 ops/sec	2,239.016	179.01	16,705.78	1.29	2,234

Transaction log > read one entry from random position from log with 1000 100 byte records

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	784.98K ops/sec	1.27	1.10	4,474.274	0.198	3,924,902
🥈 lmdb	2	440.31K ops/sec	2.27	1.10	5,428.855	0.801	2,201,546

worker-put-sync.bench.ts

putSync() > random keys - small key size (100 records, 10 workers)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	831.56 ops/sec	1,202.563	1,020.423	3,725.859	0.477	1,664
🥈 lmdb	2	1.15 ops/sec	871,047.528	854,308.75	898,816.079	1.15	10.00

worker-transaction-log.bench.ts

Transaction log with workers > write log with 100 byte records

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	23.56K ops/sec	42.45	29.76	20,537.305	2.09	47,115
🥈 lmdb	2	848.63 ops/sec	1,178.371	195.787	18,582.525	5.37	1,698

---

Results from commit 8de4e3a

[kriszyp]

Thanks — four of the five were real and are fixed at d725c32; the fifth is in the PR body for a ruling.

• destroy() after close() — fixed, and the root cause turned out to be wider than a missing capture: any per-handle snapshot goes stale the moment another open appends a storage path. The process now remembers each database's layout by path (DBRegistry::knownLayouts), so close() → reopen with [A, B] → close → destroy() on the first handle deletes both volumes. Covered by two cases, both mutation-checked.
• prepopulate_blob_cache leak — fixed as suggested; blob_cache was inheriting the same way past a noBlockCache request and got the same treatment.
• The regression test passing with the fix removed — confirmed by mutation, and seeding t2 first was not enough on its own: the later plain reopen asks for the 2 KB default anyway and writes the correct value back, so it still passed. It now asserts against the OPTIONS file, which is where the restamp actually lands.
• Removing/reordering paths — explainOpenFailure now appends your suggested guidance to any Corruption naming a .sst. Appended and conditional rather than substituted, since genuine corruption gives the same status.
• The derived blob cache — left for @kriszyp with the two alternatives (carve it out of blockCacheSize, or default to 0 and require opt-in) written up in the PR body.

Chasing the flat-family case in your third comment also turned up a bigger one: allowDirChange skipped its own "checked, not trusted" guard whenever the persisted blob_dir was empty — which is every family of an untiered database, so the flat-to-external move, the only migration those databases can make, was the one form taken on faith. Both sides now resolve an empty directory to the database directory before comparing.

— Claude Opus

[cb1kenobi]

Re-reviewed d725c32ab9882d7665039f183b336543dd9b0068 (round 12). Every prior finding was re-measured on a local build rather than read for the absence of the old code — 0 Critical, 0 High, 1 Medium open.

Confirmed fixed, by measurement

High — close() then destroy() orphaned every tiered SST. All three sequences now clean up, where the middle one was the bug:

sequence	round 11	d725c32a
destroy-while-open	1 → 0	1 → 0
close-then-destroy	1 → 1	1 → 0
second-handle-open	1 → 0	1 → 0

Proved load-bearing rather than incidental: disabling only the knownLayouts fallback in DestroyDB and rebuilding reproduces the old result exactly (close-then-destroy back to 1 → 1, the other two unaffected). The cache lifetime holds up too — it survives shutdown() (1 → 0), and reopening a path with a different list overwrites it, so it cannot go stale. Reverting the PurgeAll clear in a5fbf362 was the right call; the measurement agrees with the reasoning in that commit message.

Medium — prepopulate_blob_cache leaking into warm-created families. Fixed, and the new test at tiered-storage.test.ts:620 is load-bearing: restoring the one-sided assignment fails it with expected 'kFlushOnly' to be 'kDisable', i.e. the leak persisted into the OPTIONS file, reproduced exactly.

Low — the regression test that passed with its own fix removed. Now proves its claim. Making RecordLayout first-writer-wins (a per-handle layout) fails it with 8 orphaned SSTs on the appended volume.

Medium — removing or reordering paths. Removal and reorder still fail the open, which is inherent since db_paths is never persisted, but the corruption status now carries the explanation and the recovery, and restoring the list does recover the database. Verified all four cases directly. Appending the note rather than substituting it, and phrasing it conditionally, is the right call given a genuinely corrupt database produces the same status.

Still open

Medium — the derived blob cache is capacity on top of the block cache (db_settings.cpp:240, already commented inline). Re-measured: config({ blockCacheSize: 100 MB }) with blobCacheSize never set gives a 100 MB block cache plus a separate 10 MB blob cache — total ceiling 110 MB, +10% on upgrade with no code change. The blobCacheSizeExplicit latch is correct (an explicit blobCacheSize: 0 leaves no blob cache attached), so only the never-set case is affected — which is every existing caller. load-binding.ts:631-638 now documents this accurately, but the thread was resolved without a reply, so it reads as unanswered rather than decided. If the +10% is intended it is a release-note item and can be closed as accepted — worth saying so explicitly rather than leaving a silent resolve to stand in for the decision.

Coverage is still the headline risk

Local run is green — 62 files, 846 passed / 22 skipped / 0 failed, plus native 163 passed / 3 skipped (15 OptionsFile cases). But skips have grown 9 → 19 → 21, and all 21 are the blobs.dir block gated on the blob_dir patch. Of the seven tests this round adds, five run and two do not:

• skipped: should refuse to relocate before the blob files have moved
• skipped: should refuse to relocate a flat family before its blob files have moved

So the relocation guards added by 648678d2 / 1f17124a / 9d94879e / cdcea6be are static-only on every build without the patch, which is every build until rocksdb-prebuilds#13 — Add RocksDB blob_dir patch so blob files can live on a separate volume lands. That PR is still open and is now waiting on @cb1kenobi. The ROCKSDB_JS_REQUIRE_BLOB_DIR=1 gate is the right guard so a release cannot silently ship without the feature, but it does not make these tests run in the meantime — green CI is not evidence for the blobs.dir half.

—
Generated by Barber AI

[kriszyp]

Addressed the round-12 executable-coverage gap in ccdbcc3. The patched-only relocation decision now lives in Node-free core/blob_relocation.cpp, with its filesystem predicates injected, and the unpatched native test binary executes 18 cases covering the two skipped guards plus the full relocation blast radius, flattening, stale acknowledgements, and missing-directory failures. No relocation behavior changed. Local verification: pnpm build; pnpm test:native (184/184, including all 18 new cases); pnpm test (844 passed / 24 skipped); pnpm check.