Add a bottommost option to compact() so existing data can be re-encoded

[kriszyp]

compactRange passed a default-constructed CompactRangeOptions, whose bottommost_level_compaction is kIfHaveCompactionFilter. With no compaction filter installed that skips the bottommost level — where the bulk of the data sits — so compact() could not rewrite it.

That matters because a column family's compression governs newly written files only. Existing SST and blob files keep the codec they were written with until something rewrites them, and nothing available could. Changing the codec on a database that already held data left that data encoded as before, indefinitely.

Measured on 60k compressible records written uncompressed and reopened as zstd:

written uncompressed:            34.8M
reopened as:                     {"algorithm":"zstd"}
after plain compact():           34.8M   <-- bottommost skipped
after compact({bottommost:true}) 2.0M    <-- re-encoded
compactSync({bottommost:true})   2.0M    (idempotent)

// Re-encode everything already on disk under the column family's current codec
await db.compact({ bottommost: true });
db.compactSync({ bottommost: true });

Opt-in rather than the default because it rewrites the entire range regardless of whether RocksDB judges it worthwhile, so it costs as much as the data is large — it should be a deliberate operator action, not something an upgrade or a routine compaction does implicitly.

Why this is needed

harper #2044 makes Harper honor storage.compression on databases upgraded from 5.1, which had been silently ignored — an existing column family inherits its persisted codec on reopen rather than adopting the build default, so those instances were writing uncompressed indefinitely. That fix compresses new writes, which is the right default. This gives operators the separate, explicit action to convert the data already on disk; without it there is no path from an existing uncompressed database to a compressed one short of a dump and reload.

Changes

• db_descriptor.cpp / .h — compactRange takes a bottommost flag and sets BottommostLevelCompaction::kForce
• database.cpp / .h — the flag is read from an added trailing argument on both Compact and CompactSync, and carried on AsyncCompactState
• store.ts — CompactOptions.bottommost?: boolean, forwarded to both native calls
• load-binding.ts — native signatures updated
• README — documents the option, and adds a note under Compression that a codec change affects new files only, and that omitting compression on an existing family inherits its codec rather than applying the default

Testing

test/compaction.test.ts gains a case that writes uncompressed, reopens under zstd, asserts a plain compact() leaves the SST bytes exactly unchanged, then asserts { bottommost: true } drops them below half — through both compact() and compactSync(). Skipped when the native build lacks zstd. 39 tests pass across the compaction and compression suites.

Verified against a RocksDB prebuild carrying the compression codecs (11.1.2); the repo's currently vendored prebuild has only none/zlib, so the new test self-skips there.

Generated by Claude Opus 5.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request adds support for a bottommost option in the compact and compactSync methods, allowing users to force compaction of the bottommost level in RocksDB to re-encode existing data under a new compression codec. The changes span the C++ bindings, TypeScript wrappers, documentation, and tests. The feedback highlights a compatibility issue in the newly added tests where entry.parentPath is used, which will fail on Node.js 18; a fallback to entry.path is recommended to maintain compatibility.

[github-actions]

📊 Benchmark Results

get-sync.bench.ts

getSync() > random keys - small key size (100 records)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 lmdb	1	24.05K ops/sec	41.57	39.93	635.69	0.117	120,265
🥈 rocksdb	2	11.22K ops/sec	89.16	86.22	531.365	0.053	56,080

getSync() > sequential keys - small key size (100 records)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 lmdb	1	28.37K ops/sec	35.25	34.24	711.864	0.107	141,847
🥈 rocksdb	2	10.70K ops/sec	93.47	91.02	3,070.437	0.129	53,493

ranges.bench.ts

getRange() > small range (100 records, 50 range)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 lmdb	1	25.14K ops/sec	39.78	35.56	1,987.968	0.294	125,694
🥈 rocksdb	2	17.18K ops/sec	58.20	52.05	2,064.861	0.146	85,918

realistic-load.bench.ts

Realistic write load with workers > write variable records with transaction log

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	379.41 ops/sec	2,635.67	100.744	62,534.486	16.82	759
🥈 lmdb	2	26.60 ops/sec	37,587.795	345.791	1,203,351.882	136.43	64.00

transaction-log.bench.ts

Transaction log > read 100 iterators while write log with 100 byte records

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	38.54K ops/sec	25.95	11.90	13,419.212	0.579	192,701
🥈 lmdb	2	439.52 ops/sec	2,275.217	209.516	24,063.895	1.52	2,198

Transaction log > read one entry from random position from log with 1000 100 byte records

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	712.52K ops/sec	1.40	1.21	457.332	0.064	3,562,614
🥈 lmdb	2	455.79K ops/sec	2.19	1.11	8,513.426	0.561	2,278,943

worker-put-sync.bench.ts

putSync() > random keys - small key size (100 records, 10 workers)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	851.85 ops/sec	1,173.915	1,034.83	2,072.79	0.337	1,704
🥈 lmdb	2	1.15 ops/sec	867,978.82	835,390.772	924,588.181	2.33	10.00

worker-transaction-log.bench.ts

Transaction log with workers > write log with 100 byte records

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	23.78K ops/sec	42.05	29.46	472.173	0.572	47,560
🥈 lmdb	2	808.09 ops/sec	1,237.488	214.641	11,894.52	5.46	1,618

---

Results from commit 59666ea

[cb1kenobi]

Reviewed e2168272 — no issues found. This PR looks good, nice job!

Confirmed the native path: kForceOptimized forces bottommost compaction of pre-existing files (only skipping files the same run produced), the forced blob GC with age_cutoff = 1.0 extends re-encoding to blob files, and the bottommost=false path is byte-for-byte the prior default-constructed CompactRangeOptions — no regression to the existing compact(). Threading (worker-thread execute callback, error propagated via state->status), memory safety, the opt-in default, and input validation all check out, and the SST + multi-generation blob tests (with value-integrity assertions) cover both entry points.

—
Generated by Barber AI