Apply audit retention continuously to RocksDB transaction logs (v5.2)

[kriszyp]

Backports Apply audit retention continuously to RocksDB transaction logs to the v5.2 release line. On RocksDB, audit retention ran exactly one purgeLogs() per boot: the pass returned before scheduling its successor, so a long-lived node's transaction logs grew without bound until a restart or a disk-pressure signal. The Rocks pass now re-arms on a cadence derived from the pressure-adjusted retention window, and every teardown path — closeDatabase, dropDatabase, and the legacy per-table drop — retires the loop and deregisters its storage-reclamation handler before releasing the stores it holds.

v5.2 already pins @harperfast/rocksdb-js 2.8.0 (6ca418d), which is the version #2338 needed for the purge extent-resolution fix, and msgpackr 2.0.6. No dependency change is required here, and the unreleased rocksdb-js Keep transaction-log retention on a durable sequence floor is not a prerequisite — it is a further safety improvement on top (contiguous-prefix deletion), not something continuous retention depends on.

For the human reviewer

The cherry-pick did not apply cleanly; five files conflicted, all because main carries work v5.2 does not.

1. Whether the behavior change belongs on v5.2 at all. This carries one cadence change alongside the crash-safety and handler-leak fixes: v5.2 nodes gain a permanent background purge loop they did not have before, where Rocks previously purged only on a pressure event. I kept Apply audit retention continuously to RocksDB transaction logs #2338 whole rather than splitting the loop from the fixes, because the fixes exist because of the loop — the teardown races and the reclamation-handler leak only become reachable once a pass can be in flight at close time — and a split would ship half of a reviewed change. Reversible only by a further cherry-pick, not a local revert, so it is worth ruling on before merge.

2. removeStorageReclamation() is new to v5.2. Apply audit retention continuously to RocksDB transaction logs #2338 calls it from three teardown sites, but the helper was added to main by Let a database be opened into a caller-owned table graph instead of the global map #2285/Keep a table being created invisible to catalog scans on other threads, so replication can never announce a partial attribute list #2381, not by Apply audit retention continuously to RocksDB transaction logs #2338. I copied the function verbatim from main into server/storageReclamation.ts rather than open-coding reclamationHandlers.delete(path) at each call site. It is a pure addition — nothing on v5.2 called it before — so inlining would only duplicate the same one-liner three times and diverge the two branches' modules for no gain.

3. The branch-database hunks are dropped. Apply audit retention continuously to RocksDB transaction logs #2338 added a stopAuditCleanup() call inside closeBranchHandles() plus its openBranchDatabase test additions. Neither the function nor the suite exists on v5.2 (harper#643 is main-only), so there is nothing on this line to retire; the alternative would be backporting the whole branch feature.

4. Two tests moved out of the openBranchDatabase suite. Apply audit retention continuously to RocksDB transaction logs #2338 put deregisters the storage-reclamation handler when an ordinary database is closed and … when a database is dropped inside that describe, but they exercise closeDatabase/dropDatabase, not branches — they were there for the fixture. On v5.2 they get their own storage-reclamation deregistration on teardown describe with the same setupTestDBPath() / setMainIsWorker(true) before. The behavior under test is unchanged; only the enclosing block is.

5. Table.ts's import keeps only what v5.2 uses. Apply audit retention continuously to RocksDB transaction logs #2338's version of that import lists removeStorageReclamationHandler because main's Table.ts already used it; v5.2's does not, and importing it fails lint as unused. Same reason removeAuditEntry is absent from auditLog.test.js's import here — the main-only harper#F-264 removal-loop tests that used it are not on this line.

Two smaller adaptations, for completeness rather than decision: DESIGN.md gains only #2338's new retention section (the removal-loop section it sat below is main-only), with its one sentence about branch close() trimmed; and #2338's unitTests/resources/query-array-scoping.test.js formatting commit is dropped, since it repaired a main-only Format Check break against a file that does not exist on v5.2.

resources/auditStore.ts applied with no conflict, and diffing this branch's copy against origin/main's leaves only pre-existing main-only differences (the tombstone-removal tracking and the put: 4 audit type, both from other PRs) — #2338's content is carried in full.

The pre-push review's two surviving findings — that the Rocks integration test asserts purge cadence rather than actual segment deletion, and that #2338's added comments are denser than Harper's zero-new-comment default — both describe code that is already merged on main. I left them alone deliberately: fixing them here would make v5.2 diverge from main for the same feature. They belong on main first, if at all.

Verification

End-to-end route: the existing integration suite, extended by #2338's own new RocksDB retention test.

• npx harper-integration-test-run "integrationTests/database/audit-retention-*.test.ts" "integrationTests/database/blob-delete-reclaim-audit-expiry.test.ts" — 8 tests, 8 pass, 0 fail. That includes Apply audit retention continuously to RocksDB transaction logs #2338's new RocksDB audit retention self-rearms without storage pressure (#2140) suite, which drives a live instance and asserts more than one passive purge pass.
• npx mocha unitTests/resources/auditLog.test.js unitTests/resources/databases.test.js — 43 passing / 1 pending on RocksDB (the pending one is the LMDB-only drain-barrier test), and 35 passing / 9 pending under HARPER_STORAGE_ENGINE=lmdb. All five new Rocks cadence/retirement tests ran and passed.
• npm run test:unit:resources — 1718 passing / 17 pending / 0 failing. Under HARPER_STORAGE_ENGINE=lmdb (the test:unit:lmdb gate's first leg) — 1464 passing / 130 pending / 0 failing.
• npm run test:unit:main — 4525 passing / 195 pending / 2 failing. Both failures are known local-environment artifacts unrelated to this diff: nonInteractiveSpawn git credential scoping (a tmpdir leak between runs) and configValidator getDomainSocketPathLengthWarning (sensitive to the checkout path's length). Neither touches audit retention.
• npm run test:unit:server — not run to completion: unitTests/server/threads/fixtures/processGroupOwnerWorker.js is picked up by that suite's glob and throws at load on v5.2 before any test runs. Pre-existing, not in CI's test:unit:all, and untouched by this diff; npx mocha unitTests/server/storageReclamation.test.js — the file that actually covers the changed module — passes 44/44.
• test:unit:apitests did not run: these runs used a private HOME so the shared system database on this box could not skew them, and that root has no harper install, so the API tests' server bootstrap has no hdb_role to read.
• npm run build, npm run typecheck, npm run lint:required, and npm run format:check all clean.

Complexity: complicated

_{Review-Coverage: authored=claude; ran=gemini,codex; adjudicated=domain; blocked=cursor-grok(failed); declined=cursor-composer; rounds=1 @ 221ebf1}

_{Human-Review-Need: 3 (decisions: v5-2-cherry-pick-scope, closedatabase-engine-invariant, rocks-cadence-pressure-floor, rocks-only-rearm-guards) @ 221ebf1}

[gemini-code-assist]

Code Review

This pull request establishes an engine-independent, self-rearming audit retention cleanup lifecycle for both LMDB and RocksDB. It refactors scheduleAuditCleanup to support RocksDB log purging on a continuous cadence, introduces a stopAuditCleanup method to safely retire the cleanup loop and return a drain barrier, and ensures proper cleanup of storage reclamation handlers via removeStorageReclamation during database/table teardown (close or drop). Extensive integration and unit tests have been added to verify these behaviors, including cadence holding, retirement, disk pressure adjustments, and resource cleanup. There are no review comments, so I have no feedback to provide.