Reproduce the cross-worker drop_table race behind the blob lifecycle CI failures as a skipped unit test (blocked on rocksdb-js#806)

[kriszyp]

drop_table on one worker can drop a table's column families while another worker's source-fill cache write to that table is still committing: getFromSource resolves its caller before the write commits, and dropTable() drains only its own thread's in-flight source commits. Under rocksdb-js's default parallel OCC validation the racing commit is admitted past conflict validation, fails inside RocksDB's memtable inserter, and RocksDB latches that failure as a fatal background error on the whole environment, so every later write to that database fails with Invalid column family specified in write batch: the drop's own catalog cleanup (the 500 in integrationTests/apiTests/blob.test.mjs), every worker's interrupted-drop completion (Failed to complete interrupted drop … attempt 1), and the restarted worker's component load (the Class extends value undefined cascade, because databases.blob.BlobCache never loads). CI runs 33076221779 and 33139269321; the root cause with RocksDB's own log timeline is on harper#1381, and the binding defect is rocksdb-js#806, which reproduces it with no Harper code at all. So the invalid column family is not a stale handle in Harper's drop bookkeeping: it is a real write to a really-dropped family, admitted below Harper.

This PR is test-only. It adds the deterministic in-process reproducer as a skipped unit test with the reason in place: a worker thread source-fills a blob record (the blob write is what delays the native commit) while the main thread drops the table. Unskipped, it fails on iteration 0 with the CI signature on the rocksdb-js 2.8.0 pin main carries today. Beside it, an always-on control drops the table only after the worker's commit has settled, so the fixture's settlement signal, probe write and catalog check run on every resources run rather than waiting for the unskip. It changes no production code: the fix is the drop/commit interlock in rocksdb-js, after which harper bumps the pin and unskips the race test; acceptance for harper#1381 closes there, not here. The unhandledRejection in worker thread lines in the same runs were a second, independent defect in Scope.handleEntry; that fix landed on main in #2432, and this branch's version of it was resolved to main's in the merge commit, so squash on merge: the net change is the two test files.

For the human reviewer

1. Framing verdict (step 6): Framing-Verdict: better-alternative-exists from the planning review. The reviewer agrees the fix belongs in rocksdb-js and that the option set spans the axes (binding interlock; Harper-wide quiescence, the parked Quiesce workers before dropping RocksDB tables #2206; kValidateSerial; 2.8.0 detect-and-resume()). The alternative it names refines the interlock's lifecycle rather than its layer: take per-column-family admission at native commit time, all families in the batch all-or-nothing, released by native scope exit on every path, instead of a touch taken at transactional Put/Delete staging as rocksdb-js#806's first fix direction describes, because a staging-time touch deadlocks a same-thread dropSync() (Harper drops under a synchronous exclusive schema section). Nothing in this PR implements an interlock, so the refinement is recorded on rocksdb-js#806 for the binding task rather than adopted here. What you decide: whether this test-only PR lands now or waits for the binding release; the task owner chose now so the reproducer is in the tree.
2. A skipped race test in the tree, next to an active control. The race test asserts the post-fix contract (environment still writable, catalog clean, no worker error other than a commit that lost to the drop before RocksDB's write path, and at least one iteration that caught the commit in flight) and is it.skip with the reason and both issues in the comment above it. Alternatives: delete it until the binding ships, or make it self-enabling on a binding-version probe. Rejected because the skip has no automatic revival trigger either way and the version is not known yet; the unskip is part of the pin-bump PR, and rocksdb-js#806 links here so it is not forgotten. An active test asserting today's failure signature was also rejected: it would assert the bug and fail in the wrong direction the moment the binding fixes it. Reversal costs deleting two files.
3. The control is fixture coverage, not defect coverage. It cannot race by construction (the worker's commit has settled before the drop), so a green resources run here says the fixture works, not that harper#1381 is fixed. Cheap to change.
4. The fixture uses production startWorker, not the plain new Worker the other unitTests/resources/*-thread.js fixtures use. dropTable() broadcasts a drop to every registered worker thread, each of which holds its own handle to the same column family, and the worker needs to receive that broadcast to behave like a production worker; the cost is production thread-management state in the shared mocha process, which the full resources suite tolerated on every run. Switching later means rewriting the fixture's message protocol.
5. Worker errors are captured by intercepting the logger and allow-listed by message text (CONTAINED_COMMIT_LOSS). This couples the assertion to log wording that is not API; the alternative, asserting only on observables the test already has, was rejected because an unrelated failed source fill would release the record lock and let the race test pass vacuously. Where to look hardest: the pattern names today's contained error (Could not access column family N) and a plausible post-interlock one; the unskip should pin it to whatever the binding actually reports.
6. Settlement and the race sample come from the shared record lock. The worker reports commit-settled once primaryStore.hasLock(id) clears, the seam unitTests/resources/caching.test.js already waits on, and getFromSource releases that lock in both branches of its commit handler. The race test samples the same lock on the dropping thread immediately before dropTable(); that is a sample, so it shows the race was exercised at least once and cannot show that a given iteration overlapped. The deterministic overlap proof belongs to the binding's native tests (rocksdb-js#806).
7. The fixture's sync() is an arrival barrier, not quiescence. A ping/pong round-trip proves everything the worker posted before answering has arrived; it cannot prove the drop will provoke nothing later. That is enough for the control (nothing is in flight when it drops) and the race test uses commit-settled instead; the helper is local to this file and its comment says what it guarantees.
8. A dead fixture worker fails the pending wait, after whatever it already sent. From Gemini's review: the fixture now rejects a pending expect() on the worker's error/exit with the exit code, where a crashed worker used to hang to the 120 s mocha timeout with no cause attached. Messages the worker posted before dying are still delivered first, because after terminates the worker deliberately and that exit must not read as a crash; the cost is that a crash after a test's final barrier is invisible to the fixture, which is why the main-thread probe write stays as the independent check. A reviewer who values fail-fast over delivery order can swap the two checks in next().

Verification

Route (b): the reproducer is the end-to-end artifact. It cannot run green until rocksdb-js#806 ships, so the evidence is fails-on-current-binding for the race test and a passing control on every run.

• Unskipped copy of the race test on the merged tree with rocksdb-js 2.8.0: the control passes, the lock sampled on the main thread before the drop reads true on iteration 0, and the iteration fails with the CI signature. Worker log: Error committing cache update … Invalid column family specified in write batch; test failure: Error: Remove failed: Invalid argument: Invalid column family specified in write batch from dropTable()'s catalog cleanup.
• Committed file under mocha: 1 passing, 1 pending (the control passes with no worker event at all; the race test is pending). Under HARPER_STORAGE_ENGINE=lmdb: no tests registered.
• npm run test:unit:resources on the final HEAD: 1923 passing, 28 pending (the race test among them), exit 0; the suite was also run on each earlier review-round commit with the same result.
• Fixture death path (the Gemini fix): a scratch copy of the suite with one extra test that terminated the worker mid-wait saw the pending expect() reject with fixture worker exited with code 1 immediately; the scratch test was not committed.
• prettier --check and oxlint --deny-warnings clean on both files.
• The red Format Check job is main's, not this PR's: unitTests/resources/query-array-scoping.test.js (from Pin element-scoping semantics of queries over array-valued properties #2437, untouched and byte-identical here) fails prettier --check on main itself, and main's own Format Check runs have been red since that commit. Left for a one-file format fix on main rather than widening this PR.

Refs #1381

Complexity: medium

— Claude Fable 5.1

🤖 Generated with Claude Code

https://claude.ai/code/session_013yFRAueahXUwcVM5BjKPMq

https://claude.ai/code/session_01St6cyvN2KtgxeaboFBFcHN

_{Review-Coverage: authored=claude; ran=codex; adjudicated=domain; declined=gemini,cursor-grok,cursor-composer; rounds=4 @ d513d70}

_{Human-Review-Need: 3 (decisions: shipping-a-skipped-reproducer, control-test-as-the-shipped-coverage, death-does-not-preempt-queued-events, sync-is-liveness-not-quiescence, fixture-uses-startworker, worker-error-capture-by-log-interception) @ d513d70}