Skip to content

Give the dual-clock audit suite a monotonic allocator for fabricated origin log keys, fixing the shared-key lookup flake - #2530

Merged
kriszyp merged 1 commit into
mainfrom
kris/2412-dualclock-lookup-flake
Sep 8, 2026
Merged

Give the dual-clock audit suite a monotonic allocator for fabricated origin log keys, fixing the shared-key lookup flake#2530
kriszyp merged 1 commit into
mainfrom
kris/2412-dualclock-lookup-flake

Conversation

@kriszyp

@kriszyp kriszyp commented Sep 8, 2026

Copy link
Copy Markdown
Member

one applied transaction can carry writes at different record versions in unitTests/resources/dualClockAuditRecord.test.js failed on the Node 26 leg of the Unit Test run for #2526 with lookup scans every entry sharing the transaction log key, passed on rerun, and reproduces locally at about 1 in 20 runs pinned to two cores with CPU burners. Test-only change; no product code.

Mechanism

Every test in the suite fabricated its origin log keys as Date.now() ± n. Under load, an earlier test's key can equal a later test's: on the failing run the invalidate test's Date.now() + 20 landed on the same millisecond as this test's Date.now() + 1, nineteen milliseconds later. RocksTransactionLogStore.getSync reads only the contiguous run of entries at a key, which is correct for a real log, where one key names one transaction. Here that run was the invalidate test's transaction, and batched-new sat further down the log under the same key, never reached.

The fix mirrors the production property the lookup relies on. All fabricated keys come from one suite-level allocator, originClock, stepping 100 s per call from three hours in the past, so no two tests can share a key and the offsets a test subtracts (at most 60 s) never cross the previous anchor. The subscriber-delivery test now subscribes with omitCurrent, which leaves the subscription's high-water mark at zero, so it takes an allocator key like every other test and the suite writes no future-dated record.

For the human reviewer

  1. Fixed in the fixture, not the store. The contiguous-run read in getSync assumes an origin log never holds two transactions at one key. Origin keys come from a per-node monotonic clock, so that holds in production; this suite was the only place two transactions shared a key. If you read it as a production lookup gap worth hardening, that is a product change and a different PR.
  2. Anchors are past-dated, not future-dated. Past keys stay below every real-time write, so nothing outlives the run and no later subscription on the table is gated. The costs: the suite's local log is appended out of key order (a state production never reaches), keys older than the retention window skip the RocksDB write-time dedup scan (none of these tests assert on dedup), and the subscriber test needs omitCurrent. Reversible in one line.
  3. The subscriber test no longer covers the replay-to-live handoff. With omitCurrent it proves event.version and event.localTime on a live event only. The default-subscribe form needed a key ahead of the wall clock, which the earlier rounds of this PR tried at +1 s and +1 h and rejected as a landmine for later subscribers.
  4. Uniqueness is a convention, not an asserted invariant. The allocator does not record keys and fail on reuse, and a future test that fabricates its own Date.now() key re-arms the flake. Declined an assertion in auditEntriesFor to keep the diff to the flake; cheap to add later.

Verification

check result
suite on base, 2 cores + 2 CPU burners 3 failures in 52 runs, all lookup scans every entry sharing the transaction log key
instrumented failing run the exact-start scan yielded applied-invalidate-head-1 at the colliding key, then an entry at a different key, then stopped; batched-old and batched-new were later in the log
suite with fix, same contention 0 failures in 160 runs across four rounds
suite with fix, uncontended 20 of 20 pass
test:unit:resources, RocksDB 2190 passing, 30 pending
LMDB pass of the file 1 passing, 16 pending, as before
lint:required, format:check clean

Fails-on-base is the contended-loop row: the test asserts the same things as before, only its keys changed.

Five pre-push review rounds (codex + gemini, cursor-composer on rounds 1 and 2, Harper domain adjudication on each). Round 1 widened the anchor budget and corrected the comment's claimed invariant. Round 2 pulled the subscriber key back from an hour ahead after Gemini pointed out a far-future record gates any later subscription on the table. Round 3 fixed the comment's description of the scan and a 120 s version offset. Round 4 converged and proposed omitCurrent, adopted in round 5, which converged again.

Complexity: easy

Review-Coverage: authored=claude; ran=gemini,codex; adjudicated=domain; declined=cursor-grok,cursor-composer; rounds=5 @ 8f65302

Human-Review-Need: 3 (decisions: anchor-direction-past-vs-future, omit-current-vs-default-subscribe, shared-allocator-vs-per-test-constants, flake-fix-layer) @ 8f65302

…origin log keys (harper#2412 flake)

Every test fabricated local-log keys as Date.now() +/- n. Under load two tests
could land on the same key, and auditStore.get stops scanning at the first entry
of a different transaction sharing the key, so 'one applied transaction can carry
writes at different record versions' intermittently failed with 'lookup scans
every entry sharing the transaction log key'. Keys now come from one allocator
that increases 100 s per anchor, matching the uniqueness an origin's own log
keys have.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@kriszyp kriszyp added this to the v5.3 milestone Sep 8, 2026
@kriszyp
kriszyp requested review from cb1kenobi and ldt1996 September 8, 2026 16:41

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the dual-clock audit record unit tests by introducing an originClock helper function. This helper generates unique local log keys starting from three hours ago, spaced 100 seconds apart, replacing direct calls to Date.now() across multiple test cases to prevent key collisions. Additionally, a subscription in one test is updated to use { omitCurrent: true } to skip replay. The reviewer suggested replacing the magic numbers used for time intervals in originClock with explicit calculations to improve readability.

Comment thread unitTests/resources/dualClockAuditRecord.test.js
@kriszyp
kriszyp marked this pull request as ready for review September 8, 2026 17:05
@kriszyp
kriszyp merged commit ba48a1c into main Sep 8, 2026
51 checks passed
@kriszyp
kriszyp deleted the kris/2412-dualclock-lookup-flake branch September 8, 2026 17:05
@claude

claude Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Reviewed; no blockers found.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant