Cluster record locks: the Phase 1 substrate, with Ricart–Agrawala superseded by amortized ownership (#483)

[kriszyp]

Phase 0 (#2462) made the rocksdb-js in-memory key lock the sole authority for table.lock(id),
exclusive across one node's worker threads. This branch carries that verb across every node that
replicates the database, over three new transaction-log control entries.

The arbitration rule on this branch is superseded and will not ship as written. Ricart–Agrawala
waits for a grant from every participant, so any single unreachable peer blocks every cluster lock
— harper-pro#822 already records that in its own limitations list — and an acquisition costs P+1
durable commits and up to P²−1 frame deliveries with no amortization for a node that locks the same
record repeatedly. At P=12, a cluster size Harper runs today, that is 13 durable commits and 143
frame deliveries per lock. A lock any one unreachable peer disables is not usable for the
correctness-critical work lock() exists for, so the availability limit is the feature's value rather
than a rollout caveat.

docs/record-lock-ownership.md
specifies the replacement, and is the substance of this push. Three levels at three very different
rates:

• Membership epoch (rare, durable, agreed) — per database, (number, members[], ringVersion)
  under single-decree agreement over a majority, with acceptor promises and accepted values persisted
  before they are acknowledged. Consensus appears here and nowhere else: once per membership change
  plus one renewal per node per lease period, never per key and never per lock.
• Home node (derived, free) — within an epoch the arbiter for a key is a rendezvous hash over
  members[]. One arbiter per key is trivially exclusive, which deletes the entire grant state
  machine: no deferral queues, no (tsR, nodeName) tiebreak, no synthesized grants, no split votes,
  no revocation protocol.
• Delegation (volatile, the amortization) — a node asks a key's home for the exclusive right to
  admit critical sections on that key for a bounded time. With a live delegation lock()/unlock()
  are pure Phase 0: the local key lock, zero cluster messages. Releasing the application lock does
  not release the delegation, so a node writing the same record repeatedly pays one round and then
  nothing, and the delegate is in practice the last writer.

Steady-state cost goes from P+1 durable commits and P²−1 deliveries per lock to zero of each; a
first lock on a cold key is two unicast messages and one RTT to the home; a handoff is one durable
release plus P−1 deliveries. A node that is down blocks only its own share of the ring, until the
epoch advances.

This is close to what harper-pro#438 filed as its own "Phase 1 — single-owner delegation", and to
what #483 describes in prose ("if an exclusive lock is held by one other node, the lock can be
requested from that node"). It differs in deriving the home from replication-group membership rather
than from residency or server.shards, so it needs no operator sharding configuration — only one
customer uses sharding today.

What this push contains

• docs/record-lock-ownership.md
  — the design note, through three planning rounds. Round 2 returned better-alternative-exists
  against a stateless epoch foundation and is adopted: a restarted acceptor that forgets its
  acceptance forks the configuration, and no timing argument substitutes for durable state
  (§4.0 carries the counterexample so it is not reinvented). Round 3 returned
  Framing-Verdict: chosen-approach-sound; its surviving protocol findings are folded into §§4.2,
  4.4, 6, 7.2, 7.3, 8.
• DESIGN.md
  — the Phase 1 section now opens by saying its arbitration rule is superseded and pointing at the
  note, so the repo's own design doc does not read as a settled decision.

No code changed in this push. The Ricart–Agrawala state machine is still here, still gated off, and
is removed when the replacement lands — the note's §11 is the line-by-line split.

What survives the arbitration change, and what does not

Reused unchanged (roughly 60% of the diff, and already through fifteen review rounds):

• The three control entries end to end — the nibbles, the private Packr, recordId: null with the
  key in the payload, receive routing off the record path, and the filters on every surface that
  reports audit entries as record activity.
• The commit-time lease fence in Table.ts / DatabaseTransaction.ts. It moves from a per-round
  lease to a delegation lease with no change to the mechanism.
• recordLock.ts's Phase 0 primitives, the scope option, the monotonic lease deadline, the
  contained release hook, lock() coalescing, and the LockUnavailableError / 423 / 409 contract.
• ClusterLockTransport
  as the boundary, the per-database registry, fail-closed membership validation, and
  ownsCoordination() thread ownership.

Removed when the replacement lands: LOCK_REQUEST/LOCK_GRANT, per-peer round tracking, deferral
queues, (tsR, nodeName) ordering, synthesized grants, withdraw-on-timeout, and the agreedDown
DOWN-exclusion rule this branch's last commit added.

For the human reviewer

• The guarantee decision is made (2026-09-09): exclusion-only. lock() guarantees that at most
  one node admits a critical section for a key at a time, and that a node admitted after another
  node released cleanly has already applied that node's committed writes — but only while the key's
  home still holds that handoff's dependency set. It adds no fencing generation to conflict
  resolution and does not confirm locked writes to a quorum, so two things are part of the documented
  contract. Neither is crash-only, and both are reachable on a completely clean handoff:

  1. a predecessor's write can outrank its successor's under LWW, because LWW compares the timestamp
     assigned when the write was staged and lease expiry orders admissions, not timestamps.
     lock() changes nothing about conflict resolution — whatever two conflicting writes would
     do to the record without a lock is what they do with one, silently. (1a) the predecessor's
     clock ran ahead and its write is still in flight; (1b) it stamped a future
     context.timestamp — deliberate Phase 0 behavior — then committed, replicated, drained and
     released cleanly, and the successor's write is the one that loses. No crash, no skew, nothing
     in flight;
  2. successor freshness is not promised once the dependency set is gone, because the recovery
     barrier only drains streams from reachable members. Three routes: the predecessor crashed, it
     is unreachable, or its native commit settled after the barrier was measured. Two damaged effects
     on the one record — the predecessor's transaction not reflected, and the successor's write
     computed from the stale value it read.

  There is no caller-side mitigation for (2) at all. X-Replicate-To / confirm= is
  super-user-gated (checkContextPermissions, resources/Table.ts:7258 — a 403 for an app caller),
  and for a super-user it is a residency directive first: a numeric value truncates the record's
  residency (getResidency, :1574) and * falls back to the database's configured
  replication.replicateTo. Either way a successor's barrier can satisfy over members that never held
  the locked write. Even with cluster-wide residency, confirmation closes none of (2)'s three routes —
  the late-settling commit needs fencing, so §2's invariant as written needs both deferred arms.
  Both limitations also ship silent — a 200, no log line, no counter — which §10 records as a
  deliberate choice with the cheap lock-path-only detection routed to Record locks Phase 1: rendezvous home ring, per-record delegations, and drain/recall #2541.

  The normative text is §10 of the note, which DESIGN.md points at rather than restating.

  Ten pre-push rounds ran on this push, and the first nine each found the contract text promising
  something the code does not. In order: crash-only; the late-settling commit filed under
  the wrong limitation; clean-handoff safety; the LWW mechanism sentence stated backwards; the
  drop-vs-fold rule splitting on the wrong operand; a whole-write loser guarantee false for a patch on
  disjoint fields; a per-field one false for CRDT ops; a recommendation to use
  X-Replicate-To: N;confirm=M that would have narrowed the record's residency and made the
  limitation more reachable; that the same header is super-user-gated, so it was never a
  caller-side mitigation in the first place; and — round 10, in code rather than in the note — that
  the super-user gate itself is bypassable. It converged only when the contract stopped restating
  engine conflict-resolution and said the thing that is actually true — lock() does not change how
  conflicting writes resolve, and there is no lever a caller can pull. That history is the argument
  for §12's schedules asserting the limitations as executable expectations rather than leaving them in
  prose: a paragraph about a conflict-resolution edge does not stay true on its own, and this one is
  going into user-facing API documentation.

  The choice also removes an exception §8 was carrying: with no fencing generation, ordinary writes
  keep their ungated path with nothing added, which is the property the fenced arm would have given
  up.

• One earlier claim in this PR's body is withdrawn. It said a crashed holder's in-flight write is
  stamped older than any successor's and so resolves under LWW exactly as if the release had not
  overtaken it. That holds for a clean release. It does not hold for a crash: if the crashed
  holder's clock runs ahead, its delayed write can carry the greater timestamp and overwrite the
  successor. Monotonic lease expiry orders admissions, not timestamps — §7.3 is that hole, and the
  table above is the choice about it.

• The freshness property this branch gets for free is the one the new transport must pay for.
  A grant here rides the grantor's own replication stream behind the grantor's data writes, so
  applying a grant implies having applied that grantor's earlier writes to the key. Unicast
  delegation messages lose that, and a scalar record version does not restore it — core breaks
  equal-version conflicts by node name, so a replica can hold a losing value at the same
  timestamp and pass a version ≥ V test. §7.1 replaces it with an inherited (origin → position)
  dependency set on the LOCK_RELEASE entry.

• The work is decomposed and the remaining blocker is measurement. Record locks Phase 1: rendezvous home ring, per-record delegations, and drain/recall #2541 (home ring,
  delegations, drain and caps, plus the three inherited substrate defects below), Record locks Phase 1: successor freshness — inherited dependency sets and the recovery barrier #2542 (successor
  freshness), Record locks Phase 1: durable membership epoch protocol (single-decree agreement per database) harper-pro#825 (the membership epoch protocol), and
  Record locks: measure the Phase 1 cost baseline before the protocol change harper-pro#824 (the measurement gate, the only piece unblocked today).

• The measurement gate comes before the protocol change, not after it. No number in the note is a
  benchmark; they are message counts. Acquisition latency on a real cluster, audit growth per lock,
  and throughput with the feature disabled are the first deliverable, and they also set the baseline
  the new design has to beat.

• The pre-push review ran fifteen rounds on the code that is here. Roughly half the defects it
  found were introduced by an earlier round's own fix, which is a property of this state machine
  rather than of the review — and it is the strongest single argument for a design whose arbiter is
  one node instead of a quorum. That history is preserved in the commit log; the note's §9 records
  why quorum voting, a replicated Raft coordinator, and a server.shards-based owner map were each
  considered and not chosen.

• Three defects the cross-model review found in the substrate, not in the rule being deleted.
  They are recorded in §11 of the note as obligations on the replacement, and are not fixed in this
  push — the branch does not merge with main yet and two of the three are touched by the
  replacement anyway. Named so they are not lost: re-registering a transport closes the current
  coordinator and installs an empty one without invalidating the handles the old one issued, so a
  successor can grant the same key with no lease time elapsed (resources/Table.ts:5474); a
  LockUnavailableError from coordinator construction escapes the direct receive callback, which the
  subscription sink already contains (resources/recordLockCoordinator.ts:908); and the commit-time
  lease fence scans the whole write set on every commit, including in core-only deployments that
  never register a transport, which contradicts the ungated-path goal (resources/DatabaseTransaction.ts:1213).
  The review's other four majors are inside the Ricart–Agrawala state machine and are deliberately
  left alone.

• Two substrate decisions from the original design still stand and are still worth a check.
  Control entries carry recordId: null and the locked key in the payload, because a control entry
  carrying the key answers _writeUpdate's keyed dedup lookup at exactly ts_R and silently drops
  the holder's own first write. And node identity is the node name, never the audit nodeId:
  short ids are per-node, so a (ts, nodeId) order would order the same pair differently on two
  nodes and both would grant.

One bug this push found in main, filed rather than fixed here

Tracing X-Replicate-To to decide whether it was a usable mitigation turned up
#2546 (P1): checkContextPermissions
(resources/Table.ts:7258) gates on truthiness, so X-Replicate-To: 0 skips the super-user 403
and reaches getResidency with count = 0, pinning the record to the receiving node. Any
authenticated user with write permission can take a record out of replication with one header — 200
response, nothing logged, and the residency persists across updates. Pre-existing on main, unrelated
to this branch, and out of scope here; the fix is != undefined on both guards.

Verification

Docs-only push, so the code gates are the ones already recorded on this branch's head
(f1dd96180), unchanged here:

• test:unit:resources 2109 passing / 29 pending / 0 failing; test:unit:main 5289 passing with two
  environmental failures that pass when re-run alone; record-lock-concurrency integration 3/3.
• unitTests/resources/recordLockCoordinator.test.js (21 tests) and
  unitTests/resources/recordLockCluster.test.js are unchanged and still describe the arbitration
  rule that is being replaced. The note's §12 lists what the replacement's suite must add — most
  importantly independent per-node clocks, which the current shared fake clock cannot express.
• prettier --check on both changed files.

Refs #483

Complexity: complicated

_{Review-Coverage: authored=unknown; ran=none; rounds=1 @ cd3fea6}

_{Human-Review-Need: 4 @ cd3fea6}

[gemini-code-assist]

Code Review

This pull request implements Phase 1 of cluster-wide record locks using a Ricart-Agrawala consensus algorithm over replicated control entries. It introduces a LockCoordinator to manage lock requests, grants, and releases, integrates these control entries into the transaction log and replication flow, and updates transaction committing to enforce monotonic lease deadlines. Feedback on the changes is minimal but highlights a critical issue in resources/Table.ts where an undefined noop reference in a promise catch block could cause a runtime ReferenceError.

[gemini-code-assist]

Is noop defined or imported in this file? If not, it will cause a ReferenceError at runtime. Since lodash is already imported, you can use lodash.noop or simply an arrow function () => {} to safely catch and ignore the rejection.

Suggested change

	Promise.resolve(coordinator.release(id)).catch(noop);
	Promise.resolve(coordinator.release(id)).catch(() => {});