Record locks Phase 1: rendezvous home ring, per-record delegations, and drain/recall

[kriszyp]

Part of the Phase 1 record-lock redesign — the replacement for the Ricart–Agrawala arbitration rule
currently on #2498. Design note:
docs/record-lock-ownership.md
§§5, 6, 8. This is the arbitration rule itself: the piece that makes repeated locking by one node
cost zero cluster messages.

What it is

Within a membership epoch (HarperFast/harper-pro#825), the arbiter for a key is a rendezvous hash
over members[] — its home. A node asks a key's home for a delegation: the exclusive right
to admit critical sections on that key for a bounded time. While a delegation is live,
lock()/unlock() are pure Phase 0 — the local rocksdb key lock, zero cluster messages. Releasing
the application lock does not release the delegation, so a node writing the same record repeatedly
pays one round and then nothing, and the delegate is in practice the last writer.

One arbiter per key is trivially exclusive, which is what deletes the entire grant state machine:
no deferral queues, no (tsR, nodeName) tiebreak, no synthesized grants, no split votes, no
contention-resolution protocol.

	durable commits	frame deliveries	latency
Ricart–Agrawala (today, P=12)	13	143	slowest participant
First lock on a cold key	0	2 unicast	1 RTT to the home, 0 if local
Handoff to another node	1	P−1 + 3 unicast	2 RTT
Steady state	0	0	local key lock

Scope

• The home ring, the delegation table, and the wait queue (core).
• transport.requestDelegation(...) and a delegation server — unicast request/grant/recall over the
  existing replication connections. sendOperationToNode is the existing precedent (harper-pro).
• Ordered fencing tokens (§5.1): (epochNumber, homeIncarnation, delegationCounter) with
  homeIncarnation a durably persisted monotonic counter. A random incarnation makes a stale reply
  identifiable but not orderable, and a home that restarts and re-issues generation 1 after having
  issued generation 50 would let a delayed generation-50 write defeat its successor.
• Lease transfer and local handle bounds (§5.2): a handle may not outlive the delegation that
  admitted it.
• Drain must revoke capability, not just close the door (§6). Recall has to invalidate live
  handles, not merely stop issuing new ones; an in-flight native commit is the case the door-closing
  version misses.
• Aggregate caps (§8): the branch's per-key and per-table caps exist to bound what a broadcast
  round accumulates and are removed with it. Bound outstanding delegations per database and per
  requester, bound expiration work per tick, cancel waiters. Eviction is asymmetric and the
  asymmetry is the safety rule — a delegate may drop a delegation early, a home may never forget
  one before its expiry.
• Failure containment as an executable contract: injected synchronous throws and rejections through
  revoke, settlement callbacks, release persistence, shutdown and table removal. Callers settle,
  admission stays closed, no fabricated handoff.
• Versioning the recordLocks capability so the two arbitration rules are mutually exclusive — a
  cluster running both would have two independent arbiters for one key.

Removed by this work

LOCK_REQUEST/LOCK_GRANT nibbles, per-peer round tracking, deferral queues, (tsR, nodeName)
ordering, synthesized grants, withdraw-on-timeout, and agreedDown DOWN-exclusion.

Also fix here — three defects inherited with the substrate

A cross-model review of #2498 found these in code the redesign keeps, so they do not go away on
their own (§11 of the note):

• Transport replacement does not fence live authority (resources/Table.ts). Re-registering a
  transport — a component reload is enough — closes the current coordinator and installs an empty
  one without invalidating the handles the old one issued, so a successor can grant the same key
  with no lease time elapsed.
• The direct receive callback has no containment (resources/recordLockCoordinator.ts). A
  LockUnavailableError from coordinator construction escapes deliverLockControlEntry(); the
  source-subscription sink already contains this and the direct callback must too.
• The commit fence scans every write on every commit (resources/DatabaseTransaction.ts). The
  loop runs on ordinary transactions in core-only deployments that never register a transport, so a
  bulk transaction with 100,000 plain writes pays 100,000 property checks. The transaction must
  track whether it holds any lease-protected write and skip the pass when it does not — while still
  fencing a released-but-staged locked write, which is the case the loop exists for.

Blocked on

The measurement gate (HarperFast/harper-pro#824) and the epoch protocol (HarperFast/harper-pro#825).
The successor-freshness fence (#2542) is the sibling piece and lands with this one.

Refs #483

🤖 Filed by Claude Opus 5 on behalf of Kris.