Skip to content

Preserve origin record versions and transaction-log keys during replication - #812

Draft
kriszyp wants to merge 18 commits into
mainfrom
fix/replication-dual-clock-origin-log-key
Draft

Preserve origin record versions and transaction-log keys during replication#812
kriszyp wants to merge 18 commits into
mainfrom
fix/replication-dual-clock-origin-log-key

Conversation

@kriszyp

@kriszyp kriszyp commented Sep 3, 2026

Copy link
Copy Markdown
Member

Replication frames carry two clocks with different jobs:

  • version is the origin record's LWW version.
  • txnLogKey is the transaction's position in the origin node's log.

Previously the frame/log-position value was named localTime in parts of the implementation, inherited from the deprecated LMDB model. This PR makes txnLogKey the canonical internal name. localTime remains only on pre-existing compatibility payloads such as listener/history and end_txn; it is no longer the conceptual name of the clock.

What changes

  • The sender writes the origin txnLogKey in the existing frame header and carries the audit body's version separately.
  • The receiver validates the frame's txnLogKey before any durable state moves.
  • The RocksDB receiver applies the record at its origin version and commits it under the origin txnLogKey, then advances resume state and replication watermarks in the log-key domain.
  • A dual-clock primary record retains an audit-head reference, so out-of-order reconciliation and per-record history enter the transaction log at txnLogKey rather than the record version.
  • Full-copy rows recover that audit-head key when retained, so the copy/replay boundary remains in the log-key domain and replayed writes remain auditable on relays.
  • If retention already removed the audit head, a full-copy row falls back to its bounded record clock. This is safe because the copied row is an auditless snapshot from before copyStartTime; the receiver does not persist its per-row frame key as a replay cursor. Audit read errors still fail the copy rather than masquerading as retention.
  • Batch grouping, retention, residency reconstruction, and transaction completion use txnLogKey rather than the record version.
  • Invalid frame log keys hold and reconnect the replication leg before any record or cursor is committed.
  • The wire layout does not change; this clarifies and preserves the meaning of its existing fields.

This depends on Harper #2497, "Expose record version and txnLogKey as separate audit clocks", which supplies the core audit/read/apply contract and replay-safe log-key identity. The two PRs ship as one artifact.

Why preserve the origin log key

A source fill can have a backdated record version but a current transaction-log position. If a receiver keys that entry at its own receive time, every hop invents a new cursor identity. If it keys the entry at the record version, resume state moves into the LWW clock domain. Preserving the origin txnLogKey gives every relay the same durable write identity while preserving version for record ordering.

Verification

  • Three-node O to R to T integration: 2 passing. A backdated source fill retains the same version and txnLogKey on every hop, then survives an unclean T restart and resume through R unchanged.
  • Final post-rebase frame-key, copy-key, copy-metadata, and validation unit suites: 16 passing.
  • npm run build emits successfully but exits on the repository's existing analytics Numeric/bigint, Promise.withResolvers, WebSocket _socket, and error-property type failures; no new error is attributable to this change.

Scope and follow-ups

Deprecated LMDB keeps its existing transaction-version apply and grouping behavior; the internal audit reader still calls its log position txnLogKey. A pre-existing relayed-resume duplicate-entry issue remains outside this naming and clock-preservation change.

Refs #790

Review-Coverage: authored=codex; ran=cursor-composer,gemini; declined=claude,cursor-grok,domain; rounds=19 @ 0c5976d

Human-Review-Need: 4 @ 0c5976d

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request separates the handling of the origin's log key and the record's version within replication frames, preventing issues where the two clocks were conflated (such as during backdated cache fills). It also introduces validation for incoming frame log keys to avoid poisoning resume cursors with corrupt values, backed by new integration and unit tests. The review feedback suggests adding a safety check to ensure replicationSharedStatus is defined before accessing it to prevent potential runtime errors, and validating lastSequenceIdReceived with Number.isFinite() to prevent NaN propagation into the transaction's local time.

Comment thread replication/replicationConnection.ts Outdated
Comment thread replication/replicationConnection.ts
@kriszyp kriszyp added this to the v5.3 milestone Sep 3, 2026
@kriszyp kriszyp changed the title Apply the origin's record version and key it at the origin's log key on replication receive Preserve origin record versions and transaction-log keys during replication Sep 4, 2026
Kris Zyp and others added 14 commits September 4, 2026 11:22
…on receive

A replication frame carries two clocks and the receiver now adopts them
separately. The leading float64 is the origin's log key — the batch key its
entries occupy in the origin's own transaction log — and the forwarded audit
bytes carry the record's own version, which a cache fill sets from the source's
lastModified rather than from its commit.

The receiver used to skip that header and put the entry's version on the apply
event as the transaction timestamp, so a peer filed the origin's write in its
copy of the origin's log at the source's clock. Leaf state looked right; a third
node resuming through that peer read cursors in a domain the relay's log no
longer used, and the receive-path resume cursor and the received-version
watermark were maxima over record versions rather than log positions.

The apply event now carries `timestamp` from the frame's log key and `version`
from the entry, the sender frames transactions by the same log key, and both
receive-path cursors move with it. The frame key is validated before it can
reach setTimestamp, a durable cursor or a status field: a value that is not a
representable date — including the sentinel a body too short to hold its own
header yields — closes the connection and leaves every cursor where it was.

Refs #790

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WMMWWeFsqFuKduJHNMP6ZY
@kriszyp
kriszyp force-pushed the fix/replication-dual-clock-origin-log-key branch from 71809c8 to 21d2adf Compare September 4, 2026 17:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant