Skip to content

Expose exact transaction log positions - #1

Closed
ronag wants to merge 3 commits into
v11.1.1from
codex/wal-position-cursor
Closed

Expose exact transaction log positions#1
ronag wants to merge 3 commits into
v11.1.1from
codex/wal-position-cursor

Conversation

@ronag

@ronag ronag commented Jul 20, 2026

Copy link
Copy Markdown
Member

Summary

  • expose WAL file, logical-record offset/end, and optional XXH3 checksum on each BatchResult
  • keep checksum calculation disabled by default and opt in through TransactionLogIterator::ReadOptions
  • make complete zero-count/log-data records visible at the current WAL tail
  • retain an in-flight consuming record until its complete sequence range is published
  • document clean-tail continuation and preserve cursor metadata across moves

This intentionally targets the v11.1.1 maintenance branch created from the exact RocksDB version consumed by nxtedition/rocks-level. No upstream PR is opened.

Validation

  • make -j8 db_log_iter_test CC='ccache clang' CXX='ccache clang++'
  • ./db_log_iter_test (11/11)
  • focused log-only tail, initial-sequence, and unpublished-record tests (3/3)
  • git diff --check

@ronag
ronag requested a review from Copilot July 20, 2026 09:56
@ronag
ronag marked this pull request as ready for review July 20, 2026 09:58

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends RocksDB’s transaction log iterator to expose stable, exact WAL positioning metadata (WAL file number, logical record start/end offsets, and XXH3 checksum) via TransactionLogIterator::BatchResult, and updates iterator behavior to make log-data-only tail records visible and resumable while preserving cursor metadata across tailing.

Changes:

  • Add WAL position + checksum fields to BatchResult and propagate them from the WAL reader.
  • Adjust transaction log iteration so zero-count (e.g., log-data-only) records are visible at a clean WAL tail, and consuming records aren’t exposed until their full sequence range is published.
  • Add unit tests covering log-data-only tail behavior, WAL position/checksum propagation, and unpublished consuming-record visibility; add a new SyncPoint in the write path to support the test.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
include/rocksdb/transaction_log.h Extends public BatchResult with WAL location + checksum fields; clarifies iterator tailing semantics in comments.
db/transaction_log_impl.h Adds internal state to track per-record WAL metadata and buffer unpublished consuming records until published.
db/transaction_log_impl.cc Implements publication gating, captures WAL offsets/checksum from log::Reader, and propagates metadata to BatchResult.
db/db_log_iter_test.cc Adds tests for log-data-only tail visibility, WAL position/checksum, and unpublished consuming-record gating.
db/db_impl/db_impl_write.cc Adds a SyncPoint after WAL write to deterministically test unpublished consuming-record behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread db/transaction_log_impl.cc Outdated
Comment thread db/transaction_log_impl.cc Outdated
Comment thread db/db_log_iter_test.cc Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Comment thread include/rocksdb/transaction_log.h Outdated
Comment on lines +63 to +79
struct BatchResult {
SequenceNumber sequence = 0;

// The WAL file number containing this batch. Together with
// wal_record_offset, this identifies the batch while the WAL is retained.
uint64_t wal_file_number = 0;

// The physical byte offset of the first fragment of this logical WAL record.
uint64_t wal_record_offset = 0;

// The first physical byte offset after this logical WAL record.
uint64_t wal_record_end = 0;

// XXH3 checksum of the logical WAL record contents, or zero when
// ReadOptions::include_wal_record_checksum_ is false.
uint64_t wal_record_checksum = 0;

@ronag

ronag commented Jul 20, 2026

Copy link
Copy Markdown
Member Author

Closing this approach because it requires modifying RocksDB. I am replacing it with a rocks-level-only implementation against the unchanged v11.1.1 dependency.

@ronag ronag closed this Jul 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants