Skip to content

Fix skipped checkpoint flushes that can cause data loss in multi-journal DbLedgerStorage#4844

Open
void-ptr974 wants to merge 1 commit into
apache:masterfrom
void-ptr974:fix/dbledger-composite-checkpoint-coverage
Open

Fix skipped checkpoint flushes that can cause data loss in multi-journal DbLedgerStorage#4844
void-ptr974 wants to merge 1 commit into
apache:masterfrom
void-ptr974:fix/dbledger-composite-checkpoint-coverage

Conversation

@void-ptr974

@void-ptr974 void-ptr974 commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Motivation

In a multi-journal bookie, CheckpointSourceList builds an aggregate
checkpoint by sampling journals one at a time. The sampling is not an
atomic snapshot, so a periodic SyncThread checkpoint can interleave
with a DbLedgerStorage cache-pressure flush:

  1. SyncThread samples journal 0 at position 1.
  2. The cache-pressure flush samples journal 0 at position 2 and journal 1
    at position 1, producing [2, 1].
  3. SyncThread then samples journal 1 at position 2, producing [1, 2].

The two checkpoints are incomparable because neither covers all state
represented by the other. However, the current lexicographic comparison
treats [2, 1] as later than [1, 2]. DbLedgerStorage can therefore skip
the flush required for [1, 2].

SyncThread can still mark [1, 2] as complete, allowing the journal
checkpoint and cleanup position to advance while entries covered by the
second journal remain only in the write cache. If the bookie crashes
before a later flush, those entries may no longer be replayed during
recovery, resulting in potential data loss.

This requires multiple journals and overlapping checkpoint sampling.
Single-journal checkpoints are not affected.

Changes

  • Add a strict coverage comparison for checkpoints while preserving the existing total compareTo ordering.
  • Compare aggregate checkpoints component by component.
  • Use coverage comparison for the DbLedgerStorage checkpoint skip decision.
  • Add unit coverage for aggregate checkpoint ordering and a DbLedgerStorage regression test.

Tests

  • mvn -pl bookkeeper-server -Dtest=CheckpointSourceListTest,DbLedgerStorageTest,BookieMultipleJournalsTest,SyncThreadTest,EntryMemTableTest -DfailIfNoTests=false test
  • mvn -pl bookkeeper-server -DskipTests checkstyle:check spotless:check apache-rat:check
  • mvn -pl bookkeeper-server -DskipTests -Dspotbugs.onlyAnalyze=org.apache.bookkeeper.bookie.CheckpointSource,org.apache.bookkeeper.bookie.CheckpointSourceList,org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage spotbugs:check

@void-ptr974 void-ptr974 changed the title Fix DbLedgerStorage checkpoint comparison with multiple journals Prevent skipped checkpoint flushes in multi-journal DbLedgerStorage Jul 19, 2026
@void-ptr974 void-ptr974 changed the title Prevent skipped checkpoint flushes in multi-journal DbLedgerStorage Fix skipped checkpoint flushes that can cause data loss in multi-journal DbLedgerStorage Jul 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant