Fix skipped checkpoint flushes that can cause data loss in multi-journal DbLedgerStorage#4844
Open
void-ptr974 wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
In a multi-journal bookie,
CheckpointSourceListbuilds an aggregatecheckpoint by sampling journals one at a time. The sampling is not an
atomic snapshot, so a periodic
SyncThreadcheckpoint can interleavewith a DbLedgerStorage cache-pressure flush:
SyncThreadsamples journal 0 at position 1.at position 1, producing
[2, 1].SyncThreadthen samples journal 1 at position 2, producing[1, 2].The two checkpoints are incomparable because neither covers all state
represented by the other. However, the current lexicographic comparison
treats
[2, 1]as later than[1, 2]. DbLedgerStorage can therefore skipthe flush required for
[1, 2].SyncThreadcan still mark[1, 2]as complete, allowing the journalcheckpoint and cleanup position to advance while entries covered by the
second journal remain only in the write cache. If the bookie crashes
before a later flush, those entries may no longer be replayed during
recovery, resulting in potential data loss.
This requires multiple journals and overlapping checkpoint sampling.
Single-journal checkpoints are not affected.
Changes
compareToordering.Tests
mvn -pl bookkeeper-server -Dtest=CheckpointSourceListTest,DbLedgerStorageTest,BookieMultipleJournalsTest,SyncThreadTest,EntryMemTableTest -DfailIfNoTests=false testmvn -pl bookkeeper-server -DskipTests checkstyle:check spotless:check apache-rat:checkmvn -pl bookkeeper-server -DskipTests -Dspotbugs.onlyAnalyze=org.apache.bookkeeper.bookie.CheckpointSource,org.apache.bookkeeper.bookie.CheckpointSourceList,org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage spotbugs:check