[DNM] buffer pool e2e stack: chunk seam, upsert + compute adoption, pager deletion#37805
Draft
DAlperin wants to merge 10 commits into
Draft
[DNM] buffer pool e2e stack: chunk seam, upsert + compute adoption, pager deletion#37805DAlperin wants to merge 10 commits into
DAlperin wants to merge 10 commits into
Conversation
DAlperin
force-pushed
the
dov/chunk-pool-e2e
branch
2 times, most recently
from
July 22, 2026 16:43
dab3445 to
bba0eb0
Compare
UnloadChunk is the bulk-read capability for Chunk families: look up a sorted set of keys in a chunk, copying the matching updates out into caller-owned staging. Extraction is finished with a chunk's body when the call returns, so a spilled body is read for the scope of one call and no reference into pool memory exists outside it -- the contract that lets a buffer pool evict with no reader accounting. The trait has no key/val/time/diff opinions (Staging and Probes are family-chosen), and the one comparison the batch driver needs is delegated to the chunk via locate, answerable from resident metadata. Chunk boundaries are carried as the consume-index protocol: a chunk consumes every probe strictly below its last key and extracts-but-does-not-consume one equal to it, re-offered to the next chunk as a legal straddle. Ported from the differential fork (TimelyDataflow/differential-dataflow PR 782) so consumers here depend on a local definition rather than a git pin. One delta from the upstream shape: the batch driver is the UnloadBatch extension trait rather than inherent methods on ChunkBatch (not available to this crate); names and signatures match, so call sites survive a future switch to upstream re-exports unchanged. If the PR lands, this module retires in favor of re-exports. Contract test: a miniature row family driven through every (chunk cut x contiguous probe range) placement with a multi-chunk-spanning key, against the reference filter.
…nkSpine with the UnloadChunk drain
…tion buffers on the pool
DAlperin
force-pushed
the
dov/chunk-pool-e2e
branch
from
July 22, 2026 20:32
bba0eb0 to
a8b6d03
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Do not merge. CI-build vehicle for staging testing of the full buffer-managed-state stack (see the design doc, doc/developer/design/20260610_buffer_managed_state.md, and the sub-block feasibility note alongside it).
Contents, bottom to top: the mz_ore::pool commits from #37718 plus pool config (#37719), the differential Chunk/ColumnChunk seam, the upsert-v2 stash and feedback arrangement on ChunkBatcher/ChunkSpine, compute arrange adoption behind dyncfgs, deletion of mz_ore::pager, and benchmark/seam follow-ups (pool_steady_state, random_access_swap, ExtentCodec::decode_range).
Everything is gated off by default. To exercise it: enable_upsert_v2 + enable_upsert_paged_spill for the storage leg, enable_column_paged_batcher + enable_column_paged_batcher_spill (replica-scoped) for compute, with pool tuning under the column_paged_batcher_* dyncfgs.
The real PRs land through the #37718 stack; this branch exists so CI produces images for staging.