feat(scheduler): parallel bounded ROWS-frame windows via a rank-derived halo - #2392
Draft
avantgardnerio wants to merge 7 commits into
Draft
feat(scheduler): parallel bounded ROWS-frame windows via a rank-derived halo#2392avantgardnerio wants to merge 7 commits into
avantgardnerio wants to merge 7 commits into
Conversation
…frames
Registers a third window sibling alongside ParallelWindowRule (bounded
RANGE) and PrefixWindowRule (UNBOUNDED PRECEDING). The pass returns the
plan unchanged; what it carries is the measured shape of the problem.
The module doc holds two captures from h2o window q6 --
avg(v2) OVER (ORDER BY id3 ROWS BETWEEN 100 PRECEDING AND CURRENT ROW),
the only h2o window query that clears every ParallelWindowRule gate but
the frame units:
- the plan `optimize` is handed, logged at debug the way PrefixWindowRule
does it, so the section can be refreshed when the chain shape moves;
- the two stage plans the cluster actually runs, read back from
/api/job/{job_id}/stages. Stage 1 is a single task: the SPM collapses
8 partitions to 1 and BWAG runs the window serially, 10.03s of a
19.33s wall at 1e7.
Also records why the RANGE halo doesn't extend -- it is a value delta,
a ROWS halo is a rank delta -- and where the rank bound comes from: a
descending merge over the range-shuffle sidecar indexes, accumulating
num_rows, comparison only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The walk that turns "100 preceding rows" into a value bound, written over plain arrays so the algorithm is fixed by examples before it meets RecordBatch and remote fetches. Moves to range_shuffle::index with the real implementation. What the examples hold in place: - messages are counted only when their successor's first_key proves them wholly below the cut; a straddler counts as zero, because over-counting stops the walk short of the rows asked for; - the merge runs in key order across files, not one file at a time -- draining a file first bounds correctly but hundreds of keys lower; - the fetch set is chosen on upper bounds, never on first_key: a message starting far below the bound can still reach up into the band; - a missing per-file key_max makes every final message a straddler, which on the worked example costs 680 keys of band and pulls every candidate into the fetch set. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Turns the frame's PRECEDING count into a value per consumer, so a bounded-ROWS window can run one task per range partition and still see the rows before its own cut. The walk is over the runtime stats the producer already reports: a file whose key_max sits strictly below a cut contributes all of its rows, so walking files descending by key_min and stopping when the total reaches n gives a key whose band up to the cut holds at least n predecessors. A file straddling the cut contributes zero -- only some unknown prefix of it lies below, and over-counting would stop the walk short. It fetches nothing. Nothing downstream learns about rank. The band reaches the router as `rank_cut_partitions`, a sibling of `cut_partitions` that tests each consumer's lower edge on its own rather than shifting the cut array, and reaches the read side as resolved RangeFilterExec bounds, which `attach_reader_bounds` then hands to the shuffle reader unchanged. Verified on h2o window q6 at 1e7 (id3 unique, so DataFusion is a real row-for-row oracle): OK, 15.13s against a 19.33s serial baseline. The band is far wider than n asks for -- a stage-0 file spans ~1.25M keys, so the walk can only stop at a file's key_min, and the window does 23.63M rows of work for 10M of output. The same walk over the sidecar index stops at a message instead of a file. Frames reaching past the current row stay on the serial path: the walk widens the lower edge only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The scheduler resolves the frame's PRECEDING count against per-file key
ranges, so the furthest it can stop is a whole file below the cut. The
index beside each source says the same thing per message, and the
consumer already downloads it, so the same walk runs again on the read
side and starts a batch below the cut instead of a file.
Only local sources are consulted: a remote index costs a round trip, and
both bounds are proven independently -- the scheduler's over every routed
file, this one over the subset it read for free -- so a subset makes the
saving smaller rather than the answer wrong. The bound it produces
applies to every source, so remote fetches narrow by it too.
h2o window q6 at 1e7, K=8, MPT=4, verified row-for-row against DataFusion:
serial coarse + fine
wall 19.33s 15.13s 12.08s
read rows 10.00M 23.71M 11.49M
window in 10.00M 23.63M 11.43M
Two consumers still read a file's worth of halo: a task slice's first
partition has no previous partition to take its cut from, and deriving it
is how the reader knows where its own range starts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The reader took a consumer's own cut from the previous partition's upper bound, which is the same value -- until `max_partitions_per_task` hands a task four of eight partitions. The slice's first partition then has no previous partition to read, so it skipped the walk and kept the scheduler's per-file bound: two of eight consumers reading a whole file of halo, which was all of the remaining over-read. Cut ranges now travel beside the widened ones, sliced with them, so where a partition sits in a task stops mattering. q6 at 1e7, verified: read amplification 1.15x -> 1.03x (10.33M rows for 10.00M of output), window elapsed_compute 12.54s -> 11.80s. Seven of eight consumers tighten; the eighth is global partition 0, which has nothing below it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is for
BoundedWindowAggExecrequires a single input partition, so a window with noPARTITION BY runs serially no matter how wide the cluster is.
ParallelWindowRulealready fixes that for bounded RANGE frames and
PrefixWindowRulefor UNBOUNDEDPRECEDING. This adds the third sibling,
HaloRowRule, for bounded ROWS frames:On h2o window q6 at 1e7 that window is 10.03s of a 19.33s wall clock, in a stage
of exactly one task: stage 0's writer declares
UnknownPartitioning, so the SPMabove the reader collapses 8 partitions to 1 and BWAG does the whole window alone.
Why the RANGE halo does not extend to ROWS
ParallelWindowRule's halo is a value delta end to end: files route by sketched[min, max]overlap againstcuts[k] +/- halo, andRangeFilterExeckeepscuts[k-1] - halo_lo <= v < cuts[k] + halo_hi. A ROWS halo is a rank delta, andnothing on that path expresses rank. Widening a cut by any value can land in a gap
holding no rows at all, so "100 preceding rows" is not reachable by a choice of
value width.
Where the rank bound comes from
The range shuffle sidecar index from #2364 carries one row per IPC message, with
that message's first key and its
num_rows. Merging those index rows descending byfirst key, across every candidate file, and accumulating
num_rowsuntil the totalreaches
ngives a key: reading from it up to the cut yields a superset of the truenpredecessors. Comparison only, so strings, timestamps, multi-column and DESCkeys all work. A message straddling the cut counts as zero, since over-counting
would stop the walk short of the rows asked for.
The walk runs at two granularities:
key_min/key_max/num_rowsthe runtime stats reports already carry. This fetches nothing, and itis what keeps the file list per consumer from growing with K.
RangeShuffleReaderExecalready downloads. This is byte reduction, notcorrectness.
Cuts stay approximate KLL value cuts. Halo exactness rests on the index walk's exact
num_rows, not on cut exactness.What does not change
OrderedRangeRepartitionExec,RangeFilterExecandPartitionedBoundedWindowAggExeclearn nothing about ROWS. The filter takes a resolved half-open band per partition,
which is what it already takes; the rank lives entirely in the scheduler-side walk
that produces those bands. Only the rule's frame-units gate mentions ROWS at all.
Results
h2o window q6 at 1e7 on a 2-executor cluster,
target_partitions=8,max_partitions_per_task=4, debug build.id3is unique at that scale, so a ROWSframe has one defined answer and DataFusion is a real row-for-row oracle; every
column below verified OK against it.
elapsed_computeThe serial column is one task doing the whole window. The coarse column is eight,
each also reading a file's worth of halo it does not need — read amplification
2.37x, and the window pays for every extra row. The fine walk takes that to 1.03x.
Per consumer, what the second walk buys, from the reader's own log:
22k-36k keys is about three 8192-row batches, against a theoretical best of the 100
rows the frame asks for.
Seven of the eight consumers tighten; the eighth is global partition 0, which has
nothing below it. That took shipping each consumer's own cut alongside the widened
one: the reader used to take it from the previous partition's upper bound, which is
the same value right up until
max_partitions_per_taskhands a task a slice of thepartitions and the previous one lands in a different task.
Tests
ballista/client/tests/halo_row.rsruns the standalone cluster against single-nodeDataFusion, at K=4 over 48 unique keys:
CURRENT ROW AND CURRENT ROW(no row fromoutside a task's own range, so it isolates the plan shape),
5 PRECEDING,20 PRECEDING(a halo reaching back past a whole partition), aFOLLOWINGframeasserting the gate leaves it on the serial path, and an
EXPLAIN ANALYZEcasepinning that the rewrite fired at all.
Both walks are unit-tested on their own:
rank_halo_walk_testsinruntime_stats.rsover per-file stats,
rank_walk_testsinrange_shuffle/index.rsover real indexbatches, both covering straddlers at the cut, key-order across files, running dry,
and a zero-row frame.
Scope
Frames reaching past the current row stay on the serial path: the walk widens a
consumer's lower edge only, and the mirror above the upper cut is not written yet.
DESC likewise, as with
ParallelWindowRule.Behaviour changes only for queries in the matched shape, and only when
ballista.planner.parallel_window.enabledis set, which defaults to false. The ruleshares that flag with
ParallelWindowRulerather than adding a second knob.