diff --git a/content/storage-engine/projects/join-algorithms/cost-model.md b/content/storage-engine/projects/join-algorithms/cost-model.md new file mode 100644 index 0000000..afb6b2b --- /dev/null +++ b/content/storage-engine/projects/join-algorithms/cost-model.md @@ -0,0 +1,150 @@ +--- +title: Cost Model +sidebar_position: 3 +--- + +# The cost model + +Costs are in estimated milliseconds. Absolute values matter less than relative +ones - the planner only ever compares them. + +## Arithmetic rules + +Enforced throughout `cost.rs`, and worth preserving: + +- cardinalities are `u64` and combine with saturating operations; +- anything that could overflow is computed in `f64` and converted back with + `as u64`, whose float-to-integer casts saturate rather than wrap; +- costs are `f64`; +- **`usize` is not used at all** - it is 32 bits on some targets, and the + previous implementation's `tuple_count as usize * bytes_per_tuple` panicked + in debug builds and wrapped in release ones. + +`test_join_cost_model.rs` feeds `u64::MAX / 2` rows to every algorithm and +requires a finite, non-NaN cost. + +## Coefficients + +| Coefficient | Default (ms) | What it prices | +|---|---|---| +| `seq_page` | 0.010 | one sequential page read | +| `random_page` | 0.040 | one random page read | +| `cpu_tuple` | 0.000_10 | emitting or building one row | +| `cpu_key` | 0.000_05 | extracting or comparing one key | +| `cpu_hash` | 0.000_04 | hashing one key | +| `cpu_compare` | 0.000_02 | one value comparison | + +The ratios are PostgreSQL-shaped: random access is roughly four times a +sequential page, and CPU work is orders of magnitude below I/O. + +## Cardinality + +**Equijoin** - System-R containment. Each side's matchable rows spread over its +distinct values, paired through the *larger* of the two counts: + +``` +|L ⋈ R| = |L|·(1 − nullfrac_L) · |R|·(1 − nullfrac_R) / max(ndv_L, ndv_R) +``` + +Raising the smaller side's distinct count below that threshold correctly +changes nothing - that is the containment assumption, not a bug. + +**Composite keys** multiply the per-column counts, capped at the row count: a +table cannot hold more distinct combinations than it has rows. Independence +between key columns is assumed. + +**Semi** - `min(ndv_L, ndv_R) / ndv_L`, which is at most one, so `|L ⋉ R| ≤ |L|` +is a property of the formula rather than a clamp added afterwards. **Anti** is +`|L| − |L ⋉ R|`. **Outer** adds the unmatched rows of whichever sides are +preserved, derived from the same quantity so the family stays consistent. + +**Range predicates** convolve the two columns' 64-bucket equi-depth histograms - +4096 bucket pairs, each contributing fully, not at all, or half. Without +histograms the default is 1/3; for a conjunct nothing is known about, 0.25. + +## Per-algorithm cost + +With `B` = memory in pages, `pages` and `rows` per side: + +**Simple nested loop** - the inner relation is re-read per outer row: +`pages_L + rows_L · pages_R` I/O, `rows_L · rows_R` comparisons. + +**Block nested loop** - one inner pass per block: +`pages_L + ceil(rows_L / block) · pages_R`. + +**Index nested loop** - one descent and one fetch per outer row, plus the rows +each probe returns: `pages_L + rows_L · (1 + matches) · random_page`, with +`matches = rows_R / ndv_R`. + +**Sort merge** - per side, `passes = 0` if it fits, else +`1 + ceil(log_{B−1}(pages / B))`; each pass reads and writes the relation. +Zero when it fits is the important part: no run is written, so there is nothing +to merge. + +**Hash** - one pass over each input, plus two more over the spilled fraction +`1 − min(B / pages_build, 1)`. CPU charges the build side more than the probe +side: + +``` +cpu = rows_build · (cpu_hash + cpu_key + cpu_tuple) + + rows_probe · (cpu_hash + cpu_key) +``` + +Charging them alike would make the formula symmetric in its inputs, and the +planner could then not see why building from the smaller side is worth +anything. See "Calibration" below - this came from a measurement. + +**Symmetric hash** - one pass over each input, both resident, hashing both +sides. The planner does not offer it when the inputs will not fit, so there is +no spilling term to model. + +**Adaptive** - costed as what it will actually run: a block nested loop with no +equality; otherwise a hash join *with the sides ordered so the smaller builds*, +since that is what it does. Then a 5% premium when statistics are measured and +a 15% discount when they are guesses - adaptivity buys nothing if the +prediction is already right, and is worth paying for when it is not. + +## Calibration + +```sh +cargo run --release --bin benchmark_joins +cargo run --release --bin benchmark_joins -- --rows 100000 --json +``` + +Four scenarios: a resident selective join, a resident join with heavy +duplicates, the same under a 1 MiB budget, and a small outer relation against a +large inner one. Each algorithm gets a discarded warm-up run and five measured +ones; median and p95 are reported, along with the algorithm the planner chose. + +**The check that matters is whether the planner's choice matches the fastest +measured algorithm.** If it does not, the model has a structural gap, and +adjusting coefficients will paper over it rather than fix it. + +That is how the build-versus-probe asymmetry above was found. On 200 × 20 000 +rows the adaptive operator measured ~3.8ms against the hash join's ~6.4ms - +because it reverses and builds from the 200-row side - but the planner chose +the hash join. The formula summed `left_rows + right_rows`, so swapping the +sides changed nothing. Charging build rows more than probe rows made the model +agree with the measurement, in both directions: it still prefers the plain hash +join on symmetric inputs, where that is genuinely faster. + +To adjust the coefficients, scale `seq_page` so a scan-dominated scenario's +predicted cost lands near its measured milliseconds, then scale the CPU +coefficients against a duplicate-heavy scenario, where output volume dominates. +Keep the ratios between the CPU coefficients unless you have a reason to move +them. + +## What remains an estimate, and which way it is wrong + +| Quantity | Source | Bias | +|---|---|---| +| Distinct values | HyperLogLog, p=12 | ±1.6% typical; only for analyzed columns. Unanalyzed falls back to `n^0.75`, which **understates** a unique column and makes index joins look worse than they are | +| Conjunct interaction | assumed independent | **Under**-estimates output for correlated predicates | +| Join key skew | histograms are marginal | No model of one key dominating; the hash join reports it at run time instead | +| Buffer cache | assumed always cold | **Over**-estimates I/O uniformly, so it does not distort the comparison between plans | +| CPU coefficients | machine-dependent | Calibrated per machine, or left at defaults | +| Intermediate results in a multi-relation plan | every row assumed to carry a distinct key | **Over**-estimates the output of joins above the first | + +None of these is hidden from the user: EXPLAIN prints `stats=analyzed`, +`header-only` or `defaults`, so an estimate built on measurement and one built +on a guess do not look alike. diff --git a/content/storage-engine/projects/join-algorithms/design-rationale.md b/content/storage-engine/projects/join-algorithms/design-rationale.md new file mode 100644 index 0000000..def7032 --- /dev/null +++ b/content/storage-engine/projects/join-algorithms/design-rationale.md @@ -0,0 +1,489 @@ +--- +title: Design Rationale +sidebar_position: 1 +--- + +# Design rationale + +Why each significant decision was taken. Code comments say *what* something +does; this says *why*, and what the alternative would have cost. + +The subsystem was rewritten rather than repaired. The previous version decoded +rows assuming `Column { data_type: String }` with fixed 4-byte INT and 10-byte +space-padded TEXT and no NULLs - a storage layer the engine no longer has. It +also carried a set of defects that were not bugs in an implementation so much +as consequences of its shape, and most of the decisions below exist to make +those defects *unrepresentable* rather than merely fixed. + +--- + +## Three constraints that were not chosen + +Everything else is downstream of these. + +**Rows cannot be byte-concatenated.** `PhysicalSchema::from_logical` regroups +all fixed-length columns before all variable-length ones, across the whole +schema, and `RowLayout::compute` aligns them. `left_bytes ++ right_bytes` is +not a valid row. Every output row is therefore decode → concatenate values → +re-encode. This is not an optimisation we declined to make; it is forced. + +**`deserialize_nullable_row` needs an exactly-sized slice.** The last +variable-length payload's length is derived from `row_bytes.len()`, so a row +read out of an over-long buffer silently mis-decodes its final column. This is +why every spill file is length-framed, and why nothing may hand the decoder a +padded buffer. + +**There is no `DataValue::Null`.** NULL is `Option::None` plus a +per-row bitmap. The previous implementation's worst defect - NULL keys encoded +as `0` or ten spaces when spilling, so the same query returned different +answers depending on which algorithm ran - came from materialising a NULL as a +sentinel value. The design had to make that impossible, not merely avoided. + +--- + +## The join key + +### One order-preserving byte encoding + +`JoinKey(Box<[u8]>)` derives `Hash`, `Eq`, `PartialOrd` and `Ord` from the same +byte slice. + +The alternative was what the previous implementation had: three unrelated hash +functions, three unrelated serialisers, and a comparator that returned +`Ordering::Equal` whenever two values were incomparable. That last one is worse +than it sounds - it violates the total-order contract `BinaryHeap` and +`sort_by` rely on, so the merge phase of the sort-merge join was reading runs +that were not actually sorted. + +Deriving all three traits from one encoding makes "hash agrees with equality" +and "the ordering is total" true by construction rather than by review. + +### The encoding follows `Comparable`, not `PartialEq` + +The engine's own two APIs disagree, and the key had to pick one: + +| Case | `DataValue`'s derived `PartialEq` | `Comparable::compare` | +|---|---|---| +| `Real(+0.0)` vs `Real(-0.0)` | different (bit patterns) | equal | +| `Char("ab")` vs `Char("ab ")` | different | equal (trailing whitespace stripped) | +| `Numeric(1.00)` vs `Numeric(1.0)` | different | equal (scales normalised) | + +`compare` gives the SQL answer, so the encoder normalises `-0.0` to `+0.0`, +canonicalises every NaN, trims CHAR with `trim_end` (all trailing Unicode +whitespace, not only spaces - matching `str::trim_end`), and normalises NUMERIC +scale. `to_bytes()` does none of that, so hashing it directly would have made +values the engine calls equal fail to join. + +### NULL is unrepresentable, not handled + +`KeySpec::left_key` returns `Result, _>`; `Ok(None)` means some +component was NULL. There is no byte sequence a NULL encodes to, so two NULLs +cannot collide however many times a row is spilled and re-read. The `Option` +also forces every operator to decide, at compile time, what a NULL-keyed row +does - skip the hash table, route to the unmatched stream, sit outside the +merge. + +### Type mismatch is a plan-time error + +Only one coercion is automatic: integer widening, which mirrors `Comparable`'s +own SMALLINT ↔ INT ↔ BIGINT promotion and is lossless. + +Everything else - INT vs REAL, CHAR vs VARCHAR, NUMERIC of differing scale - is +refused before a row is read. An implicit cast would route through a string +literal: `cast(Real(0.1), Int)` silently truncates, and `cast(Varchar("abc"), +Int)` fails per row, turning a schema error into either a wrong answer or a +failure ten thousand rows in. Implicit lossy casts convert "your query is +wrong" into "your answer is wrong". The escape hatch is explicit and shows up +in EXPLAIN. + +Cross-scale NUMERIC is refused for a second reason: `Comparable::compare` +rescales with `10_i128.pow(...)` and an unchecked multiply, which panics in +debug builds for scale gaps beyond 38. + +--- + +## Predicates + +### The representation is the engine's own + +`executor::selection`'s `Predicate`, `Expr`, `ComparisonOp` and `TriValue` are +reused unchanged, as is `apply_and` / `apply_or`. Two upstream helpers, +`compute_arithmetic` and `constant_to_data_value`, were made public rather than +copied, so arithmetic and literal handling inside a join condition cannot drift +from the same operations anywhere else in the engine. + +What is *not* reused is `SelectionExecutor` for the join condition itself. It +evaluates one row against one table, so using it would mean materialising a +concatenated row for every candidate pair - O(|L| × |R|) serialisations even +under a hash join, which is exactly what the decode/re-encode constraint makes +expensive. It also resolves bare column names against a single schema, which is +how the previous implementation's self-joins ended up comparing a column to +itself. + +`SelectionExecutor` *is* used where it fits: single-relation filters pushed into +a scan, evaluated against raw bytes. + +### Splitting conditions is what makes multi-predicate joins correct + +The previous implementation took its join key from `conditions[0]`. So: + +- `A.x < B.y AND A.k = B.k` keyed on `x`/`y` and silently returned almost + nothing; +- a pure non-equi join fed to hash or sort-merge returned roughly nothing, and + the planner *routed* non-equi joins to sort-merge; +- an equality written `right = left` made sort-merge return zero rows, because + it never checked which relation a column belonged to. + +`split_conjuncts` flattens the `AND` tree, promotes *every* cross-relation +equality into the key with its orientation normalised once, and keeps +everything else as a residual that every key-based operator also applies. No +executor ever sees an unnormalised condition, and no executor has to guess +which conjunct is "the" join key. Zero equalities means hash, sort-merge and +index joins are simply *not applicable* rather than fed something arbitrary. + +Equality under `NOT`, inside an `OR`, or over arithmetic is deliberately not +hoisted: those are not equijoins, and hoisting them would change the result. + +### Resolution is qualifier-aware + +An unqualified column present on both sides is an `AmbiguousColumn` error, not +a silent pick. Two relations sharing an alias is refused outright, so a +self-join must alias its sides apart. This is the direct fix for the old hash +join, which resolved both build and probe keys to the same column whenever +`build_table == probe_table` and silently missed every match. + +### Pushdown legality is part of the split + +A conjunct touching only the row-preserving side of an outer join cannot be +pushed into that side's scan - it would drop rows the join must emit +NULL-extended. `split_conjuncts` therefore takes the join type: under LEFT +OUTER a left-only conjunct is emitted into the *residual*, already rewritten +into the concatenated index space, because the residual and a local filter use +different index spaces and a later pass could not have moved it. + +--- + +## Capability, enforced by types + +The compatibility matrix is one `const ALGORITHMS` array. Every executor's +constructor requires a `ValidatedJoinSpec`, whose fields are private and which +only `AlgorithmSpec::validate` produces. A hash join for CROSS, or an index +join without an index, is unconstructable. + +The previous implementation accepted SEMI, ANTI and NATURAL into its hash, +symmetric-hash, sort-merge and direct executors and silently computed an INNER +join. Here that cannot recur twice over: the token cannot be obtained, and SEMI +and ANTI produce a left-only output schema, so an executor emitting a +concatenated row would fail on the first row. + +A CROSS join carrying a condition is refused rather than having it dropped or +applied - it is a mis-stated INNER join, and guessing which was meant is worse +than asking. + +--- + +## Execution + +### Operators are `Iterator`s, and `close` is `Drop` + +`RowStream` is `Iterator, JoinError>>` plus a schema. +`open` is `PhysicalPlan::execute`; there is no `close`. + +An explicit `close()` is strictly worse in Rust: it is skippable, and an early +`?` in the consumer, a `LIMIT`, or a panic would leak. `Drop` runs during +unwind, so a panic mid-join removes the spill directory. This is what makes the +old `cleanup()` - which deleted *every* file in the shared temp directory, +including other operators' - impossible to reintroduce. + +### Rows travel as bytes + +Spilling requires re-serialisation regardless, so bytes make a spill a memcpy. +`Vec>` with heap `String`s is several times the footprint of a +packed row, which would make the memory accounting fiction. Decoding happens +once per row, never per comparison. + +`RowCodec` exists because `deserialize_nullable_row` rebuilds `PhysicalSchema` +and `RowLayout` on *every* call - five allocations per row, on the hot path of +every operator. It precomputes them per schema and adds single-column +extraction, which upstream keeps private. The cost of not calling upstream +directly is drift, so `tests/test_join_row_codec.rs` asserts byte-identity with +`serialize_nullable_typed_row` and value-identity with +`deserialize_nullable_row` over randomly generated schemas. + +### One definition of "these rows join" + +`MatchEvaluator` is shared by every operator, so a nested loop, a hash join and +a sort-merge cannot disagree about what a match is. That shared definition is +what makes the cross-algorithm differential test meaningful. + +--- + +## Memory and spilling + +### The budget is a budget, not a limit + +Rust exposes no allocator introspection, so `MemoryAccountant` is a deliberate +over-estimate of what an operator holds - row bytes plus a per-row constant, +plus a per-entry constant for hash tables. Its job is to trigger a strategy +change well before the real allocator is stressed. Treating it as a hard limit +would be wrong; treating it as a signal is exactly right. + +Accountants form a tree so a resident partition is charged against the +operator's budget. There is deliberately no global accountant: a global would +couple every operator, and every test, to every other. + +The floor is 4 KiB, not something larger. It only has to exceed one row plus +its hash entry - and keeping it small is what makes Grace and hybrid reachable +from a fast unit test. The previous implementation's Grace and hybrid paths ran +**zero** times under test, because its fixtures were two pages against a +ten-page budget. + +### Spill files are framed and scoped + +Length-framed records, not heap files. A heap file would add an 8 KiB header +page and a free-space-map fork *per partition* - 128 files before a single row +for a 64-way Grace join - and charge an FSM search, a page read, a page write +and a header rewrite per row, to buy random access and space reuse a spill file +never needs. Exact length framing is also the only framing that satisfies the +exact-slice constraint. + +Each operator gets `{root}/join-{pid}-{epoch}-{counter}/`, with a process-global +counter, so the two sides of a self-join cannot collide - the old fixed names +(`sort_run_{table}_{n}.tmp`) meant the second `sort_relation` call truncated the +first's files. `SpillScope::drop` removes only its own directory and never +enumerates the root. A `RunHandle` holds an `Arc` to the scope, so a directory +outlives any reader still in flight. + +Run files carry a magic number and the schema fingerprint. Reading a run under a +different schema is a hard error - which is precisely the failure the old +colliding run files hid. + +`Drop` covers normal exits and panics. `sweep_orphans` covers SIGKILL: it +removes only directories whose names it recognises, whose owning process is +gone, and which are older than a threshold. + +--- + +## Algorithms + +### Hash is one operator, not three + +In-memory, hybrid and Grace are runtime consequences of what fits, not separate +algorithms for the planner to choose between. On the first over-budget insert +the rows *already built* are repartitioned in place and partition 0 stays +resident - the build input is never re-read, which is what makes the transition +single-pass. Probe rows for the resident partition are joined as they arrive +rather than written out and read back; that is the whole point of the hybrid +form. + +A partition that still does not fit is repartitioned with the recursion depth +mixed into the hash, so rows actually redistribute - hashing the same way again +would put every row straight back into one partition. + +Past depth 3 the partition is loaded anyway and counted in +`oversized_partitions`. A partition that will not shrink is one where a single +key dominates it, and no amount of hashing separates rows that share a key. +Saying so is more useful than pretending otherwise. + +### Sort-merge bounds its duplicate group + +Every left row of a key group must see every right row of it, so the right +group is buffered and replayed. The previous implementation held it in a plain +`Vec`, so one hot key larger than memory took the process down. Here it is a +buffer that moves to disk when the budget is spent. Only one side is ever +buffered - left rows stream through the group one at a time. + +NULL-keyed rows never enter the merge. They cannot match, so ordering them +against real keys would be meaningless; the sort sets them aside and they are +emitted as unmatched if the join type wants them. + +### Symmetric hash says when it cannot continue + +It has no build phase to partition, so it cannot spill. When the budget runs +out it returns `JoinError::OutOfMemory` with a suggestion, rather than growing +without bound. The planner does not offer it when the inputs will not fit, so +the cost model never has to describe a case that cannot happen. + +### Adaptive measures rather than predicts + +Both inputs are read in alternation; whichever ends first is *provably* the +smaller and becomes the build side. This holds with no ANALYZE, no statistics +and wrong estimates, because it is a measurement. + +Reversal changes nothing observable. Rather than swapping rows and re-encoding +them, the hash join knows which relation it is probing and fills each half of +an output row from whichever input the *declared* schema says belongs there. +Key extraction and residual evaluation are side-aware for the same reason - +probing a reversed join with the left key, or evaluating a residual with its +arguments swapped, would be a silent wrong answer. + +SEMI and ANTI are never reversed: they are defined in terms of left rows. + +--- + +## Statistics + +**Cardinality comes from the heap header.** `HeaderMetadata::total_tuples` is +maintained on insert and delete, so it is exact and free. The engine's +`collect_table_statistics` counts slot entries without skipping dead ones, so +it over-reports after a DELETE until compaction; it is used only for tuple +widths, and only behind the cache. `test_join_statistics.rs` records that +discrepancy as a characterisation test, so a future upstream fix is noticed. + +**Column values are measured in join key encoding.** A distinct-value count is +only useful for join selectivity if it counts the equivalence classes the join +matches on. Encoding first means two CHARs differing in trailing spaces count +once - exactly as the join treats them - so estimates and execution cannot +drift apart. This is worth more than the choice of sketch. + +**HyperLogLog rather than an exact set.** 4 KiB per column regardless of +cardinality, and mergeable. The HLL++ small-range correction is not optional: +without it a fifty-row fixture estimates wildly, and a cost model that cannot be +checked on small fixtures cannot be checked at all. + +**Sampling is seeded.** Histogram boundaries - and therefore every estimate and +every EXPLAIN - are reproducible between runs. That reproducibility is itself +tested. + +**Stale statistics are ignored, not partially trusted.** A sidecar whose +validity stamp no longer matches the table drops the plan to `header-only`, +where cardinality is still exact and per-column values are admitted guesses. +There is no auto-analyze and no background refresh: planning from numbers that +no longer describe the data is worse than planning from admitted ignorance. + +The stamp is read *after* opening the heap, because `HeapManager::open` +synchronises the header and may rewrite the file. Taking it beforehand made +freshly written statistics look stale the moment they were saved - a latent +flakiness bug that only surfaced once enough tests ran to cross mtime's +one-second granularity. + +--- + +## Cost + +**Arithmetic rules are enforced module-wide.** Cardinalities are `u64` with +saturating operations; anything that could overflow goes through `f64`, whose +casts saturate; costs are `f64`; `usize` is not used at all. The previous +implementation's `tuple_count as usize * bytes_per_tuple` panicked in debug +builds and wrapped in release ones. + +**Semi selectivity is bounded by construction.** `min(ndv) / ndv_left` is at +most one, so `|L ⋉ R| ≤ |L|` is a property of the formula rather than a clamp +bolted on afterwards. The old formula scaled by both row counts and +over-estimated a ten-row semi join a hundredfold. + +**Sort passes are pass counts.** Zero when the input fits - no run is written, +so there is nothing to merge - otherwise `1 + ceil(log_fanin(runs))`. The old +model set `io_passes: rows as u32`, which is a row count wearing a pass count's +name. + +**Estimate confidence is a cost input.** Adaptive carries a 5% premium when +statistics are measured and a 15% discount when they are guesses: adaptivity +buys nothing if the prediction is already right, and is worth paying for when +it is not. This is the honest way to spend a self-correcting operator. + +**Building costs more than probing.** Charging hash build and probe rows alike +makes the formula symmetric in its inputs, and the planner then cannot see why +building from the smaller side is worth anything. Benchmarking a 200-row +relation against a 20 000-row one surfaced this: the adaptive operator was +measurably a third faster and the model could not tell. + +--- + +## Ordering and multi-relation execution + +**A predicate is applied at the lowest node that can evaluate it.** Every +conjunct carries the set of relations it mentions, and waits until a node's +subtree covers that set. A three-relation conjunct therefore joins in when the +third relation does, rather than being forced onto a two-relation edge where a +third of it cannot be resolved. + +**Conditions are rewritten onto synthetic positional names.** A node's inputs +are subtrees, not relations, so its columns are named `l0`, `r3` and the +conjuncts being applied are rewritten to match. That reuses the entire +two-relation apparatus - resolution, conjunct splitting, key extraction, +three-valued evaluation - instead of growing a second one. The *output* schema +keeps the real qualified names, so a column is still called `orders.id` however +many joins it passes through. + +**Output column order does not depend on the plan.** The optimiser may join the +relations in any order, but a caller who asked for `a JOIN b` gets a's columns +first: the root is permuted back into the declared order if the chosen plan +produced another. This is the same rule the adaptive operator follows for +build-side reversal, for the same reason - the shape of an answer should not +reveal how it was computed. + +**Only connected subsets are searched.** Enumerating disconnected ones means +costing plans that form a Cartesian product in the middle of a query that never +asked for one - which the previous implementation did, because it stored its +join conditions and never consulted them. + +Cross products remain reachable where the query genuinely has no path: the +graph is split into components, each optimised alone, and they are combined +smallest-first *outside* the search, where the cost is explicit and the node is +labelled. + +**Every node is costed across all applicable algorithms.** The old DP hardcoded +block nested loop for every node, which made its cost-based ordering a +cost-based ordering of one plan shape. + +**Bushy plans are considered.** With spilling hash joins, building two small +intermediates can beat dragging one large intermediate through every step, and +this engine has no parallelism for a left-deep pipeline to exploit. + +--- + +## Deliberate limitations + +Recorded here rather than left to be discovered. + +**Outer joins are reordering barriers.** Reordering across one changes the +answer, and doing it correctly needs conflict and eligibility sets that are easy +to get subtly wrong. Only inner-join blocks are reordered. + +**Interesting orders are not tracked.** A memo entry keeps the cheapest plan for +a subset, not the cheapest per sort order, so a chain of sort-merge joins +re-sorts rather than reusing an existing order. + +**Multi-relation execution materialises its right inputs.** The left spine of a +plan streams, so a left-deep join holds one intermediate at a time and that one +spills if it outgrows the budget. A bushy plan holds one per level. Nothing is +pipelined through a join node's right side, because the operators below need a +re-openable input. + +**LATERAL is absent** because nothing in the engine evaluates a correlated +subquery. **NATURAL** is a resolution-time rewrite, with FULL OUTER's shared +column correctly `COALESCE(l.c, r.c)`. + +**CHAR compared to a text literal is an error**, because `Constant::Text` +becomes VARCHAR and the two are not comparable. That limitation is the engine's, +shared with single-relation selection, and is recorded as a characterisation +test. + +--- + +## Upstream defects found while building this + +Worked around here; each deserves an upstream fix. + +1. **`Comparable::compare` can panic on NUMERIC** - `10_i128.pow((a.scale − + b.scale) as u32)` and `b.unscaled * factor` are unchecked. Worked around by + refusing cross-scale NUMERIC keys at plan time. +2. **`serialize_nullable_typed_row` corrupts rows past 65535 bytes** - the + var-len offset is cast with `as u16` and wraps; the result does not survive + a round trip and currently panics on a reversed slice range. `RowCodec` + refuses instead. Latent for the heap, since a row that large cannot fit an + 8 KiB page, but reachable through spill files. +3. **VARCHAR length is enforced on decode but not on encode**, so upstream can + write a row it cannot read back. `RowCodec` refuses at encode time. +4. **`collect_table_statistics` over-counts after DELETE** - it does not skip + dead slots. +5. **`DataValue`'s derived `PartialEq` disagrees with `Comparable::compare`** on + signed zero. Any code mixing the two is latently wrong. +6. **`deserialize_nullable_row` rebuilds its layout per call** - five + allocations per row. +7. **`HeapScanIterator` reopens the file on every page transition.** +8. **`load_catalog()` is infallible** - a corrupt catalog silently yields an + empty one, so a real table would report "not found". The join subsystem + never calls it. +9. **`HeapManager::create` deletes any existing file at its path.** diff --git a/content/storage-engine/projects/join-algorithms/developer-guide.md b/content/storage-engine/projects/join-algorithms/developer-guide.md new file mode 100644 index 0000000..eb8c798 --- /dev/null +++ b/content/storage-engine/projects/join-algorithms/developer-guide.md @@ -0,0 +1,214 @@ +--- +title: Developer Guide +sidebar_position: 2 +--- + +# Developer guide + +## Module map + +``` +src/backend/join/ + mod.rs public surface; the `unwrap`/`expect`/`panic` denials + error.rs JoinError - every fallible path returns it + schema.rs RelationSchema, OutputSchema, the type fingerprint + row.rs RowCodec (layout precomputed per schema), RowBuilder + key.rs KeyClass, JoinKey, the order-preserving encoding + predicate.rs side-aware resolution, split_conjuncts, 3VL evaluation + algorithm.rs JoinType, the capability matrix, ValidatedJoinSpec + config.rs JoinConfig - work memory, spill root + memory.rs MemoryAccountant + spill.rs SpillScope (RAII), run files, spillable row buffers + sort.rs external sort: run generation, k-way merge + cost.rs cardinality estimation and the cost model + stats/ HyperLogLog, histograms, ANALYZE, the stats cache + index/ JoinIndex trait, SortedKeyIndex + order.rs join graph, connectivity, DP over connected subsets + exec/multi.rs running a chosen order + plan.rs JoinBuilder, PhysicalPlan, EXPLAIN + source.rs RowSource / RowStream, table scans, buffer adapters + catalog_bridge.rs the ONLY place the subsystem reads the catalog + exec/ the seven operators +src/frontend/join_cmd.rs the interactive menu +src/bin/benchmark_joins.rs timings and cost calibration +``` + +Dependencies run downward: `exec/` uses `key`, `row`, `spill`, `memory`; +`plan` uses `cost`, `stats`, `index`, `order`; nothing below `plan` knows what +a catalog is. + +## Invariants worth not breaking + +1. **`JoinKey`'s three traits come from one encoding.** If you add a key class, + add it to `KeyClass::of`, `tag`, `encode_component`, and to the invariant + test in `test_join_key_encoding.rs` that checks byte order against + `Comparable::compare`. +2. **A NULL key is `Ok(None)`, never a sentinel.** Nothing may invent an + encoding for it. +3. **Executors are constructed from `ValidatedJoinSpec`.** Do not add a + constructor that skips it. +4. **Spill directories are removed by `Drop`.** Do not add a `close()`. +5. **Statistics are measured in key encoding.** Hashing a raw value would make + the distinct count count the wrong thing. +6. **Nothing below `catalog_bridge` reads the catalog.** That is what keeps the + tests hermetic. + +## Adding an algorithm + +1. Add a variant to `JoinAlgorithm` and an entry to `ALGORITHMS` in + `algorithm.rs`, declaring the join types it supports, whether it needs + equality keys or an inner index, and whether it can spill. +2. Write the operator in `exec/`. Take `&ValidatedJoinSpec` in the constructor; + implement `Iterator, JoinError>>` and `RowStream`. Use + `MatchEvaluator` for matching and `RowBuilder` for output rows - do not + re-derive either. +3. Add a cost arm in `CostModel::cost`. +4. Add it to `AVAILABLE` in `plan.rs` and to the dispatch in `build_operator`. +5. Add it to `EQUI_ALGORITHMS` in `tests/test_join_algorithms.rs`. + +Step 5 is the one that matters. That test runs every algorithm against the +reference join for six join types at two memory budgets, over fixtures with +duplicates on both sides, NULL keys on both sides, composite keys and empty +inputs. If it passes, the operator is very likely correct; if you skip it, no +other test will catch a wrong answer. + +`test_join_types.rs` walks the whole `ALGORITHMS × JoinType` cross product and +requires each pair either to be refused or to match the reference, so a matrix +entry claiming support that is not implemented fails. + +## Testing + +```sh +cargo test # everything +cargo test --test test_join_algorithms # the cross-algorithm matrix +cargo test --test test_join_key_encoding # the encoding against Comparable +cargo test --test test_join_spill # framing, scoping, cleanup +cargo test --test test_join_cli_no_panic # adversarial input +``` + +Randomised tests print their seed and honour `ROOKDB_JOIN_SEED`: + +```sh +ROOKDB_JOIN_SEED=12345 cargo test --test test_join_key_encoding +``` + +### The reference join + +`tests/join_common/mod.rs` has `reference_join` - a deliberately naive O(n·m) +loop. It shares the match evaluator with the real operators, because predicate +semantics have their own thorough tests and re-deriving them would only test +the test. What it does *not* share is the loop: no blocking, no partitioning, +no spilling, no sorting, no early exit. That is the part operators get wrong. + +`assert_rows_eq` compares **multisets of decoded rows**, not lengths. A join +that emits a row twice is as wrong as one that drops it, and comparing counts - +which is all the previous test suite ever did - cannot see a wrong value, a +wrong column order, or a NULL in the wrong place. + +### Forcing the spilling paths + +Set a tiny budget. Everything is reachable from a unit test: + +```rust +let config = JoinConfig::with_work_memory(8 * 1024).spill_root(db.path()); +``` + +Then assert the path was *taken*, not merely that nothing crashed: + +```rust +assert!(stats.partitions > 0); // the hash join partitioned +assert!(stats.sort_runs > 0); // the sort spilled runs +assert!(stats.spilled_groups > 0); // a duplicate group spilled +assert!(stats.oversized_partitions > 0);// one key dominated a partition +assert!(stats.role_reversed); // adaptive built from the left +``` + +### Hermetic tests + +`TempDb` creates a scratch directory named by pid and an atomic counter, and +removes it on drop. Because the subsystem never reads the catalog and never +resolves paths through `layout::*`, tests share no state: no global mutex, and +therefore no mutex to poison when one of them fails. + +Insert through `TableHandle::insert`, not `insert_single_tuple` - the latter +takes `&[&str]` and cannot express a NULL at all, which would make the entire +NULL test matrix impossible. + +## Reading EXPLAIN + +``` +Hash Join (Inner) [rows=2998 cost=1.24ms stats=analyzed] + Join Cond: emp.dept = dept.id + Residual: emp.salary < dept.budget + Index: 8000 entries on dept + Cost: io=0.28ms cpu=0.96ms extra passes=0 + -> Scan on emp [rows=300 pages=3] + Filter: emp.salary > 100 + -> Scan on dept [rows=300 pages=3] + Considered: Sort Merge 1.51ms, Block Nested Loop 2.34ms +``` + +- **`stats=`** - `analyzed` (a current sidecar), `header-only` (exact rows and + pages, inferred per-column values), or `defaults` (the table could not be + read). Treat estimates under the last two accordingly. +- **`Join Cond`** - the equalities driving the key. Empty means no key-based + algorithm was applicable. +- **`Residual`** - what the key could not express, re-checked per candidate. +- **`Filter`** - pushed into a scan. Absent under an outer join on the + row-preserving side, where pushing is illegal. +- **`extra passes`** - merge passes or partitioning rounds beyond the first. +- **`Considered`** - the alternatives, cheapest first. + +## Tuning + +Nothing behavioural is compiled in. `JoinTuning` holds the thresholds and every +one has an environment override, so they can be changed without a rebuild: + +| Variable | Default | Controls | +|---|---|---| +| `ROOKDB_JOIN_WORK_MEM` | a quarter of free RAM | bytes an operator may hold | +| `ROOKDB_JOIN_SPILL_ROOT` | `database/tmp/join` | where spill directories go | +| `ROOKDB_JOIN_FAN_OUT` | 16 | partitions per level in a hash join | +| `ROOKDB_JOIN_MAX_REPARTITION` | 3 | repartitioning attempts before giving up | +| `ROOKDB_JOIN_BLOCK_ROWS` | 1024 | outer rows a nested loop buffers | +| `ROOKDB_JOIN_SAMPLE_ROWS` | 8192 | rows read to find the smaller side | +| `ROOKDB_JOIN_PRESSURE_ROWS` | 65536 | rows between system-memory checks | +| `ROOKDB_JOIN_PRESSURE_FRACTION` | 0.10 | free-memory fraction that halves the budget | +| `ROOKDB_JOIN_MERGE_BUFFER` | 65536 | read-ahead assumed per run when merging | +| `ROOKDB_JOIN_MAX_DP_RELATIONS` | 8 | relations above which ordering goes greedy | +| `ROOKDB_JOIN_HISTOGRAM_BUCKETS` | 64 | buckets per column histogram | +| `ROOKDB_JOIN_HISTOGRAM_SAMPLE` | 20000 | values sampled for boundaries | +| `ROOKDB_JOIN_MAX_DISPLAY_ROWS` | 200 | rows the CLI prints | +| `ROOKDB_JOIN_FALLBACK_ROWS` / `_PAGES` / `_ROW_BYTES` | 1000 / 10 / 100 | assumed for an unreadable relation | + +`JoinConfig::resolve()` reads them; `JoinConfig::with_work_memory()` does not, +so a config built by hand is not altered behind the caller's back - which is +what keeps tests independent of the ambient environment. + +Values that are *not* tunable, because they are format or algorithm +definitions rather than policy: the spill and index file magic numbers, the +FNV constants, the HyperLogLog precision (it is baked into saved sketches), the +reservoir seed (determinism), and the capability matrix. + +## Common tasks + +**Improve estimates**: run ANALYZE (menu 15 → 3). Until then the planner has +exact cardinality but guesses distinct values at `n^0.75`, which understates a +unique column and makes index joins look worse than they are. + +**Make a join use an index**: build one (menu 15 → 4) on the *inner* relation's +key column. It is discovered automatically while the table is unchanged; any +modification invalidates the stamp and the index is ignored until rebuilt. + +**Investigate memory**: set `ROOKDB_JOIN_WORK_MEM` (bytes) or use menu 15 → 6. +Lowering it forces spilling and is the quickest way to reproduce a +partitioning problem. + +**Recalibrate the cost model**: see [cost-model.md](cost-model.md). + +## Where the spill files go + +`database/tmp/join/join-{pid}-{epoch}-{counter}/`, removed when the operator +drops - including during a panic. If a process is killed outright, the next +join sweeps directories whose owning process is gone and which are older than +an hour. Nothing else in that tree is ever touched. diff --git a/content/storage-engine/projects/join-algorithms/join-algorithms.md b/content/storage-engine/projects/join-algorithms/join-algorithms.md index f5f626a..94a2416 100644 --- a/content/storage-engine/projects/join-algorithms/join-algorithms.md +++ b/content/storage-engine/projects/join-algorithms/join-algorithms.md @@ -3,4 +3,96 @@ title: Join Algorithms sidebar_position: 2 --- -# Join Algorithms +# The RookDB join subsystem + +Equi and non-equi joins over the engine's typed row format, with a cost-based +planner, spilling operators, and statistics that are measured rather than +guessed. + +| Document | What it covers | +|---|---| +| [design-rationale.md](design-rationale.md) | Why each significant decision was taken, and what it prevents | +| [developer-guide.md](developer-guide.md) | Module map, how to add an algorithm, how to run and read the tests | +| [cost-model.md](cost-model.md) | The formulas, what remains an estimate, and how to recalibrate | + +## What it does + +Seven physical operators: + +| Algorithm | Needs | Supports | +|---|---|---| +| Simple nested loop | nothing | every join type | +| Block nested loop | nothing | every join type | +| Index nested loop | an equality and an index on the inner side | INNER, LEFT, SEMI, ANTI | +| Sort merge | an equality | all but CROSS | +| Hash (in-memory / hybrid / Grace) | an equality | all but CROSS | +| Symmetric hash | an equality, and both inputs in memory | all but CROSS | +| Adaptive | nothing | every join type | + +Join types: INNER, LEFT OUTER, RIGHT OUTER, FULL OUTER, CROSS, SEMI, ANTI. +NATURAL is a rewrite, not a join type. LATERAL is not supported, because +nothing in the engine can evaluate a correlated subquery. + +## Using it + +From the CLI, main menu option **15 - Join Operations**. Everything is chosen +from numbered prompts; there is no query language. + +``` +1. Run a join two tables, any join type +2. Explain a join the same, showing only the plan +3. Run a multi-table join three or more tables, order chosen by cost +4. Analyze a table measure distinct values, NULLs and histograms +5. Build a join index a sorted index on one column +6. Drop a join index +7. Join settings working memory, spill directory +``` + +From Rust: + +```rust +use storage_manager::join::{JoinBuilder, JoinType, catalog_bridge}; + +let left = catalog_bridge::resolve(&catalog, "shop", "orders", "o")?; +let right = catalog_bridge::resolve(&catalog, "shop", "customers", "c")?; + +let mut stream = JoinBuilder::new(left, right, JoinType::Inner) + .with_condition(condition) + .execute()?; + +while let Some(row) = stream.next() { + let row = row?; // serialized, in the join's output schema +} +``` + +Multi-relation joins go through the optimiser: + +```rust +use storage_manager::join::{JoinGraph, TableStatsCache, execute_ordered, optimize}; + +let graph = JoinGraph::build(relations, Some(&condition), &TableStatsCache::new())?; +let plan = optimize(&graph, config.work_memory_bytes)?; +let mut stream = execute_ordered(&graph, &plan, &config)?; +``` + +## Two things worth knowing before reading the code + +**A join key is a byte string.** `JoinKey` derives `Hash`, `Eq` and `Ord` from +one order-preserving encoding, so hash equality and merge ordering cannot +disagree. A NULL key has no encoding at all - `try_key` returns `Ok(None)` - +which is how "NULL never matches NULL" holds in every algorithm at once, +including after a row has been written to and read back from a spill file. + +**Statistics are measured, or admitted to be absent.** Every plan carries a +confidence - `analyzed`, `header-only`, or `defaults` - and EXPLAIN prints it. +A stale statistics sidecar is ignored, never partially trusted. + +## Running the tests + +```sh +cargo test # everything +cargo test --test test_join_algorithms # all operators against the reference join +cargo test --test test_join_multi_way # multi-relation execution +cargo test --test test_join_key_encoding # the key encoding against Comparable +cargo run --release --bin benchmark_joins # timings, and cost-model calibration +``` diff --git a/sidebars.ts b/sidebars.ts index e05b16b..ce5e78d 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -99,6 +99,9 @@ const sidebars: SidebarsConfig = { collapsed: true, items: [ "storage-engine/projects/join-algorithms/join-algorithms", + "storage-engine/projects/join-algorithms/design-rationale", + "storage-engine/projects/join-algorithms/developer-guide", + "storage-engine/projects/join-algorithms/cost-model", ], }, // Buffer Manager