Reduce reader planning overhead and qualify against Iceberg - #13
Merged
Conversation
Add optional deterministic checkpoint hash trees, publish and exhaustively verify their objects, and preserve index-free materialized fixtures. Add exact, version-consistent bounded ranges for local, memory and S3 storage and bounded pack-index decoding. Validation: isolated staged snapshot passed 150 workspace tests with all features, including deterministic S3 range contracts. Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Pin current or historical metadata and selected snapshots through explicit references. Resolve authenticated page ranges through a bounded shared cache and a read-only Turso worker, validate selected relational metadata, and enumerate files with bounded keyset cursors. Bound native record allocation and page references, propagate asynchronous storage errors, cancel pending reads, and reserve engine working memory. Existing materialized pins and exhaustive verification keep their validation guarantees. Validation: independent staged snapshot passed 182 workspace tests without DataFusion, including reader integrity, isolation, cancellation and resource limits. Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Add a native TableProvider with typed inexact pruning, bounded 256-file metadata batches, immutable schema-ID adapters, defaults, nested types, version-pinned Parquet reads, and bounded shared footer inspection. Reserve retained file descriptors through the DataFusion memory pool and preserve residual filters and provider lifetime pins. Include runnable query and qualification examples plus real Parquet parity, historical selection, schema evolution, randomized pruning and budget tests. Keep the S3 storage dependency independent and size CI builds for the native query stack. Validation: final source passed 214 workspace tests, strict Clippy, doctests, provider harness, fixture checks, crash scenarios, protocol WASM, cargo deny and cargo audit. The 276 MiB metadata workload registered with 0.146 percent transfer, then pruned 14 of 16 files and produced the expected native SQL result. Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Document snapshot and schema binding, authentication versus exhaustive verification, cache and planning budgets, and runnable SQL and qualification examples. Preserve the final three-phase measurements, source commit and local log hashes, the complete local matrix and prior COW evidence. Keep live provider qualification separate. Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
ethan-tyler
changed the base branch from
feat/reader-scale-qualification
to
main
September 8, 2026 23:18
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Warm selective planning over 16,384 files previously took 3.43 seconds. Indexed branch pagination, bounded metric probes, shared authenticated pages and validated footer-byte reuse reduce it to 131 ms p50 (26× faster), while preserving snapshot pins, authentication, residual filters and existing budgets.
This PR also adds a reproducible comparison with Iceberg Rust on the same DataFusion 55 / Parquet 59 graph. Iceberg parity is not achieved: matched warm selective SQL takes 124 ms versus Iceberg's 13.7 ms; broad scans retain considerably more process memory. At 256 files with 10 ms injected request delays, cold planning improves from 17.58 to 9.86 seconds, but warm planning remains about 3.2 seconds because each scan still stats every retained file. Large-current-commit availability limits also remain documented.
88cefa7f6cab3bd7670668cb99c43b974f63bd6e, including the standalone comparator Rust checks and Linux crash tests.Targets main after #11 and #12 landed. The ancestry reconciliation preserves the exact qualified source tree; fresh CI runs on the updated head. This is local and deterministic qualification; simulated delays are not AWS/R2 evidence. The report identifies the remaining metadata CPU, per-file round-trip and broad-scan memory costs.