Bound reader preflight concurrency and share cache fills - #14
Merged
Conversation
Add optional deterministic checkpoint hash trees, publish and exhaustively verify their objects, and preserve index-free materialized fixtures. Add exact, version-consistent bounded ranges for local, memory and S3 storage and bounded pack-index decoding. Validation: isolated staged snapshot passed 150 workspace tests with all features, including deterministic S3 range contracts. Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Pin current or historical metadata and selected snapshots through explicit references. Resolve authenticated page ranges through a bounded shared cache and a read-only Turso worker, validate selected relational metadata, and enumerate files with bounded keyset cursors. Bound native record allocation and page references, propagate asynchronous storage errors, cancel pending reads, and reserve engine working memory. Existing materialized pins and exhaustive verification keep their validation guarantees. Validation: independent staged snapshot passed 182 workspace tests without DataFusion, including reader integrity, isolation, cancellation and resource limits. Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Add a native TableProvider with typed inexact pruning, bounded 256-file metadata batches, immutable schema-ID adapters, defaults, nested types, version-pinned Parquet reads, and bounded shared footer inspection. Reserve retained file descriptors through the DataFusion memory pool and preserve residual filters and provider lifetime pins. Include runnable query and qualification examples plus real Parquet parity, historical selection, schema evolution, randomized pruning and budget tests. Keep the S3 storage dependency independent and size CI builds for the native query stack. Validation: final source passed 214 workspace tests, strict Clippy, doctests, provider harness, fixture checks, crash scenarios, protocol WASM, cargo deny and cargo audit. The 276 MiB metadata workload registered with 0.146 percent transfer, then pruned 14 of 16 files and produced the expected native SQL result. Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Document snapshot and schema binding, authentication versus exhaustive verification, cache and planning budgets, and runnable SQL and qualification examples. Preserve the final three-phase measurements, source commit and local log hashes, the complete local matrix and prior COW evidence. Keep live provider qualification separate. Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sequential per-file preflight serialized storage latency and concurrent scans could duplicate cache fills. This change admits up to eight preflights per provider by default (configurable from 1 to 32), overlaps metadata retrieval with validation, shares compatible footer/range loads, and groups bounded SQL work behind FIFO engine admission.
Contract and scope
Preserves snapshot pins, authenticated metadata, full schema validation, residual filters, existing byte-budget defaults, and error codes/causes. Cache admission accounts for active cached validation leases, including option-dependent decoding and final schema rebinding. Cancellation releases scan-owned work; no partial plan is published. Runtime isolation and parallel metadata execution remain follow-ups.
Includes deterministic provider/cancellation tests, four bounded Loom models, and a harness that rejects incomplete qualification evidence. The alpha API compatibility notes and runtime-isolation brief are included.
Verification
cargo fmt --all, comparator formatting,cargo nextest run --locked --workspace --all-features(251 passed), and reader-scale Python tests (23 passed). Staged bytes match all 239 qualified source hashes.Evidence: V8 repair and qualification report, with source/binary hashes, raw samples, failures, and the independent audit. Earlier V7 artifacts remain unchanged; archival reports describe their original local-only status.
Targets main after #11, #12, and #13 landed. The ancestry reconciliation preserves the complete source/evidence tree from commit
734408c. All seven GitHub checks passed on the original head and final headc8e8051: CI and audit. Merged as10ceea0d3543c2c288df09d6887f83820143e413.