Qualify reader file growth, commit limits, and request costs - #12
Merged
Conversation
Add optional deterministic checkpoint hash trees, publish and exhaustively verify their objects, and preserve index-free materialized fixtures. Add exact, version-consistent bounded ranges for local, memory and S3 storage and bounded pack-index decoding. Validation: isolated staged snapshot passed 150 workspace tests with all features, including deterministic S3 range contracts. Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Pin current or historical metadata and selected snapshots through explicit references. Resolve authenticated page ranges through a bounded shared cache and a read-only Turso worker, validate selected relational metadata, and enumerate files with bounded keyset cursors. Bound native record allocation and page references, propagate asynchronous storage errors, cancel pending reads, and reserve engine working memory. Existing materialized pins and exhaustive verification keep their validation guarantees. Validation: independent staged snapshot passed 182 workspace tests without DataFusion, including reader integrity, isolation, cancellation and resource limits. Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Add a native TableProvider with typed inexact pruning, bounded 256-file metadata batches, immutable schema-ID adapters, defaults, nested types, version-pinned Parquet reads, and bounded shared footer inspection. Reserve retained file descriptors through the DataFusion memory pool and preserve residual filters and provider lifetime pins. Include runnable query and qualification examples plus real Parquet parity, historical selection, schema evolution, randomized pruning and budget tests. Keep the S3 storage dependency independent and size CI builds for the native query stack. Validation: final source passed 214 workspace tests, strict Clippy, doctests, provider harness, fixture checks, crash scenarios, protocol WASM, cargo deny and cargo audit. The 276 MiB metadata workload registered with 0.146 percent transfer, then pruned 14 of 16 files and produced the expected native SQL result. Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Document snapshot and schema binding, authentication versus exhaustive verification, cache and planning budgets, and runnable SQL and qualification examples. Preserve the final three-phase measurements, source commit and local log hashes, the complete local matrix and prior COW evidence. Keep live provider qualification separate. Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
ethan-tyler
changed the title
ethan-tyler
marked this pull request as ready for review
September 7, 2026 19:55
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds repeatable qualification for file-count growth, large current commits, request delay, and planning memory on top of the OTMP reader in #11. The 16,384-file workload shows that cheap registration does not imply cheap planning; a controlled cache experiment identifies one contributor while preserving the remaining performance questions.
The release-mode matrix contains 1,130 samples across 65 configurations: 960 successful measurements (including 40 planning-only runs) and 170 retained resource-limit failures. Executed queries match fixture count/sum; no sample reports a secondary invariant violation, forbidden full read, fixture mutation, or unreleased planning reservation. All 23 fixtures were produced by the writer and exhaustively verified.
Measured findings:
The report includes p50/p95, counters, limits, source/binary/fixture provenance, compressed per-case distributions, and hashes for 5,974 retained raw artifacts. The harness keeps compilation and verification outside timed subprocesses, separates failures from successful distributions, preserves phase/pass boundaries, and bounds process-group timeouts. CI runs native and Python harness tests plus a six-sample end-to-end smoke. Existing crash scripts now honor CARGO_TARGET_DIR.
Local validation: 214 workspace tests; 11 native harness tests; 18 strict Python tests; native smoke; strict Clippy; doctests; conformance regeneration; deterministic S3 contracts; crash tests; protocol WASM; cargo-deny and cargo-audit. Reaggregation matches every stored summary and all evidence hashes verify. Final revision
ca11c04c999649cf0f26e38fa3640478cd2469f2passed all six CI jobs and the separate Audit job. The Linux test job also passed the native harness, all 18 Python checks, and the six-sample smoke.This PR is stacked on
feat/datafusion-reader(#11). It changes qualification tooling and evidence; runtime defaults are unchanged. Request-delay simulation is not live AWS/R2 evidence. Neither reader PR is merged by this delivery.