fix: equalize with delta/delta-rs (v0.52) - #55
Open
mandrush wants to merge 645 commits into
Open
Conversation
github-actions
Bot
added
binding/python
binding/rust
delta-inspect
documentation
adampolomski
previously approved these changes
Feb 17, 2026
mandrush
requested review from
amlynczak-rel,
bexway-relativity,
gregott-rel and
gtrawinski
February 17, 2026 14:17
Signed-off-by: R. Tyler Croy <rtyler@brokenco.de>
This test doesn't really perform a useful check. The reader being _capable_ of reading DeletionVectors and ColumnMapping is not tied to the features on the table per se Signed-off-by: R. Tyler Croy <rtyler@brokenco.de>
Based on my read of the dioscussion [here](delta-io/delta-kernel-rs#2267) the id ordering assertion made by this test was not correct and should just be removed Signed-off-by: R. Tyler Croy <rtyler@brokenco.de>
Variant support was initially introduced in [21.0.0](https://arrow.apache.org/release/21.0.0.html) released in July 2025. Lower versions simply won't work with variant types IMHO it's not worth trying to maintain compatibility with older pyarrow and carving out a bunch of conditional logic for variant Signed-off-by: R. Tyler Croy <rtyler@brokenco.de>
Signed-off-by: R. Tyler Croy <rtyler@brokenco.de>
Signed-off-by: R. Tyler Croy <rtyler@brokenco.de>
Signed-off-by: R. Tyler Croy <rtyler@brokenco.de>
Signed-off-by: R. Tyler Croy <rtyler@brokenco.de>
…cuuming Closes #3896 Signed-off-by: R. Tyler Croy <rtyler@brokenco.de>
Closes #4468 Signed-off-by: R. Tyler Croy <rtyler@brokenco.de>
Skip child pushdown adaptation for filters that reference columns materialized by DeltaScanExec rather than present in the Parquet child schema. This preserves correctness for join dynamic filters while retaining Parquet pushdown for native child columns. Refs #4467 Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
This is largely a port of the regression test in #3172, with a little human intelligence added in 😄 Signed-off-by: R Tyler Croy <rtyler@brokenco.de>
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
…park This is mostly to prove to myself the behavior of append writes to a table which is also being optimized by Apache Spark. I don't fully trust these tests even though they logically make sense and demonstrate the behavior I was expecting. I feel like there _must_ be some possible race condition with a distributed system like S3 but because the Rust side of the equation will always use conditional puts, I _think_ that ensures safey in a multi-write system where a Rust process is appending and a Spark process is optimizing. I **don't** think this would be safe if there were multiple Spark writers running concurrently however. References #4482 Co-Authored-By: Devstral 2 <vibe@mistral.ai> Signed-off-by: R. Tyler Croy <rtyler@brokenco.de>
…right jobs Signed-off-by: R. Tyler Croy <rtyler@brokenco.de>
DeltaTable.scan(columns, predicate) returns an arro3 RecordBatchReader backed by the kernel DataFusion provider, reusing the QueryBuilder session setup and the DataFrame API with no SQL string assembly. The predicate is a row filter executed by the engine, unlike the listing APIs' file_pruning_predicate which only skips whole files. Works on column-mapped and deletion-vector tables. Signed-off-by: Frank Portman <frank1214@gmail.com>
Adds a scan section to the querying usage page and notes in both the guide and the docstring that the predicate follows DataFusion SQL semantics (three-valued NULL logic, SQL type coercion), which differ from pyarrow dataset filters in edge cases. Signed-off-by: Frank Portman <frank1214@gmail.com>
Review nit on #4675: DataFusion SQL, generic dialect, identifier normalization disabled per DeltaParserOptions, so unquoted column names match case sensitively. Signed-off-by: Frank Portman <frank1214@gmail.com>
Signed-off-by: Frank Portman <frank1214@gmail.com>
… methods On to_pandas and to_pyarrow_table, filters already prunes files: each dataset fragment carries partition values and min/max statistics as a guarantee expression, so files that cannot match are skipped before any data is read, then rows filter exactly. A pruning parameter there is a second way to pass the same tuples with weaker semantics. It stays where nothing else can do the job: file_uris, partitions, to_pyarrow_dataset. The parameter landed in #4585 and has not been released, so this is not a breaking change. It is one only if it waits past the next release. Follow up to #4585. Signed-off-by: Frank Portman <frank1214@gmail.com>
The warning told users to pass file_pruning_predicate, which this change removes from these two methods. The dataframe methods now warn with the replacement that exists there and route the legacy value through the dataset parameter that still accepts it. Signed-off-by: Frank Portman <frank1214@gmail.com>
The dataframe methods are thin wrappers over to_pyarrow_dataset; showing the equivalent unrolled chain makes it apparent where pre-scan pruning via file_pruning_predicate slots in, which can matter on tables with very large file counts. Signed-off-by: Frank Portman <frank1214@gmail.com>
A row group cannot span files, so when target_file_size is reached the PartitionWriter closes the file with whatever partial row group is buffered, truncating every file's last group wherever the byte target happens to land. Readers that map one Parquet row group to one in-memory segment degrade on those runt tail groups. With DELTARS_ROLL_ON_ROW_GROUP_BOUNDARY=1 (or PartitionWriterConfig::with_roll_on_row_group_boundary), the roll defers until the open row group completes: slices are aligned so they never straddle a group boundary, and the byte check only fires when the writer sits exactly on one. Every row group is then exactly max_row_group_row_count rows -- the single end-of-data remainder at close is the only exception -- and a file may overshoot target_file_size by up to one row group. Only effective when row groups are bounded by row count alone; byte-bounded groups keep the legacy roll. Default off: existing behavior (including test_unflushed_row_group_size) is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: djouallah <djouallah@users.noreply.github.com>
…nating Address review comments on #4677: - PartitionWriterConfig::try_new now errors on write_batch_size = 0, which would otherwise spin the write loop forever (previously step_by panicked). - The roll-on-row-group-boundary test now writes 700-row batches so slices straddle the 1024-row group boundary, and asserts as a negative control that the plain byte roll truncates groups under the same setup. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: djouallah <djouallah@users.noreply.github.com>
Signed-off-by: Adam Reeve <adam.reeve@gr-oss.io>
Signed-off-by: Adam Reeve <adam.reeve@gr-oss.io>
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
When deletion vectors are present the inner parquet plan's row counts do not account for deleted rows; an exact num_rows lets DataFusion's AggregateStatistics optimizer short-circuit COUNT(*) with inflated counts. Ported from Nordalf/delta-rs (original: 6b523d7) onto the statistics_from_inputs contract. Signed-off-by: Alexander Falk <alexfalk7@gmail.com>
…tables While cleaning up logging in delta-kernel-rs I decided to point the :parrot: at our codebase for log statements that might produce excess log lines when working with larger tables. Any log statement which may result in a debug print of a `Snapshot` runs the risk of printing a `Debug` representation of the entire reconciled transaction log v_v Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: R Tyler Croy <rtyler@scribd.com>
Verify FSCK behavior for physical paths containing spaces and literal percent sequences, including dry runs, Remove encoding, and snapshot replay. Exercise restore_file_key through Snapshot with valid deletion vectors and distinct offsets. Replace the invalid inline offset fixture and duplicated formatter assertion. Follow up to #4684. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: R Tyler Croy <rtyler@scribd.com> Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Updates the requirements on [brotli-decompressor](https://github.com/dropbox/rust-brotli-decompressor) to permit the latest version. - [Commits](dropbox/rust-brotli-decompressor@5.0.1...6.0.0) --- updated-dependencies: - dependency-name: brotli-decompressor dependency-version: 6.0.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
# Description Right now, there are several duplicated read and write paths for creating parquet files within delta-rs. The purpose of this PR is to consolidate these paths behind a more abstract interface in two parts: 1. A basic interface that needs only the parquet library. 2. An extended interface using datafusion and supporting advanced features such as predicates and projections. There are a few reasons for doing this: 1. Code maintenance. Having (mostly) a single path for reads and writes helps reduce maintenance and makes it easier to control behavior around how underlying reads and writes are handled. 2. Consolidating these paths makes it a lot easier to attach advanced parquet properties to reads and writes. Examples would include future encryption support and content defined chunking as discussed in #4465 3. I believe that having the simplified read and write paths will also help with delta-rs kernel integration. This PR follows from the ideas discussed at the deltabricks summit and the slack channel at: https://delta-users.slack.com/archives/C013LCAEB98/p1781571891482589 AI notes: This PR was created with (and reviewed with) the help of AI. However, the design and individual pieces were reviewed and tested by me. I believe them to be correct and an improvement to the project. ## Behavior changes in the legacy writers The `JsonWriter`/`RecordBatchWriter` rewrite (now streaming through the consolidated write path) intentionally changes a few observable behaviors — flagging them explicitly for reviewers: 1. **All-or-nothing flush windows.** Batches stream to storage as they are written, and a flush window commits atomically: an IO error on any write discards *every* batch buffered since the last flush (the aborted uploads are invisible to vacuum; finalized files are left for vacuum). The old writers buffered in memory and lost only the failing batch. Documented on both writers. 2. **`JsonWriter` no longer quarantines bad records.** A write containing malformed JSON records now rejects the whole call and writes nothing, instead of writing the good records and returning `PartialParquetWrite` with the skipped values. (The error enum was already crate-private, so this is a behavior change, not an API break.) 3. **`write_partition` + `MergeSchema`:** unpartitioned widening now updates the writer's schema, and a later `flush_and_commit` commits the metadata evolution; widening on a partitioned table is rejected (fixes the class of bug in #3783 where files could carry columns absent from table metadata). 4. **Counter semantics:** `buffered_record_batch_count` counts write calls (previously one per partition touched per call), and `buffer_len` approximates encoded parquet size written this window (previously in-memory buffer bytes). Documented on the methods. --------- Signed-off-by: Corwin Joy <corwin.joy@gmail.com> Signed-off-by: R. Tyler Croy <rtyler@brokenco.de> Signed-off-by: R Tyler Croy <rtyler@scribd.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: R. Tyler Croy <rtyler@brokenco.de> Co-authored-by: R Tyler Croy <rtyler@scribd.com>
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Remove the stale selection_vectors reference introduced by the main merge; map_statistics already makes deletion-vector statistics inexact. Rename the absent DV state to NotPresent, restore DashMap in the replay helper, and document the per-execution mask copy. Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Signed-off-by: Adam Reeve <adam.reeve@gr-oss.io>
Signed-off-by: Adam Reeve <adam.reeve@gr-oss.io>
Signed-off-by: Ion Koutsouris <15728914+ion-elgreco@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The description of the main changes of your pull request
Related Issue(s)
Documentation