Skip to content

fix: reassign predicate column indices by name before building pruning predicates#7

Merged
radustoenescu merged 1 commit into
hstack:main-53from
radustoenescu:fix-parquet-fmt-extra-column-index-mismatch
Jul 14, 2026
Merged

fix: reassign predicate column indices by name before building pruning predicates#7
radustoenescu merged 1 commit into
hstack:main-53from
radustoenescu:fix-parquet-fmt-extra-column-index-mismatch

Conversation

@radustoenescu

Copy link
Copy Markdown
Collaborator

Problem

ParquetSource::try_pushdown_filters (and other producers of self.predicate) can store a filter whose Column carries an index that is only valid for the schema it was originally built against. In particular, when a join's filter-pushdown/inference produces a filter for one side from a condition on the other side, the resulting Column keeps the name of the join key but can keep the other side's index.

build_pruning_predicates (used both by the real per-file pruning path and by ParquetSource::fmt_extra's Display output) uses that predicate's Column indices as-is against file_schema. When the index happens to land on a differently-typed column (e.g. a leading struct-typed column), this can panic in debug builds:

PhysicalExprSimplifier::simplify runs a debug-only type-preservation assertion (node.data_type(schema).unwrap()) before/after each rewrite. Resolving the mismatched Column by its (wrong) index against file_schema produces a type-coercion error (e.g. Struct(..) = Float64), and the .unwrap() panics.

This is most visible via Display (EXPLAIN, logging, or any code that displays the plan) because that path does not go through per-file schema adaptation the way the real execution path does - the real path (ParquetOpener::open) is protected by whatever PhysicalExprAdapter is configured, which re-resolves columns by name before anything type-sensitive runs. fmt_extra has no such adapter available (it doesn't have the per-file physical schema), so it's exposed to the raw, possibly-mismatched predicate.

Fix

In build_pruning_predicates, reassign every Column's index by name against file_schema before building the pruning/page-pruning predicates, using the existing reassign_expr_columns helper (datafusion_physical_expr::utils::reassign_expr_columns), which is already used elsewhere in this crate for the same purpose (row_filter.rs, opener.rs). This is a 4-line change and makes build_pruning_predicates robust regardless of which producer handed it a stale-indexed predicate.

Test

Added test_display_does_not_panic_on_predicate_with_mismatched_column_index in source.rs: builds two real parquet-backed ParquetSources (one with a leading struct column, matching the real-world shape that triggered this), joins them via HashJoinExec, and attaches to one side a Column-mismatched predicate wrapped in a DynamicFilterPhysicalExpr (matching the shape that makes PruningPredicate::try_new take its snapshot/simplify path). Asserts displayable(...).to_string() does not panic.

Verified:

  • Fails with the exact original panic message (Cannot infer common argument type for comparison operation Struct(..) = Float64) when the fix is reverted.
  • Passes with the fix applied.
  • Full crate test suite (cargo test -p datafusion-datasource-parquet) and full workspace test suite (cargo test --workspace) pass with the fix applied, no regressions.

…ng pruning predicates

A filter's Column can carry an index that is only valid for the schema it
was originally built against (e.g. a filter inferred across a join keeps
the other side's index). ParquetSource::try_pushdown_filters and other
producers store such predicates as-is, so build_pruning_predicates (used
both for real per-file pruning and for ParquetSource::fmt_extra's Display
output) could receive a Column whose index no longer matches its own name
in file_schema.

In debug builds this can panic: PhysicalExprSimplifier::simplify runs a
debug-only type-preservation assertion that calls Column::data_type()
using the raw index, which can resolve to an unrelated column (e.g. a
struct-typed one), producing a type-coercion error that gets unwrapped.
This is most visible via fmt_extra/Display (EXPLAIN, logging, or any code
that displays the plan), since it does not go through per-file schema
adaptation the way the real execution path does.

Fix build_pruning_predicates to reassign every Column's index by name
against file_schema before use, via the existing reassign_expr_columns
helper (already used elsewhere in this crate for the same purpose).

Adds a regression test reproducing the panic with real parquet-backed
ParquetSources joined via HashJoinExec, using a DynamicFilterPhysicalExpr
to match the shape of the predicate that triggers PruningPredicate::try_new's
snapshot/simplify path.
@radustoenescu
radustoenescu merged commit 8df4805 into hstack:main-53 Jul 14, 2026
18 of 28 checks passed
@radustoenescu
radustoenescu deleted the fix-parquet-fmt-extra-column-index-mismatch branch July 14, 2026 15:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants