Skip to content

feat(parquet): reconstruct unknown fields on reads - #3073

Draft
manuzhang wants to merge 3 commits into
apache:mainfrom
manuzhang:add-unknown-datatype-parquet-read
Draft

feat(parquet): reconstruct unknown fields on reads#3073
manuzhang wants to merge 3 commits into
apache:mainfrom
manuzhang:add-unknown-datatype-parquet-read

Conversation

@manuzhang

Copy link
Copy Markdown
Member

Which issue does this PR close?

No issue is currently linked.

What changes are included in this PR?

This is the Parquet-read follow-up to #2773 and #3072 and is intentionally opened as a draft.

  • Excludes non-physical Unknown fields from Parquet projection masks and reconstructs them as Arrow null arrays.
  • Preserves row counts for empty and Unknown-only projections.
  • Makes embedded field IDs and name mappings authoritative, with guarded positional fallback for files without either.
  • Reconstructs missing nested fields and materializes primitive and container initial defaults.
  • Includes the requested top-level struct/list/map default fix and regression rather than discarding non-primitive defaults.

Until #2773 and #3072 merge, this draft temporarily includes their commits in its cumulative diff. After both predecessors merge, the branch will be rebased onto main so this PR contains only the Parquet-read commit.

Are these changes tested?

  • Added read-path regressions for Unknown-only projections, filters, schema evolution, missing/deleted IDs, nested fallback, and primitive/nested/top-level defaults.
  • cargo test -p iceberg --lib (1,610 passed)
  • make check-fmt
  • make check-clippy
  • git diff --check

AI Disclosure

This PR was prepared with assistance from Codex.

manuzhang and others added 3 commits August 26, 2026 12:38
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Codex <codex@openai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant