Skip to content

feat(parquet): omit unknown fields from writes - #3072

Draft
manuzhang wants to merge 2 commits into
apache:mainfrom
manuzhang:add-unknown-datatype-parquet-write
Draft

feat(parquet): omit unknown fields from writes#3072
manuzhang wants to merge 2 commits into
apache:mainfrom
manuzhang:add-unknown-datatype-parquet-write

Conversation

@manuzhang

Copy link
Copy Markdown
Member

Which issue does this PR close?

No issue is currently linked.

What changes are included in this PR?

This is the Parquet-write follow-up to #2773 and is intentionally opened as a draft.

  • Builds the physical Parquet Arrow schema by omitting Unknown fields.
  • Rejects containers that would have no valid physical Parquet representation.
  • Projects incoming record batches to the physical schema before writing while preserving nested container structure and row counts.
  • Updates NaN statistics traversal to work against the projected physical batch.

Until #2773 merges, this draft temporarily includes the Unknown logical-type commit in its cumulative diff. After #2773 merges, the branch will be rebased onto main so this PR contains only the Parquet-write commit.

Are these changes tested?

  • Added schema and batch-projection regressions for top-level and nested Unknown fields, invalid empty containers, and name-based matching.
  • cargo test -p iceberg --lib (1,594 passed at this follow-up layer)
  • make check-fmt
  • make check-clippy
  • git diff --check

AI Disclosure

This PR was prepared with assistance from Codex.

manuzhang and others added 2 commits August 26, 2026 12:38
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Codex <codex@openai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant