Track and remove native Variant compatibility workarounds after upstream fixes

[peterxcli]

What is the problem the feature request solves?

Native Variant projection in #5407 includes compatibility paths that exist because Comet currently uses Arrow/Parquet 58.4.0, because DataFusion does not expose every Arrow reader policy needed by a Spark-compatible scan, or because Comet must interoperate with older Spark Variant encodings. These paths are intentionally narrow, but some perform row-by-row decoding or metadata rebuilding and should not become permanent accidentally.

This issue records exactly which code is removable after an upstream fix ships, which upstream change owns it, and which nearby code is required Comet/Spark integration and must remain.

This is maintenance follow-up for #5407 under the Native Variant epic #5438. Spark UTF-16 output cleanup is tracked separately by #5474. Unicode case-insensitive Parquet field matching is tracked by #5495.

Arrow-rs compute workarounds

• Encoded Variant storage children

  • Comet recursively decodes Dictionary arrays before VariantArray::try_new in normalize_variant_storage.
  • Upstream [Variant] Accept Dictionary and RunEndEncoded metadata in VariantArray arrow-rs#10802 and fix(variant): accept Dictionary and RunEndEncoded metadata in VariantArray arrow-rs#10810 cover Dictionary and RunEndEncoded canonical metadata. Embedded Arrow schemas can also restore dictionary-encoded value and typed_value, which those changes do not cover.
  • After Comet upgrades to a release with encoded-metadata support, stop decoding metadata locally and retain only compatibility for encoded value/typed_value until upstream supports those children or Comet establishes a narrower fallback policy. Keep focused regressions so the upstream path is exercised directly.

• Empty Variant object keys

  • Arrow-rs 58.4 rejects equal metadata offsets used by empty dictionary entries. Comet retries unshredding in unshred_variant_for_spark, canonicalizes affected rows in canonicalize_spark_empty_key_metadata, and uses shallow metadata parsing while rebuilding values.
  • Upstream: Add proptest fuzzing to parquet-variant and implement fixes for findings arrow-rs#10352, merged but not present in Comet's current dependency.
  • After upgrading to a release containing the fix, remove the retry and canonicalizer, restore strict VariantMetadata::try_new at source-consumption sites, and change tests that assert the old rejection to assert successful validation.
  • Here, “empty field name” means an empty Variant object key such as {"": 1}, not an empty Arrow schema field name.

• Unsigned shredded typed_value fields

  • Comet widens UInt8 -> Int16, UInt16 -> Int32, and UInt32 -> Int64 before unshredding. UInt64 -> Decimal(20,0) remains Spark-specific.
  • Upstream options are unresolved: Parquet Variant rejects canonical Arrow UInt8/UInt16/UInt32 typed values arrow-rs#10416 / variant: Support Arrow UInt8/UInt16/UInt32 typed values arrow-rs#10417 add widening, while [Docs][Format] Variant extension: align typed_value primitive type mappings with the Parquet shredding spec arrow#50622 / GH-50622: [Docs][Format] Align Variant typed_value primitive type mappings with the Parquet shredding spec arrow#50810 propose removing unsigned mappings from the canonical Variant shredding table.
  • Remove the first three unsigned arms if Arrow-rs provides the widening in a released dependency. If the canonical format rejects unsigned shredding instead, retain them only as an explicit legacy/noncanonical-file compatibility policy or fall back safely. Do not remove the shared recursive normalizer wholesale.

• Wide physical DECIMAL typed_value fields

  • Arrow-rs 58.4 infers FIXED_LEN_BYTE_ARRAY wider than 16 bytes as Decimal256 even when the declared precision is at most 38. VariantArray rejects Decimal256, while Spark reads the declared DecimalType in its 128-bit representation.
  • Comet recursively narrows Decimal256(p, s) -> Decimal128(p, s) for p <= 38. Runtime regressions cover positive and negative DECIMAL(38,2) values stored in 17- and 32-byte physical fields.
  • This is not fixed by stripping ARROW:schema or by arrow-rs#10417. Remove the arm only if upstream Variant compute accepts the representation or the reader can supply Spark-compatible physical inference without it.

DataFusion/Arrow reader integration workaround

• Use Spark-compatible physical schema inference for projected Variant scans
  • Spark ignores the advisory ARROW:schema footer hint, while Arrow-rs honors it and can restore shredded leaves as Decimal256, Date64, Dictionary, or fixed-size types. Comet therefore replaces an embedded hint with physical Parquet inference for an unencrypted scan whose pruned required schema projects Variant.
  • Arrow-rs also maps physical Parquet ENUM to Binary, while Spark maps ENUM to String. Once Arrow inference has discarded that annotation, the Variant normalizer cannot distinguish ENUM from raw BINARY. Comet therefore adds back a physical-schema-derived hint that changes only ENUM leaves to Utf8 (ENUM mapping, footer rewrite). Runtime controls verify ENUM and STRING become Variant strings while unannotated BINARY remains binary.
  • All other key/value metadata, row groups, column orders, column indexes, and offset indexes are preserved. Using the pruned schema also preserves feat: Support native scans with unprojected Spark 4 VARIANT columns #5377: an unread Variant column does not activate this path. Encrypted Variant scans remain on Spark because reconstructing public FileMetaData cannot preserve arrow-rs's private decryptor state.
  • ArrowReaderOptions::with_skip_arrow_metadata(true) can replace only the hint suppression. Remove the manual reconstruction and encrypted fallback when DataFusion exposes that option per source while preserving the metadata cache and encryption state; remove the ENUM hint only when the reader exposes a nested physical-leaf coercion hook or directly provides Spark-compatible ENUM inference.

Describe the potential solution

For each checklist item:

1. Wait for the relevant upstream behavior to merge and appear in a released dependency or reachable DataFusion reader API.
2. Upgrade Arrow/Parquet/DataFusion through the normal dependency update.
3. Delete only the mapped compatibility branch; do not duplicate an upstream implementation locally.
4. Keep or adapt the existing regression tests to prove the upstream path works through Comet.
5. Run the focused Rust Variant tests, Spark 4 Variant SQL tests, Spark 3 compilation boundary, formatting/lint, and git diff --check.

Definition of done:

• Every temporary upstream workaround is either removed or explicitly retained with a current compatibility reason.
• Encoded metadata, empty object keys, the chosen unsigned-input policy, and physical-schema inference remain covered.
• Whole-value Variant projection still returns Spark's required [value, metadata] layout and preserves SQL NULL versus Variant JSON null.
• Unread Variant pruning remains native, and encrypted scans do not lose encryption state.
• No Spark/FFI integration code is removed merely because an Arrow-rs dependency was upgraded.

Additional context

The following nearby code is not covered by the upstream fixes above:

• Millisecond timestamp, unannotated fixed-length binary, UInt64 -> Decimal(20,0), and FixedSizeList -> List normalization are Spark-reader compatibility for physical types outside the canonical Arrow Variant mapping. variant: Support Arrow UInt8/UInt16/UInt32 typed values arrow-rs#10417 does not cover them.
• prepare_variant_for_unshredding, SparkMetadataBuilder, and the Spark-format reconstruction handle canonical UTF-8 versus legacy Spark UTF-16 ordering and Spark byte-format compatibility. Their output-side simplification belongs to Remove Variant UTF-16 output rewriting #5474; an input-side path may remain for historical Spark-written files.
• ffi_schema_for_field substitutes embedded NUL because the Arrow C Data Interface uses NUL-terminated names. That is an ABI limitation, unrelated to Add proptest fuzzing to parquet-variant and implement fixes for findings arrow-rs#10352.
• Exporting the complete Arrow Field, preserving ARROW:extension:name=arrow.parquet.variant, unshredding to [value, metadata], materializing ordinary Binary children for Spark, preserving parent nulls, and retaining explicit fallback gates are required Comet integration.
• Spark-compatible case-insensitive Parquet column matching is a schema-adapter concern tracked by Support Spark-compatible Unicode case-insensitive Parquet field matching #5495, not an Arrow Variant compute workaround and not Remove Variant UTF-16 output rewriting #5474's encoded object-key ordering.

Current dependency: Arrow/Parquet 58.4.0.