[EPIC] Native Variant Support

[peterxcli]

What is the problem the feature request solves?

Spark 4 introduced VariantType for semi-structured values. Comet currently has isolated Variant fallbacks and partial scan work, but no single roadmap for carrying Variant through an otherwise native query:

Parquet / Iceberg
  -> Arrow Field + array
  -> native expressions and operators
  -> Arrow C Data Interface
  -> Spark ColumnVector / UnsafeRow / Python
  -> shuffle, spill, and writes

This epic coordinates the existing Variant issues into an incremental, fail-closed implementation plan. It does not replace the focused trackers: #4295 remains the scan umbrella and #3983 remains the shredded-Parquet tracker.

Why this matters

• Keep otherwise native Spark 4 plans native when they project, inspect, transform, transport, or write semi-structured data.
• Avoid unnecessary JVM/native transitions and repeated Variant decoding and encoding.
• Establish one Spark-compatible representation across Parquet, Arrow, FFI, native operators, UnsafeRow, Python Arrow, shuffle, and spill.
• Unlock useful JSON-like workloads incrementally: whole-value projection first, then extraction and construction, followed by wider transport and storage.
• Make shredded subfield pruning and predicate pushdown an optimization built on a correct whole-value path rather than a separate representation.
• Preserve explicit Spark fallback for every boundary that has not yet been audited, with Spark 3.x behavior unchanged.

Canonical representation

The contract used by every tracker in this epic should be:

• Spark logical type: VariantType.
• Arrow physical storage: Struct<value: Binary, metadata: Binary>.
• Spark output child order: exactly [value, metadata]. Spark's Arrow conversion creates that order, and ColumnVector.getVariant consumes child 0 as value and child 1 as metadata.
• Logical identity: ARROW:extension:name=arrow.parquet.variant on the individual outer Arrow Field representing the Variant column. It is not a property of DataType::Struct, and an arbitrary struct named value/metadata must never be inferred as Variant. Arrow-rs recognizes the marker when constructing a VariantArray.

Describe the potential solution

Implementation principles

1. Reuse the canonical marked Arrow Field and whole-value normalization established by #5407.
2. Normalize Parquet output once. Use Arrow-rs unshred_variant, remove typed_value, and export ordinary Binary children in Spark order.
3. Preserve the full Arrow Field—not only its DataType—at every schema-producing and FFI boundary. This also matters for expressions because DataFusion's default PhysicalExpr::return_field derives a Field from the DataType unless an expression supplies richer metadata.
4. Keep Variant admission explicit per operator. Serialization of the type is not evidence that an expression, C2R path, shuffle, writer, or Python boundary supports its semantics.
5. Use Spark-version shims so Spark 3.x neither references Variant classes nor changes behavior.
6. Deliver each tracker with value-parity, plan/admission, schema/vector-layout, and fallback tests appropriate to that boundary.

Implementation order

Phase 0 — safety baseline and representation foundation

• #5377 — prune an unread Variant column without decoding it.
• #5407 — project a top-level whole Variant value from ordinary Parquet; establish protobuf identity, canonical Arrow Field construction, one-time unshredding/normalization, Field-aware FFI export, and Spark getVariant bridging.

All later phases should reuse #5407's representation instead of adding a second Variant encoding.

Phase 1 — first native read/compute vertical slice

• #5429 — is_variant_null and Spark 4.2 is_valid_variant. This can proceed after feat: project Spark 4 VARIANT columns in native Parquet scans #5407 and validates basic decoding, SQL NULL, and Variant JSON null behavior.
• #5424 — literal-path variant_get/try_variant_get with scalar targets. This establishes the path parser, strict/try errors, and scalar conversion semantics.
• #5425 — Variant-valued expression output and two-argument variant_get. This depends on feat: project Spark 4 VARIANT columns in native Parquet scans #5407's Field contract and should reuse Support native variant_get and try_variant_get for literal paths and scalar targets #5424's extraction kernel.

Phase 2 — parallel scan, storage-shape, and row consumers

Once #5407 is merged, these can proceed largely in parallel:

• #4295 — complete whole-value Variant projection for Iceberg, including tables with legal primitive equality-delete keys.
• #5435 — project Variant nested in struct, array, and map columns.
• #5433 — write top-level whole-value Variant to ordinary Parquet.
• #5436 — encode Variant in native columnar-to-row/UnsafeRow. Spark uses a dedicated length + value + metadata payload, not ordinary Struct row encoding.

Phase 3 — expression completion and Variant producers

• #5426 — dynamic paths and nested targets for variant_get/try_variant_get, after Support native variant_get and try_variant_get for literal paths and scalar targets #5424 and Support Variant-valued native expression output and two-argument variant_get #5425.
• #5430 — casts to and from Variant, reusing the extraction/conversion and Variant-output contracts from Support native variant_get and try_variant_get for literal paths and scalar targets #5424/Support Variant-valued native expression output and two-argument variant_get #5425.
• #5428 — parse_json/try_parse_json, after Variant-valued output is supported.
• #5431 — to_variant_object, preferably reusing the scalar-to-Variant encoder from Support native casts to and from VariantType #5430.
• #5427 — schema_of_variant/schema_of_variant_agg, reusing the decoder established by Support native variant_get and try_variant_get for literal paths and scalar targets #5424.
• #5432 — variant_explode/variant_explode_outer, after Variant-valued output Fields are supported by Support Variant-valued native expression output and two-argument variant_get #5425.

Independent items in this phase may be implemented in parallel once their listed prerequisites land.

Phase 4 — wider transport

• #5434 — carry Variant through Comet shuffle and spill. Arrow-native transport can build on feat: project Spark 4 VARIANT columns in native Parquet scans #5407; any JVM row path should reuse the UnsafeRow codec from Support VariantType in native columnar-to-row conversion #5436.
• #5437 — support top-level Variant in MapInArrow/MapInPandas, after feat: project Spark 4 VARIANT columns in native Parquet scans #5407 and Support Variant-valued native expression output and two-argument variant_get #5425 establish input and output Field identity. Spark's Python conversion explicitly preserves the Variant extension Field.

Phase 5 — shredded storage and optimization

• #3983 — implement in dependency-sized subphases:
  • shredded reader/unshredding on top of feat: project Spark 4 VARIANT columns in native Parquet scans #5407;
  • shredded writer on top of Support whole-value VariantType in native Parquet writes #5433;
  • subfield pruning and predicate pushdown on top of Support native variant_get and try_variant_get for literal paths and scalar targets #5424/Complete native variant_get and try_variant_get support for dynamic paths and nested targets #5426.

#3983 should not be one serial blocker for whole-value scans, expressions, or transport.

Difficult points

1. Logical identity lives on an Arrow Field. The extension marker can be lost whenever code reconstructs a schema from only an array DataType. Every projection, expression result, batch rewrite, FFI export, shuffle schema, Python schema, and writer boundary must preserve it.
2. Physical Parquet input is not one fixed Struct shape. Children can arrive in another order, may include typed_value, and metadata may be dictionary encoded. Binary children may be Binary, LargeBinary, or BinaryView. Parent nulls, field name, nullability, and metadata must survive normalization.
3. Spark and Arrow use different object-key ordering. Spark lookup follows Java UTF-16 ordering, while Arrow validation/encoding uses UTF-8 ordering. Values may need normalization without changing metadata field IDs or SQL-visible content.
4. There are two kinds of null. SQL NULL is the parent Struct null; Variant JSON null is a valid non-null Variant payload. Operators, generators, casts, writers, and transports must preserve that distinction.
5. Expression semantics are broader than decoding bytes. variant_get, casts, and parsers need Spark-compatible path syntax, strict versus try behavior, target-type conversion, decimal overflow, timestamp/time-zone behavior, malformed-input errors, and size limits.
6. Each boundary has a different physical contract. Arrow needs the extension Field, Spark vectors need [value, metadata], UnsafeRow uses a dedicated packed payload, Python Arrow needs child metadata, and Parquet needs the Variant logical annotation.
7. Nested Variant is recursive schema work. Struct, list, and map offsets, child nullability, Field metadata, pruning, and rebuilding must all remain aligned.
8. Fallback admission is part of correctness. Until a boundary is implemented and tested, Variant must remain on Spark rather than leak into a native operator as an ordinary Struct.
9. Version boundaries matter. Spark 3.x has no VariantType; Spark 4.0, 4.1, and 4.2 expose different functions. Code and tests need version-specific shims rather than reflection spread through shared paths.
10. Iceberg adds schema IDs and spec constraints. Whole-value projection must preserve Iceberg field identity and delete semantics in addition to the Arrow/Parquet contract.

Additional context

Upstream semantic boundaries

This epic intentionally does not add trackers for behavior that upstream does not define:

• Spark does not define general Variant ordering: PhysicalVariantType.ordering throws.
• Spark hash expressions reject schemas containing Variant. Native Variant comparisons and hashing are therefore not goals until Spark defines them.
• Iceberg identifier fields must be primitive, and equality-delete columns inherit those restrictions (identifier fields, equality deletes). Variant itself is not a valid equality-delete key. Reading Variant alongside legal primitive-key equality deletes remains part of [FEATURE] Native scan support for VariantType columns (Iceberg + Spark 4.0) #4295.
• Functions introduced only after the supported Spark releases should receive their own tracker when the corresponding Spark profile exists, rather than being speculatively implemented here.

Epic-level definition of done

• Supported Spark 4 profiles preserve Variant logical identity and exact [value, metadata] layout through every completed boundary.
• Object, array, scalar, Variant JSON null, SQL NULL, nullable parents, Unicode keys, and malformed values have Spark parity where applicable.
• Whole-value, shredded, and nested inputs produce the same logical values.
• Supported plans remain native; unsupported plans have focused tests proving explicit Spark fallback.
• Spark 3.x compilation and behavior remain unchanged.
• No code path identifies Variant from an unmarked ordinary Struct.
• Each completed issue updates this checklist and documents any remaining fallback boundary.

Historical fallback evidence: #2209.