Support Variant-valued native expression output and two-argument variant_get

[peterxcli]

What is the problem the feature request solves?

Phase A of #4295 (#5407) preserves Spark VariantType for a whole value coming directly from an ordinary native Parquet scan. #5424 separately tracks native variant_get / try_variant_get evaluation for literal paths and scalar, non-Variant targets.

Spark's two-argument forms default the target type to VariantType:

SELECT variant_get(v, '$.customer') FROM t;
SELECT try_variant_get(v, '$.items[0]') FROM t;

These results need more than the physical Arrow Struct<value, metadata> datatype. Their logical identity is the parent Field extension marker ARROW:extension:name = arrow.parquet.variant, which Comet's JVM bridge recognizes explicitly in Utils.fromArrowField.

Comet builds native projects with DataFusion's ProjectionExec. DataFusion's default PhysicalExpr::return_field constructs a Field from only the expression datatype and nullability, so a Variant-producing expression that uses the default would describe its result as an ordinary Struct. DataFusion already provides the needed propagation point: ProjectionExprs::project_schema copies metadata returned by the expression into the aliased output Field.

Comet's generic ScalarFunctionExpr construction currently supplies an unannotated Field::new(...), so merely adding Variant as a function return datatype would still lose the logical marker.

Without an explicit Variant result Field, a computed Variant can lose its logical identity and be imported by Spark as StructType instead of VariantType.

Describe the potential solution

Define the result contract for direct, top-level native expressions that return Variant, using existing Field-aware paths rather than adding another schema layer:

• make each admitted Variant-producing physical expression return the canonical marked Field from return_field, reusing Comet's to_arrow_field;
• return physical storage with exactly two ordinary Binary children in Spark order [value, metadata], preserving parent nullability;
• rebuild an extracted Variant so unused metadata entries are removed, matching Spark's Variant-target behavior;
• allow VariantType through JVM expression/operator admission only for serializers that guarantee this contract, while retaining the general Variant fallback gates;
• rely on DataFusion's existing projection metadata propagation and feat: project Spark 4 VARIANT columns in native Parquet scans #5407's RecordBatch-Field FFI export; and
• use literal-path, two-argument variant_get and try_variant_get as the first end-to-end consumers, sharing the extraction semantics/kernel tracked by Support native variant_get and try_variant_get for literal paths and scalar targets #5424.

Focused tests should verify:

• the native result's parent Field contains arrow.parquet.variant;
• Spark reports the output datatype as VariantType, not StructType;
• children are exactly [value: Binary, metadata: Binary] through aliases and with columns before/after the expression;
• object, array, scalar, Variant JSON null, SQL NULL, and nullable-parent results match Spark; and
• unsupported Variant-producing expressions and downstream consumers still fall back without losing the native scan beneath them.

Additional context

Related work:

• [FEATURE] Native scan support for VariantType columns (Iceberg + Spark 4.0) #4295 — Variant scan/Iceberg umbrella
• feat: project Spark 4 VARIANT columns in native Parquet scans #5407 — whole-value ordinary-Parquet scan projection and Field-preserving FFI export
• Support native variant_get and try_variant_get for literal paths and scalar targets #5424 — scalar-target variant_get / try_variant_get evaluation
• [FEATURE] Shredded Parquet Reader/Writer support for Variant type #3983 — shredded storage, subfield pruning, and predicate pushdown

Non-goals: dynamic paths, nested Variant output, parse_json / to_variant and other Variant producers, C2R, shuffle/spill, Python transport, writes, and Iceberg. Supporting one direct Variant-valued native projection must not enable Variant indiscriminately for other operators.

[peterxcli]

take