Support native variant_get and try_variant_get for literal paths and scalar targets

[peterxcli]

What is the problem the feature request solves?

Phase A of #4295 (#5407) lets Comet carry a direct, top-level Spark VariantType value through an ordinary native Parquet scan. It intentionally leaves Variant expression evaluation on Spark: Comet still rejects a Variant input in CometAttributeReference and rejects non-scan native operators whose schemas contain Variant in CometExecRule.

Common queries therefore still evaluate the extraction in Spark even when v came from a native scan:

SELECT variant_get(v, '$.customer.id', 'bigint') FROM t;
SELECT try_variant_get(v, '$.amount', 'decimal(18,2)') FROM t;

Spark implements both functions with the same VariantGet expression, with different cast-failure behavior. #4295 tracks Variant scans and separately mentions predicate pushdown for variant_get(...) = literal; it does not track native evaluation of these expressions. There is currently no dedicated issue for that work.

Describe the potential solution

Add expression-specific admission and native implementations for variant_get and try_variant_get, initially limited to foldable/literal paths and scalar, non-Variant target types.

The implementation should:

• admit VariantType only for these expressions instead of enabling it in the general datatype or operator gates;
• reuse the whole-value [value, metadata] representation established by feat: project Spark 4 VARIANT columns in native Parquet scans #5407;
• implement Spark's object-key and array-index path syntax for foldable paths;
• return SQL NULL for a missing path, an incompatible container shape, SQL NULL, or Variant JSON null;
• match Spark's strict/try distinction: variant_get raises on a target-type cast failure while try_variant_get returns NULL;
• preserve Spark-compatible decimal, datetime/time-zone, nullability, invalid-path, and error behavior for supported scalar targets; and
• keep dynamic paths, omitted target types / Variant-returning results, nested target types, and every unsupported shape on explicit Spark fallback.

Add focused Spark SQL parity and plan tests covering projection and filtering, object and array paths, missing fields, JSON null versus SQL NULL, invalid paths, cast failure versus try_variant_get, decimals, and timestamp targets. The expression should run natively above an ordinary native Parquet scan for admitted inputs.

Additional context

Related work:

• [FEATURE] Native scan support for VariantType columns (Iceberg + Spark 4.0) #4295 — Variant scan/Iceberg umbrella and the separate predicate-pushdown follow-up
• feat: project Spark 4 VARIANT columns in native Parquet scans #5407 — whole-value ordinary-Parquet projection
• [FEATURE] Shredded Parquet Reader/Writer support for Variant type #3983 — shredded reader/writer support, subfield pruning, and predicate pushdown
• test: add CometExpressionCoverageSuite for runtime expression coverage audit #5031 — expression coverage audit that identifies Variant functions as gaps
• fix: fall back for shredded Variant scans on Spark 4.0 #4084 / [native_iceberg_compat] VariantShreddingSuite test failures with Spark 4.0.0 #2209 — safe fallback for Spark PushVariantIntoScan / VariantStruct

Non-goals: predicate pushdown or subfield pruning, parse_json / to_variant, general Variant casts, C2R, shuffle/spill, Python, writes, and Iceberg.

This boundary avoids pulling Variant-valued expression output and every Spark Variant function into the first native evaluator. Those cases should remain safe fallbacks and can be tracked separately when implemented.

[peterxcli]

take