Support native casts to and from VariantType

[peterxcli]

What is the problem the feature request solves?

Spark 4.x supports CAST / TRY_CAST between VariantType and a defined set of scalar and nested types, but Comet explicitly rejects every pair involving Variant in CometCast.

Examples that therefore fall back include:

SELECT CAST(v AS STRING), TRY_CAST(v AS ARRAY<INT>) FROM t;
SELECT CAST(id AS VARIANT), CAST(array(1, 2) AS VARIANT) FROM t;

Spark admits casts from Variant through VariantGet.checkDataType and supports the reciprocal cast to Variant for scalar values and supported arrays. Structs and maps use to_variant_object instead. The runtime routes Variant casts through Spark's Variant cast implementation, including generated execution.

Describe the potential solution

Add only Spark-supported Variant pairs to Comet's cast matrix and native cast implementation:

• reuse the extraction/conversion semantics from Support native variant_get and try_variant_get for literal paths and scalar targets #5424 for casts from Variant, including nested targets once supported;
• encode casts to Variant with canonical [value, metadata] storage and the Variant result Field contract from Support Variant-valued native expression output and two-argument variant_get #5425;
• preserve SQL NULL versus Variant JSON null, decimal, binary, date/timestamp, time-zone, overflow, and recursive null behavior;
• match CAST versus TRY_CAST error behavior; and
• keep casts that Spark does not admit unsupported. In particular, maps and structs must use to_variant_object, not a broader Variant cast.

Add focused parity and native-plan tests across supported scalar types, arrays and nested targets, SQL/JSON nulls, invalid conversions, TRY_CAST, aliases, and columns around a Variant-producing result. Spark 3.x compilation and behavior must remain unchanged through the version shim.

Additional context

Related work: #4295, #5407, #5424, #5425, and the dedicated to_variant_object tracker.

Non-goals: comparisons and hash expressions (Spark rejects Variant for both), to_variant_object, parsing JSON, shuffle/spill, C2R, writing, and Python transport.

[peterxcli]

take