Draft: Add Spark 4.1 support

[billpratt]

Summary

Draft implementation for Spark 4.1 support, tracked by #286.

• Adds the Spark 4.1 Deequ coordinate: com.amazon.deequ:deequ:2.0.18-spark-4.1
• Updates Scala/Py4J collection conversion for Scala 2.13 compatibility
• Adds a Spark 4.1 CI entry, focused configuration coverage, and an optional Dockerfile
• Adds Python-versioned pandas/NumPy constraints for the Spark 4.1 runtime path

Coordination

This draft is intended to coordinate with #283. Its final form should adopt #283's Spark 3.5 baseline and resolve overlapping dependency, CI, and documentation changes after that PR merges.

Follow-up before merge

• Regenerate poetry.lock from a public-PyPI-capable environment
• Run clean Spark 3.5 and Spark 4.1 CI after dependency resolution
• Rebase or port the final implementation after feat: upgrade Deequ to 2.0.21, Spark 3.5 only (drops Spark 3.1–3.4) #283
• Confirm whether maintainers want the Spark 4.1 Dockerfile

Validation

A local full test run against PySpark 4.1.2, Java 17, Scala 2.13, and deequ:2.0.18-spark-4.1 completed successfully.

[github-actions]

No issues found.

---

Generated by AI (model: us.anthropic.claude-opus-4-8, prompt: 295226dd) — may not be fully accurate. Reply if this doesn't help.
Reviewed by Shadow · github.com/sudsali/shadow

[github-actions]

---

Generated by AI (model: us.anthropic.claude-opus-4-8, prompt: 295226dd) — may not be fully accurate. Reply if this doesn't help.
Reviewed by Shadow · github.com/sudsali/shadow

Additional feedback:

tests/test_checks.py:677 — DESIGN: The one-ULP-tolerance rationale applies to all mutual-information results, but only one of three MI assertions in test_hasMutualInformation was loosened; the remaining x == 1.0986122886681096 comparisons at lines 677 and 682 may still be flaky on Spark 4.1.

tests/test_checks.py:677 lambda x: x == 1.0986122886681096, line 682 same pattern, line 686 (loosened by diff to math.isclose(x, 0.7324081924454064, rel_tol=0.0, abs_tol=1e-15)). The diff comment states 'Spark 3.5 and 4.1 can differ by one ULP in mutual information results' — this rationale applies equally to all three MI assertions.

Refutation trail (why this survived the Critic's disprove pass)

Hypothesis (Investigator): The incremental diff loosens only the ("c", "b") mutual-information assertion to a tolerance-based check, but the sibling exact-equality assertions in the same test (x == 1.0986122886681096 at lines 675-684) are left as strict ==, so they remain vulnerable to the same one-ULP Spark 4.1 divergence the comment cites.

Disprove attempt (Critic): Read tests/test_checks.py lines 674-688 at PR-head SHA. The test_hasMutualInformation body contains three assertions: two comparing against 1.0986122886681096 via lambda x: x == ... and one against 0.7324081924454064. The incremental diff only wraps the third (0.7324...) in math.isclose(..., abs_tol=1e-15); the two 1.0986... comparisons remain exact ==. If Spark 4.1 can differ by one ULP for MI generally (per the added comment), those two assertions can still flake on Spark 4.1.

The Critic's default verdict is OVERTURNED. UPHELD findings are those it tried — and failed — to refute.

[github-actions]

No issues found.

---

Generated by AI (model: us.anthropic.claude-opus-4-8, prompt: 295226dd) — may not be fully accurate. Reply if this doesn't help.
Reviewed by Shadow · github.com/sudsali/shadow

[github-actions]

No issues found.

---

Generated by AI (model: us.anthropic.claude-opus-4-8, prompt: 295226dd) — may not be fully accurate. Reply if this doesn't help.
Reviewed by Shadow · github.com/sudsali/shadow

[github-actions]

No issues found.

---

Generated by AI (model: us.anthropic.claude-opus-4-8, prompt: 295226dd) — may not be fully accurate. Reply if this doesn't help.
Reviewed by Shadow · github.com/sudsali/shadow