Cap Enzyme Lagrangian Hessian batches at eight on every Julia version

[ChrisRackauckas-Claude]

Ignore this PR until it has been reviewed by @ChrisRackauckas.

What changed and why

Generated Enzyme Lagrangian Hessians now use BatchDuplicated with a batch width capped at 8 on every Julia version. In-place and out-of-place evaluations share the batch loop, including zero-padded final batches. The Julia 1.12 row-wise fallback is removed.

The constraint contraction uses an explicit scalar loop instead of BLAS dot: restoring the BLAS contraction produces NaN Hessian entries under nested differentiation on Julia 1.12. The scalar contraction preserves conjugation of the multipliers. Tests cover dense and packed outputs, nonzero constraint curvature, zero and nonzero objective weights, changing parameters/multipliers, dimensions through 183, and full/partial batches.

The out-of-place Hessian regression is marked broken only on Julia 1.10 LTS, as requested, because Enzyme aborts the process on Zen 3/4 targets: EnzymeAD/Enzyme.jl#3574. The marker does not execute the crashing code; in-place LTS tests and all out-of-place cases on Julia 1.11+ remain enabled. Production batching stays capped at eight on every version.

Contraction regression: failing before / passing after

I ran the same committed 134-assertion regression in an isolated checkout with only lagrangian_constraints restored to LinearAlgebra.dot, and with the fix:

timeout 3600 ~/.juliaup/bin/julia +1.12 --startup-file=no --check-bounds=yes \
  --project=lib/OptimizationBase/test/AD \
  lib/OptimizationBase/test/AD/enzyme_lagrangian_hessian.jl

BLAS contraction restored:
Enzyme Lagrangian Hessian | 116 passed, 18 failed, 134 total | 2m02.7s
ERROR: LoadError: Some tests did not pass: 116 passed, 18 failed, 0 errored, 0 broken.

Scalar contraction:
Enzyme Lagrangian Hessian | 134 passed, 134 total | 2m09.7s

Both runs use width-8 batches and Enzyme 0.13.201. Separately, simply removing the old version fallback without fixing the contraction failed 30 of the original 82 assertions on Julia 1.12.

LTS compiler regression: failing before / passing after

I reproduced the upstream LLVM verifier abort locally with Enzyme 0.13.203 and its GPUCompiler target explicitly set to znver3 in an isolated diagnostic dependency checkout. Julia's -C option alone does not override that target. The same environment and command before and after the test gate:

ENZYME_DIAGNOSTIC_CPU=znver3 TMPDIR="$PWD/review-tmp" timeout 3600 \
  ~/.juliaup/bin/julia +1.10.12 --startup-file=no --check-bounds=auto \
  --project=enzyme-lts-gate-env \
  Optimization.jl/lib/OptimizationBase/test/AD/enzyme_lagrangian_hessian.jl

Before: Instruction does not dominate all uses!
        LLVM ERROR: function failed verification (4)
        Process aborted, exit 134
After:  Enzyme Lagrangian Hessian | 62 passed, 6 broken, 68 total | 43.1s
        Exit 0

This quarantines the affected LTS tests; it does not fix the upstream compiler defect. The six markers replace 72 assertions across six dimensions on LTS. The standalone MWE and hardware/version comparisons are in EnzymeAD/Enzyme.jl#3574. Hosted diagnostic evidence is at https://github.com/SciML/Optimization.jl/actions/runs/34698785686; diagnostics are kept separate in #1355.

Local verification of the final test gate

Focused regression (same file as the compiler reproduction above):

TMPDIR="$PWD/review-tmp" timeout 3600 ~/.juliaup/bin/julia +1.11.9 --startup-file=no --check-bounds=yes --project=Optimization.jl/lib/OptimizationBase/test/AD Optimization.jl/lib/OptimizationBase/test/AD/enzyme_lagrangian_hessian.jl
TMPDIR="$PWD/review-tmp" timeout 3600 ~/.juliaup/bin/julia +1.12.7 --startup-file=no --check-bounds=auto --project=latest-enzyme-check Optimization.jl/lib/OptimizationBase/test/AD/enzyme_lagrangian_hessian.jl
TMPDIR="$PWD/review-tmp" timeout 3600 ~/.juliaup/bin/julia +1.13.0 --startup-file=no --check-bounds=auto --project=latest-enzyme-check Optimization.jl/lib/OptimizationBase/test/AD/enzyme_lagrangian_hessian.jl

Julia 1.11.9: Enzyme Lagrangian Hessian | 134 passed / 134 total | 1m37.1s
Julia 1.12.7: Enzyme Lagrangian Hessian | 134 passed / 134 total | 1m27.9s
Julia 1.13.0: Enzyme Lagrangian Hessian | 134 passed / 134 total | 1m26.7s

The full LTS AD group uses the package's official harness and an unmodified Enzyme dependency:

cd Optimization.jl
OPTIMIZATION_TEST_GROUP=AD JULIA_PKG_PRECOMPILE_AUTO=0 TMPDIR="$PWD/../review-tmp" timeout 3600 ~/.juliaup/bin/julia +1.10.12 --project=lib/OptimizationBase -e 'using Pkg; Pkg.test(;coverage=true, julia_args=["--check-bounds=auto", "--compiled-modules=yes", "--depwarn=yes"], force_latest_compatible_version=false, allow_reresolve=true)'

Test Summary: | Pass  Broken  Total     Time
AD            |  866       6    872  9m43.5s
Testing OptimizationBase tests passed
Exit 0

The first full-suite attempt ended with SIGTERM before a test summary; the detached retry above completed successfully.

GROUP=QA JULIA_PKG_PRECOMPILE_AUTO=0 TMPDIR="$PWD/../review-tmp" timeout 3600 ~/.juliaup/bin/julia +1.12.7 --project=. -e 'using Pkg; Pkg.test()'

QA | 21 passed / 21 total | 1m49.5s
Testing Optimization tests passed

Runic, typos, and git diff --check passed for the changed test file. No production code or assertion tolerances changed in the LTS gate commit.

Earlier implementation validation

Before adding the LTS gate, the focused regression passed 134/134 on Julia 1.10.12 (native Zen 2), 1.11.9, 1.12.7, and released 1.13.0. The full OptimizationBase AD group passed 938/938 on Julia 1.12.7 (31m24.9s) and native Julia 1.10.12 (26m26.4s). OptimizationBase QA passed 20 tests with one pre-existing broken test.

CI and remaining limitations

The LTS test gate is locally validated. The new commit's OptimizationBase AD jobs on LTS and stable Julia are still running: https://github.com/SciML/Optimization.jl/actions/runs/34746305607. This remains a draft until reviewed by @ChrisRackauckas.

I read the new ModelingToolkit downstream failure log: it is the same SymbolicAnalysis 0.5 versus ConvexOptimization 0.3 resolver conflict tracked separately, not an Enzyme test failure: https://github.com/SciML/Optimization.jl/actions/runs/34746305501/job/103694764247.

The separate ModelingToolkit resolver fix is #1353; its downstream CI passed 254 tests with one pre-existing broken test. The unrelated NeuralPDE failure reproduces on unmodified master and is tracked at SciML/NeuralPDE.jl#1164. Neither change is included here.

The complete monorepo Everything suite, GPU paths, and 32-bit environments were not run locally. Documentation was not built; this changes no public API, docstrings, or documentation. Eight is the batch cap; optimality across objectives and machines has not been benchmarked.

Earlier investigation: https://chatgpt.com/codex/tasks/01a03a17-ad6f-7131-82fc-d0fd57ea6512

🤖 Generated with Codex CLI 0.153.4 (model: gpt-6-astra; local session: 01a08039-86d4-7683-9601-30fd79501e27).

[ChrisRackauckas-Claude]

Follow-up 11591555ed3ee2b7676bf578ece89287ca8e87f1 fixes the Julia 1.12 NaN failures seen in the first row-wise implementation. The exact Julia 1.12 focused test passed twice (26/26 each); the full Julia 1.11 OptimizationBase AD group passed 821/821, root QA passed 21/21, and Runic/typos/diff checks are clean. The PR body now contains the exact failing-before and passing-after evidence.

[ChrisRackauckas-Claude]

CI audit after #1331 merged:

• The PR head 11591555ed3ee2b7676bf578ece89287ca8e87f1 merges cleanly into current master 9bfa701305b0a5e2913f17d1a195a2ae9c1e5e15.
• I tested that exact local merge with Julia 1.11.9:

timeout 3600 /usr/bin/time -f "wall=%e maxrss_kb=%M exit=%x" \
  ~/.juliaup/bin/julia +1.11 --startup-file=no \
  --project=lib/OptimizationBase/test/AD \
  -e 'using Pkg; Pkg.instantiate(); include("lib/OptimizationBase/test/AD/enzyme_lagrangian_hessian.jl")'

Test Summary:             | Pass  Total   Time
Enzyme Lagrangian Hessian |   26     26  30.9s
wall=71.13 maxrss_kb=976492 exit=0

• All nine red check names are identical to the red checks on merged Cap AutoEnzyme Hessian batches at width 8 #1331. Their logs fail in dependency resolution or unrelated downgraded/downstream environments before exercising this extension: docs (SimpleOptimization/SimpleNonlinearSolve), downstream (LogExpFunctions, OrderedCollections, or Tullio), and downgrade jobs (ModelingToolkitBase, ForwardDiff, or NOMAD result fields).
• The relevant hosted OptimizationBase AD tests pass on Julia 1 and LTS, as do Core and QA. Current master also contains Fix downgrade compat floors for Ipopt, NLopt, and NOMAD sublibraries #1333, which addresses three of the downgrade compat failures, so a branch update/re-run should remove stale failures from this head.

I found no PR-specific failing check. The PR remains draft and still requires review by @ChrisRackauckas.

[ChrisRackauckas]

This is too extreme: it completely removes all batch duplicated. Just like the hessians and such, shouldn't we limit it to like 8?

[ChrisRackauckas-Claude]

Addressed the review comment: Lagrangian Hessians now use the same width-8 Enzyme FoR batching as objective Hessians (_hessian_batch_width / #1331), instead of one seed at a time.

Julia 1.12 still falls back to row-wise Duplicated seeds via _lag_hessian_batch_width, because BatchDuplicated through the Lagrangian currently produces NaN Hessian rows there. On Julia ≤1.11 the batch width is capped at 8 as requested.

Verified: Enzyme Lagrangian Hessian 26/26 on Julia 1.11.9 (width 8) and Julia 1.12.7 (row-wise fallback).

[ChrisRackauckas-Claude]

Follow-up addressing the sol review:

• Fixed OOP lag_h to use lagrangian_oop (cons(x,p)) and return a dense symmetric matrix (matching OOP hess).
• Narrowed the row-wise fallback to Julia 1.12 ≤ VERSION < 1.13 with a TODO to re-check on 1.13+.
• Expanded tests: nonzero clnlbeam points (constraint Hessians contribute), quadratic σ/μ cases including σ=0, and OOP coverage at n=9 and n=17.

Enzyme Lagrangian Hessian: 82/82 on Julia 1.11.9 and 1.12.7.

[ChrisRackauckas-Claude]

Implemented in 9a8a5c7: Lagrangian Hessians now use batches capped at 8 on every Julia version, with no row-wise fallback. The in-place and out-of-place paths share the same batch loop.

The NaNs came back when I restored the BLAS constraint contraction in an isolated checkout: 116 passed / 18 failed. The scalar contraction passes all 134 assertions. Focused tests passed on Julia 1.10.12, 1.11.9, 1.12.7, and 1.13.0-rc4; coverage checks also pass, including a separate Enzyme 0.13.203 environment. The full OptimizationBase AD group passed 938/938, root QA passed 21/21, and OptimizationBase QA passed with its existing one broken check unchanged. Runic, typos, and diff checks passed locally.

The PR description now contains the final implementation and exact verification results. CI is running; the PR remains draft.

AI agent: Codex CLI 0.153.4; model: gpt-6-astra; local session: 01a08039-86d4-7683-9601-30fd79501e27.

[ChrisRackauckas-Claude]

The batch cap is now 8 for every Julia version, with no version-specific width-1 fallback. Local focused tests pass 134/134 on Julia 1.10, 1.11, 1.12, and 1.13. The complete OptimizationBase AD suite also passes locally on Julia 1.10.12 (938/938, 26m26.4s with the exact CI command) and Julia 1.12.7 (938/938, 31m24.9s). Full commands and before/after evidence are in the PR body.

This is not ready to merge: the LTS CI job hit an Enzyme/LLVM verifier error compiling the out-of-place Hessian test. I could not reproduce that failure with the same Julia/Enzyme versions, CI flags, multiple CPU targets, fresh processes, or compiled modules disabled. No speculative workaround or test weakening was pushed. GitHub denied my diagnostic rerun request because this account lacks repository admin rights; a maintainer rerun of the failed LTS job is needed to establish whether it repeats.

The other observed failures reproduce on master and are tracked separately: ModelingToolkit's incompatible SymbolicAnalysis requirements have a validated draft fix; NeuralPDE's PDE IV test fails the same two unchanged assertions locally on clean master. DiffEqFlux's downstream job is still running.

Links:

• LTS failure: https://github.com/SciML/Optimization.jl/actions/runs/34684825479/job/103530021677
• Separate resolver fix: Allow SymbolicAnalysis 0.5 in ConvexOptimization #1353
• NeuralPDE failure investigation: NNPDE1 PDE IV test is underdetermined on the full square (standalone counterexample) NeuralPDE.jl#1164

🤖 Generated with Codex CLI 0.153.4 (model: gpt-6-astra; local session: 01a08039-86d4-7683-9601-30fd79501e27).

[ChrisRackauckas-Claude]

The LTS rerun failed again on the unchanged commit, with the same Julia 1.10.12 / Enzyme 0.13.203 / Enzyme_jll 0.0.293+0 combination and the same Instruction does not dominate all uses! / LLVM ERROR: function failed verification (4) error in check_oop_quadratic at enzyme_lagrangian_hessian.jl:106.

This confirms a repeated CI failure; the successful local runs do not resolve it. The PR remains a draft and is not ready to merge. The next investigation needs to reproduce the CI-specific trigger and validate a fix against it.

Links:

• Failed rerun: https://github.com/SciML/Optimization.jl/actions/runs/34684825479/job/103554419756

🤖 Generated with Codex CLI 0.153.4 (model: gpt-6-astra; local session: 01a08039-86d4-7683-9601-30fd79501e27).

[ChrisRackauckas-Claude]

Added the requested Julia 1.10 LTS broken markers for the six out-of-place Hessian dimensions, with a source comment linking EnzymeAD/Enzyme.jl#3574. The markers do not execute the compiler-aborting cases. In-place LTS tests, all tests on Julia 1.11+, and the production batch cap of eight remain unchanged.

Local validation:

• Forced Zen 3 target with Enzyme 0.13.203: before, LLVM verifier abort / exit 134; after, 62 passed and 6 broken / exit 0.
• Full official LTS AD group with coverage: 866 passed, 6 broken, 872 total in 9m43.5s / exit 0.
• Focused Julia 1.11.9, 1.12.7, and released 1.13.0: 134/134 each.
• Root QA: 21/21. Runic, typos, and whitespace checks passed.

The first full LTS attempt received SIGTERM before a result; a detached retry completed successfully. The PR body now distinguishes the authorized LTS exclusion from fixing the upstream compiler defect. CI is pending.

🤖 Generated with Codex CLI 0.153.4 (model: gpt-6-astra; local session: 01a08039-86d4-7683-9601-30fd79501e27).