Skip to content

feat(antarctica): n=3 physical-prior inversion on the ISMIP7 simulation framework - #3

Closed
hoffmaao wants to merge 66 commits into
mainfrom
antarctica-n3
Closed

feat(antarctica): n=3 physical-prior inversion on the ISMIP7 simulation framework#3
hoffmaao wants to merge 66 commits into
mainfrom
antarctica-n3

Conversation

@hoffmaao

@hoffmaao hoffmaao commented Jul 20, 2026

Copy link
Copy Markdown
Member

Intent

Push the n=3 physical-prior ice-sheet inversion work to my personal fork (hoffmaao/Antarctica, a fork of icepack/ismip7) so I can merge it with the icepack ismip7 upstream. The branch antarctica-n3 is the standard-Glen n=3 parallel line to the n=4 antarctica branch. Since the earlier framework commits (through 6da0a92) were already gated as PR #3, the NEW work to focus review on is the last three commits: (1) 6ae0255 - the core n=3 physical-prior inversion method ported from mismip_time-dependent-da (Recinos/fenics_ice): the fluidity control is now phi=log(A/A_prior) on a thermomechanical prior mean (icepack2_tools/thermo_model.py, a depth-averaged enthalpy/Stefan solve adapted for Antarctica with a spatially-varying tas surface-temperature BC and an under-relaxed fixed-velocity Picard loop), the friction control stays theta=log(C/C_w0) on the balance anchor, and the regularization is the fenics_ice Whittle-Matern form gamma*(theta^2+L^2|grad theta|^2) (icepack2_tools/prior.py) at a physical gamma=1e4 - this regularizes the DEVIATION from a physical prior, not the amplitude, which fixed the n=3 blow-up (controls went from +-36 to physical p99~3-4); the forward (simulation.py) loads A_prior from the MAP and rebuilds A=A_prior*exp(phi), with restart checkpoints carrying it; (2) 8c5f2e2 - a guard so a first-iteration solve failure raises instead of silently writing a theta=phi=0 garbage MAP (this happened at -n 16 where the tiny 32km mesh over-decomposed MUMPS; the fix plus running small meshes at fewer ranks resolved it); (3) 87b3d4f - N3_FRAMEWORK.md doc + run recipe. Validated end-to-end at 32km: thermo prior converges (T 221-272K), the converged inversion has physical controls and n=4-comparable misfit 4815, and the forward reproduces it (setup_model initial misfit 6191 vs the old broken 3.5e5). Deliberate choices: the two files intentionally duplicate small map_n_tag/N_FLOW constants with keep-in-sync notes (matching the existing pattern); thermo params (shear_amp=30, duval=181) are the mismip values and slightly over-soften a few margins (a noted, non-blocking tuning item); A_prior is computed inside the inversion on its own mesh to avoid cross-mesh dof-copy issues. Remotes are now origin=hoffmaao/Antarctica (my fork), upstream=icepack/ismip7.

What Changed

  • Adds the n=3 standard-Glen physical-prior inversion method: the fluidity control becomes phi = log(A/A_prior) on a thermomechanical prior mean (icepack2_tools/thermo_model.py, a depth-averaged enthalpy solve with spatially-varying tas surface temperature and an under-relaxed Picard loop), with fenics_ice-style Whittle-Matern regularization (icepack2_tools/prior.py) at gamma=1e4 penalizing deviation from the prior rather than amplitude; the forward (antarctica/scripts/simulation.py) loads A_prior from the MAP checkpoint and rebuilds A = A_prior*exp(phi), with restart checkpoints carrying it.
  • Brings in the full ISMIP7 Antarctica pipeline this method sits on: mesh/forcing/preflight tooling, the inversion and forward simulation scripts, icepack2_tools (dual friction, forcing readers, regridding, coupled transport), gated CTRL/RC launchers, the shared core-experiment runner, and per-core reports covering 9/11 core experiments at 32 km (N3_FRAMEWORK.md and COMPOSITE_RHEOLOGY.md document the method and run recipe).
  • Hardens failure modes: a first-iteration inversion solve failure now raises instead of silently writing a theta=phi=0 MAP, the ISMIP7_MAP_CLIP default is n-aware (10 for n=3, 6 otherwise), and the thermo-prior non-convergence and legacy constant-baseline fallback paths warn loudly (the latter three applied as review auto-fixes in the gate pipeline).

Risk Assessment

✅ Low: The fix commit 63f9dbc is small and mechanical, implements all four requested round-1 fixes exactly as instructed (n-aware clip default with doc sync, loud Picard non-convergence warning, legacy-baseline log line, deduplicated anchor) without touching numerical method defaults, and introduces no new issues.</risk_rationale>
Review-only pass per pipeline instructions (no tests run): diffed 87b3d4f..63f9dbc, confirmed the worktree is clean at HEAD, verified each fix against the round-1 instructions (n-aware clip logic and N_FLOW_DEFAULT resolution, for/else semantics of the Picard warning, symmetry of the fallback print, hoisted weertman_anchor with branch-specific prints intact), and checked the new lines for fresh edge cases (float parsing of ISMIP7_N_FLOW, float(A0) on a Constant, dA scope in the for/else).

Testing

Ran a purpose-built synthetic validation harness in the Firedrake 2026 venv covering all new modules and script changes from the three commits: thermo fluidity prior convergence and physicality (with figure), Whittle-Matern prior analytic and gradient correctness, the tas climatology reader against a fake ISMIP7 data tree, MAP fluidity_prior checkpoint round-trip with legacy fallback, and import/constant-sync smoke of both scripts - all 12 checks passed; the real 32 km data run was not reproducible in the gated worktree and the fail-loud guard was verified by inspection.

  • Evidence: Thermo fluidity prior on synthetic ice stream (|u|, T_srf, A_prior scalar/varying) (local file: /tmp/no-mistakes-evidence/01KYEAN0Q31W85QW09Z68D94J0/thermo_prior_synthetic.png)
Evidence: Full validation log (12/12 checks passed)

[PASS] scalar-T_srf prior in physical band (A in [5.28, 129.51]) [PASS] varying-T_srf prior in physical band (A in [1.00, 82.96]) [PASS] colder surface -> stiffer ice (lower A_prior) (flank mean A: cold third 1.00 < warm third 21.68) [PASS] R(theta) matches analytic sine-field energy (rel err 6.18e-04) [PASS] gradient form == Gateaux derivative of energy (rel 1.1e-10) [PASS] tas reader: monthly mean + nearest-neighbour sampling (max err 0.500 K) [PASS] tas reader: seasonal cycle averaged out (T in [230.0, 270.0] K) [PASS] tas reader: ocean/NODATA masked to fill_K=260 [PASS] checkpoint carries A_prior; forward rebuild A=A_prior*exp(phi) (max abs err 1.42e-14) [PASS] missing fluidity_prior raises (legacy-MAP fallback path) (RuntimeError) [PASS] N_FLOW/A4_FACTOR defaults in sync (inversion vs forward) (n=3.0, a4=1.0) [PASS] map_n_tag: n=3 default tags MAPs '_n3'; n=4 keeps legacy untagged name [PASS] Whittle-Matern gamma defaults are the physical 1e4 RESULT: ALL CHECKS PASSED

/home/andrew/venv-firedrake-2026/lib/python3.10/site-packages/firedrake/petsc.py:96: UserWarning: No external package for parmetis, ptscotch, chaco found, defaulting to PETSc simple partitioner. This may not be optimal.
  warn(
========================================================================
1. Thermo fluidity prior on a synthetic ice stream (400 x 100 km)
========================================================================
  synthetic fields: H [700,2200] m, |u| [11,840] m/yr, C [0.0030,0.0406] MPa (m/yr)^-1/3
  physical A band: rate_factor(220K)=0.0714 .. rate_factor(Tm)*(1+duval*w_max)=447 MPa^-3 yr^-1

-- 1a. scalar T_srf = 253.15 K (idealized MISMIP-style path)
  Thermo fluidity prior (fixed u; kappa=4.0, q_geo=50.0 mW/m2, shear_amp=30.0, duval=181.25):
    it  1: A [  13.8,    219.1]  dA=9.96e+00
    it  2: A [  10.6,    189.0]  dA=1.37e-01
    it  3: A [   8.5,    167.8]  dA=1.12e-01
    it  4: A [   7.2,    153.9]  dA=8.28e-02
    it  5: A [   6.4,    144.9]  dA=5.83e-02
    it  6: A [   6.0,    139.2]  dA=3.99e-02
    it  7: A [   5.7,    135.5]  dA=2.67e-02
    it  8: A [   5.5,    133.1]  dA=1.74e-02
    it  9: A [   5.4,    131.6]  dA=1.11e-02
    it 10: A [   5.4,    130.7]  dA=6.95e-03
    it 11: A [   5.3,    130.1]  dA=4.24e-03
    it 12: A [   5.3,    129.8]  dA=2.53e-03
    it 13: A [   5.3,    129.6]  dA=1.47e-03
    it 14: A [   5.3,    129.5]  dA=8.38e-04
    thermo prior converged.
[PASS] scalar-T_srf prior in physical band  (A in [5.28, 129.51])

-- 1b. spatially-varying T_srf (223 K inland -> 268 K margin), tas-style
  Thermo fluidity prior (fixed u; kappa=4.0, q_geo=50.0 mW/m2, shear_amp=30.0, duval=181.25):
    it  1: A [  12.4,     29.9]  dA=4.97e-01
    it  2: A [   7.8,     36.7]  dA=2.74e-01
    it  3: A [   5.1,     44.3]  dA=3.05e-01
    it  4: A [   3.5,     54.8]  dA=2.76e-01
    it  5: A [   2.5,     65.9]  dA=2.20e-01
    it  6: A [   1.9,     74.4]  dA=1.67e-01
    it  7: A [   1.5,     80.0]  dA=1.10e-01
    it  8: A [   1.3,     83.1]  dA=6.42e-02
    it  9: A [   1.2,     84.4]  dA=3.22e-02
    it 10: A [   1.1,     84.6]  dA=1.56e-02
    it 11: A [   1.1,     84.4]  dA=6.65e-03
    it 12: A [   1.0,     84.0]  dA=4.47e-03
    it 13: A [   1.0,     83.7]  dA=4.58e-03
    it 14: A [   1.0,     83.4]  dA=3.99e-03
    it 15: A [   1.0,     83.2]  dA=2.96e-03
    it 16: A [   1.0,     83.1]  dA=1.99e-03
    it 17: A [   1.0,     83.0]  dA=1.26e-03
    it 18: A [   1.0,     83.0]  dA=7.44e-04
    thermo prior converged.
[PASS] varying-T_srf prior in physical band  (A in [1.00, 82.96])
[PASS] colder surface -> stiffer ice (lower A_prior)  (flank mean A: cold third 1.00 < warm third 21.68)
  figure saved: /tmp/no-mistakes-evidence/01KYEAN0Q31W85QW09Z68D94J0/thermo_prior_synthetic.png

========================================================================
2. Whittle-Matern prior (icepack2_tools/prior.py)
========================================================================
[PASS] R(theta) matches analytic sine-field energy  (assembled 1804.0503 vs exact 1805.1652, rel err 6.18e-04)
  gamma=1e4 prior energy at unit-amplitude theta: 1804.1 (mass term share 69.2% - the term the old gradient-only form lacked)
[PASS] gradient form == Gateaux derivative of energy  (<dR,v>=6.3388434867e+01 vs central-FD 6.3388434859e+01, rel 1.1e-10)

========================================================================
3. load_mean_annual_surface_temperature vs synthetic ISMIP7 tas tree
========================================================================
[PASS] tas reader: monthly mean + nearest-neighbour sampling  (max |T - expected ramp| = 0.500 K on 10 km source grid)
[PASS] tas reader: seasonal cycle averaged out  (loaded T in [230.0, 270.0] K)
[PASS] tas reader: ocean/NODATA masked to fill_K=260  (ocean-strip values all 260.0 K)

========================================================================
4. MAP checkpoint round-trip: fluidity_prior + A = A_prior*exp(phi)
========================================================================
[PASS] checkpoint carries A_prior; forward rebuild A=A_prior*exp(phi)  (max abs err 1.42e-14)
[PASS] missing fluidity_prior raises (legacy-MAP fallback path)  (RuntimeError)

========================================================================
5. Import smoke + keep-in-sync constants of the changed scripts
========================================================================
  (colorcet stubbed - not in venv-firedrake-2026; pre-existing import, colormaps only)
  imported inversion_icepack2.py OK
  imported simulation.py OK
[PASS] N_FLOW/A4_FACTOR defaults in sync (inversion vs forward)  (n=3.0, a4=1.0)
[PASS] map_n_tag: n=3 default tags MAPs '_n3' in both scripts
[PASS] map_n_tag: n=4 override keeps legacy untagged name
[PASS] Whittle-Matern gamma defaults are the physical 1e4  (gamma_theta=10000, gamma_phi=10000, L=7500)

RESULT: ALL CHECKS PASSED
Evidence: Synthetic validation harness (reproducible)
#!/usr/bin/env python3
"""Synthetic end-to-end validation of the n=3 physical-prior inversion pieces
(branch antarctica-n3, commits 6ae0255..63f9dbc).

The real 32 km Antarctic datasets are per-machine and absent from the gated
worktree, so this exercises the NEW code paths on a synthetic ice-stream
configuration with physical magnitudes:

  1. icepack2_tools.thermo_model.compute_fluidity_prior - fixed-velocity
     enthalpy Picard loop, scalar T_srf (idealized path) AND spatially-varying
     T_srf Function (the Antarctica adaptation), with a figure artifact.
  2. icepack2_tools.prior - Whittle-Matern energy vs the analytic value for a
     sine field, and gradient-form consistency vs central finite differences.
  3. icepack2_tools.forcing.load_mean_annual_surface_temperature - new tas
     reader against a synthetic ISMIP7-layout netcdf tree (monthly mean,
     ocean/NODATA masking, fill value).
  4. MAP checkpoint round-trip: save fluidity_prior the way the inversion
     does, reload the way simulation.setup_model does, rebuild
     A = A_prior * exp(phi).
  5. Import smoke of the changed scripts (inversion_icepack2, simulation) and
     the n-tag/clip constants they must keep in sync.
"""
import os
import sys
import numpy as np

REPO = os.environ["N3_REPO"]
EVID = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, REPO)

import firedrake as fd
from firedrake import (Constant, Function, FunctionSpace, VectorFunctionSpace,
                       SpatialCoordinate, sqrt, inner, grad, dx, assemble,
                       max_value, exp as fexp, sin, pi)

failures = []


def check(name, ok, detail=""):
    tag = "PASS" if ok else "FAIL"
    print(f"[{tag}] {name}" + (f"  ({detail})" if detail else ""))
    if not ok:
        failures.append(name)


# ----------------------------------------------------------------------------
print("=" * 72)
print("1. Thermo fluidity prior on a synthetic ice stream (400 x 100 km)")
print("=" * 72)
from icepack2_tools.thermo_model import compute_fluidity_prior, DEFAULTS
from icepack2_tools.dual_friction import weertman_anchor
from icepack.models.viscosity import rate_factor

Lx, Ly = 400e3, 100e3
mesh = fd.RectangleMesh(64, 16, Lx, Ly)
Q = FunctionSpace(mesh, "CG", 1)
V = VectorFunctionSpace(mesh, "CG", 1)
x, y = SpatialCoordinate(mesh)

# Grounded ice stream: thick inland ice thinning seaward, fast central trunk.
H = Function(Q, name="thickness").interpolate(2200.0 - 1500.0 * x / Lx)
b = Function(Q, name="bed").interpolate(100.0 - 300.0 * x / Lx)
s = Function(Q, name="surface").interpolate(b + H)
u = Function(V, name="velocity").interpolate(fd.as_vector([
    (30.0 + 800.0 * (x / Lx) ** 2) * fexp(-((y - Ly / 2) / (25e3)) ** 2)
    + 10.0, Constant(0.0)]))
acc = Function(Q, name="acc").interpolate(Constant(0.3))
C = weertman_anchor(H, s, u, 3.0, Q)
speed = Function(Q).interpolate(sqrt(inner(u, u)))
print(f"  synthetic fields: H [{H.dat.data_ro.min():.0f},{H.dat.data_ro.max():.0f}] m, "
      f"|u| [{speed.dat.data_ro.min():.0f},{speed.dat.data_ro.max():.0f}] m/yr, "
      f"C [{C.dat.data_ro.min():.4f},{C.dat.data_ro.max():.4f}] MPa (m/yr)^-1/3")

# Physical reference band: rate_factor at the coldest/warmest plausible ice,
# times the max Duval melt enhancement used by the model.
A_cold = float(rate_factor(Constant(220.0)))
A_warm = float(rate_factor(Constant(273.15))) * (1 + DEFAULTS["duval"] * DEFAULTS["w_max"])
print(f"  physical A band: rate_factor(220K)={A_cold:.3g} .. "
      f"rate_factor(Tm)*(1+duval*w_max)={A_warm:.3g} MPa^-3 yr^-1")

print("\n-- 1a. scalar T_srf = 253.15 K (idealized MISMIP-style path)")
A_scal = compute_fluidity_prior(u, H, s, b, C, acc, 253.15, max_picard=60)
ok_band = (A_scal.dat.data_ro.min() >= 1.0 - 1e-9
           and A_scal.dat.data_ro.max() <= A_warm * 1.05)
check("scalar-T_srf prior in physical band", ok_band,
      f"A in [{A_scal.dat.data_ro.min():.2f}, {A_scal.dat.data_ro.max():.2f}]")

print("\n-- 1b. spatially-varying T_srf (223 K inland -> 268 K margin), tas-style")
T_srf = Function(Q, name="T_srf").interpolate(223.0 + 45.0 * x / Lx)
A_var = compute_fluidity_prior(u, H, s, b, C, acc, T_srf, max_picard=60)
ok_band_v = (A_var.dat.data_ro.min() >= 1.0 - 1e-9
             and A_var.dat.data_ro.max() <= A_warm * 1.05)
check("varying-T_srf prior in physical band", ok_band_v,
      f"A in [{A_var.dat.data_ro.min():.2f}, {A_var.dat.data_ro.max():.2f}]")

# The surface BC must actually matter: cold-side ice softer than warm-side?
# No - cold side must be STIFFER (lower A). Compare slow-flank means (away
# from the shear-heated trunk) between the cold inland third and warm third.
xy = Q.mesh().coordinates.dat.data_ro
flank = np.abs(xy[:, 1] - Ly / 2) > 30e3
cold = flank & (xy[:, 0] < Lx / 3)
warm = flank & (xy[:, 0] > 2 * Lx / 3)
m_cold = A_var.dat.data_ro[cold].mean()
m_warm = A_var.dat.data_ro[warm].mean()
check("colder surface -> stiffer ice (lower A_prior)", m_cold < m_warm,
      f"flank mean A: cold third {m_cold:.2f} < warm third {m_warm:.2f}")

# Figure artifact: the reviewer-visible product of the thermo prior.
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
from matplotlib.colors import LogNorm
fig, axes = plt.subplots(2, 2, figsize=(13, 7))
for ax, (f, ttl, kw) in zip(axes.flat, [
        (speed, "synthetic |u_obs| (m/yr)", dict(cmap="viridis")),
        (T_srf, "surface temperature T_srf (K)", dict(cmap="inferno")),
        (A_scal, "A_prior, scalar T_srf=253 K", dict(
            norm=LogNorm(max(1.0, A_scal.dat.data_ro.min()), A_scal.dat.data_ro.max()),
            cmap="magma")),
        (A_var, "A_prior, varying T_srf (223-268 K)", dict(
            norm=LogNorm(max(1.0, A_var.dat.data_ro.min()), A_var.dat.data_ro.max()),
            cmap="magma"))]):
    col = fd.tripcolor(f, axes=ax, **kw)
    fig.colorbar(col, ax=ax, shrink=0.85)
    ax.set_title(ttl, fontsize=10)
    ax.set_aspect("equal")
    ax.axis("off")
fig.suptitle("thermo_model.compute_fluidity_prior on a synthetic ice stream "
             "(fixed-velocity enthalpy Picard)", fontsize=11)
fig.tight_layout()
figfn = os.path.join(EVID, "thermo_prior_synthetic.png")
fig.savefig(figfn, dpi=140, bbox_inches="tight")
print(f"  figure saved: {figfn}")

# ----------------------------------------------------------------------------
print()
print("=" * 72)
print("2. Whittle-Matern prior (icepack2_tools/prior.py)")
print("=" * 72)
from icepack2_tools.prior import (regularization_form,
                                  regularization_gradient_form, L_REG)

Lp = 100e3
mesh2 = fd.RectangleMesh(128, 128, Lp, Lp)
Q2 = FunctionSpace(mesh2, "CG", 1)
x2, y2 = SpatialCoordinate(mesh2)
theta = Function(Q2).interpolate(sin(2 * pi * x2 / Lp) * sin(2 * pi * y2 / Lp))
gamma, area = 1.0e4, Lp * Lp
k2 = 2 * (2 * np.pi / Lp) ** 2
R_exact = 0.5 * gamma / area * (area / 4) * (1.0 + L_REG ** 2 * k2)
R_num = float(assemble(regularization_form(theta, gamma, area)))
rel = abs(R_num - R_exact) / R_exact
check("R(theta) matches analytic sine-field energy", rel < 0.01,
      f"assembled {R_num:.4f} vs exact {R_exact:.4f}, rel err {rel:.2e}")
print(f"  gamma=1e4 prior energy at unit-amplitude theta: {R_num:.1f} "
      f"(mass term share {(gamma/8)/R_exact:.1%} - the term the old "
      f"gradient-only form lacked)")

v = Function(Q2)
rng = np.random.default_rng(42)
v.dat.data[:] = rng.standard_normal(v.dat.data_ro.shape)
dR = assemble(regularization_gradient_form(theta, fd.TestFunction(Q2), gamma, area))
directional = float(np.dot(dR.dat.data_ro, v.dat.data_ro))
eps = 1e-3
tp = Function(Q2); tm = Function(Q2)
tp.dat.data[:] = theta.dat.data_ro + eps * v.dat.data_ro
tm.dat.data[:] = theta.dat.data_ro - eps * v.dat.data_ro
fd_dir = (float(assemble(regularization_form(tp, gamma, area)))
          - float(assemble(regularization_form(tm, gamma, area)))) / (2 * eps)
rel_g = abs(directional - fd_dir) / max(abs(fd_dir), 1e-30)
check("gradient form == Gateaux derivative of energy", rel_g < 1e-9,
      f"<dR,v>={directional:.10e} vs central-FD {fd_dir:.10e}, rel {rel_g:.1e}")

# ----------------------------------------------------------------------------
print()
print("=" * 72)
print("3. load_mean_annual_surface_temperature vs synthetic ISMIP7 tas tree")
print("=" * 72)
import xarray as xr
from icepack2_tools.forcing import load_mean_annual_surface_temperature

root = os.path.join(EVID, "fake_ismip7_root")
tdir = os.path.join(root, "CESM2-WACCM", "climatology", "SDBN1", "tas", "v1")
os.makedirs(tdir, exist_ok=True)
xs = np.linspace(-50e3, 450e3, 51)   # covers the 400x100 km mesh
ys = np.linspace(-50e3, 150e3, 21)
months = np.arange(1, 13)
XX, YY = np.meshgrid(xs, ys)
base = 230.0 + 40.0 * np.clip(XX, 0, Lx) / Lx            # 230 -> 270 K west-east
seas = 10.0 * np.cos(2 * np.pi * (months[:, None, None] - 1) / 12)
tas = base[None, :, :] + seas                             # monthly cycle, mean=base
tas[:, ys > 105e3, :] = 0.0                               # "ocean" NODATA rows ~0 K
xr.Dataset({"tas": (("month", "y", "x"), tas)},
           coords={"month": months, "y": ys, "x": xs}).to_netcdf(
    os.path.join(tdir, "tas_synthetic_test.nc"))
os.environ["ISMIP7_DATA_ROOT"] = root

Tload = load_mean_annual_surface_temperature(Q, var="tas")
node_x = xy[:, 0]
expect = 230.0 + 40.0 * np.clip(node_x, 0, Lx) / Lx
err = np.abs(Tload.dat.data_ro - expect).max()
# nearest-neighbour sampling on a 10 km grid of a 0.1 K/km ramp -> <= ~0.55 K
check("tas reader: monthly mean + nearest-neighbour sampling", err < 0.6,
      f"max |T - expected ramp| = {err:.3f} K on 10 km source grid")
check("tas reader: seasonal cycle averaged out", True,
      f"loaded T in [{Tload.dat.data_ro.min():.1f}, {Tload.dat.data_ro.max():.1f}] K")

# NODATA fill: point-load onto a tiny mesh sitting in the "ocean" rows.
mesh_o = fd.RectangleMesh(4, 4, 20e3, 20e3)
mesh_o.coordinates.dat.data[:, 1] += 125e3               # shift into ocean strip
Qo = FunctionSpace(mesh_o, "CG", 1)
T_o = load_mean_annual_surface_temperature(Qo, var="tas")
check("tas reader: ocean/NODATA masked to fill_K=260",
      np.allclose(T_o.dat.data_ro, 260.0),
      f"ocean-strip values all {T_o.dat.data_ro.min():.1f} K")

# ----------------------------------------------------------------------------
print()
print("=" * 72)
print("4. MAP checkpoint round-trip: fluidity_prior + A = A_prior*exp(phi)")
print("=" * 72)
phi = Function(Q, name="log_fluidity").interpolate(0.5 * sin(2 * pi * x / Lx))
chk_fn = os.path.join(EVID, "map_roundtrip.h5")
with fd.CheckpointFile(chk_fn, "w") as chk:
    chk.save_mesh(mesh)
    chk.save_function(phi, name="log_fluidity")
    chk.save_function(A_var, name="fluidity_prior")
with fd.CheckpointFile(chk_fn, "r") as chk:
    mesh_r = chk.load_mesh()
    phi_r = chk.load_function(mesh_r, name="log_fluidity")
    A_prior_r = chk.load_function(mesh_r, name="fluidity_prior")
Qr = phi_r.function_space()
A_rebuilt = Function(Qr).interpolate(A_prior_r * fexp(phi_r))
expected = A_var.dat.data_ro * np.exp(phi.dat.data_ro)
err_rt = np.abs(np.sort(A_rebuilt.dat.data_ro) - np.sort(expected)).max()
check("checkpoint carries A_prior; forward rebuild A=A_prior*exp(phi)",
      err_rt < 1e-8, f"max abs err {err_rt:.2e}")
# older MAPs without the field must raise (simulation.py catches -> legacy)
try:
    with fd.CheckpointFile(chk_fn, "r") as chk:
        m2 = chk.load_mesh()
        chk.load_function(m2, name="nonexistent_prior")
    check("missing fluidity_prior raises (legacy-MAP fallback path)", False)
except Exception as e:
    check("missing fluidity_prior raises (legacy-MAP fallback path)", True,
          type(e).__name__)

# ----------------------------------------------------------------------------
print()
print("=" * 72)
print("5. Import smoke + keep-in-sync constants of the changed scripts")
print("=" * 72)
import importlib.util as ilu
import types
# colorcet (a colormap-only library used in the plotting helpers) lives in the
# older ~/venv-firedrake env, not venv-firedrake-2026; its import predates this
# change. Stub it so the smoke test exercises THIS change's module-level code.
if "colorcet" not in sys.modules:
    try:
        import colorcet  # noqa: F401
    except ModuleNotFoundError:
        sys.modules["colorcet"] = types.ModuleType("colorcet")
        print("  (colorcet stubbed - not in venv-firedrake-2026; "
              "pre-existing import, colormaps only)")
sys.path.insert(0, os.path.join(REPO, "antarctica", "scripts"))
mods = {}
for name in ("inversion_icepack2", "simulation"):
    spec = ilu.spec_from_file_location(
        name, os.path.join(REPO, "antarctica", "scripts", f"{name}.py"))
    mod = ilu.module_from_spec(spec)
    spec.loader.exec_module(mod)
    mods[name] = mod
    print(f"  imported {name}.py OK")
inv, sim = mods["inversion_icepack2"], mods["simulation"]
check("N_FLOW/A4_FACTOR defaults in sync (inversion vs forward)",
      inv.N_FLOW_DEFAULT == sim.N_FLOW_DEFAULT == "3.0"
      and inv.A4_FACTOR_DEFAULT == sim.A4_FACTOR_DEFAULT == "1.0",
      f"n={inv.N_FLOW_DEFAULT}, a4={inv.A4_FACTOR_DEFAULT}")
os.environ.pop("ISMIP7_N_FLOW", None)
check("map_n_tag: n=3 default tags MAPs '_n3' in both scripts",
      inv.map_n_tag() == sim.map_n_tag() == "_n3")
os.environ["ISMIP7_N_FLOW"] = "4.0"
check("map_n_tag: n=4 override keeps legacy untagged name",
      inv.map_n_tag() == sim.map_n_tag() == "")
os.environ.pop("ISMIP7_N_FLOW", None)
check("Whittle-Matern gamma defaults are the physical 1e4",
      inv.GAMMA_THETA == 1.0e4 and inv.GAMMA_PHI == 1.0e4 and inv.L_REG == 7.5e3,
      f"gamma_theta={inv.GAMMA_THETA:g}, gamma_phi={inv.GAMMA_PHI:g}, L={inv.L_REG:g}")

print()
if failures:
    print(f"RESULT: {len(failures)} FAILURE(S): {failures}")
    sys.exit(1)
print("RESULT: ALL CHECKS PASSED")
- Outcome: ⚠️ 2 issues (1 warning, 1 info) across 1 run (7m18s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

⏭️ **Rebase** - skipped
  • ⚠️ antarctica/scripts/control/run.py - merge conflict rebasing onto origin/main
  • ⚠️ antarctica/scripts/simulation.py - merge conflict rebasing onto origin/main
🔧 **Review** - 6 issues found → auto-fixed ✅
  • ⚠️ antarctica/scripts/simulation.py:336 - Default ISMIP7_MAP_CLIP=6.0 clips legitimate n=3 controls: N3_FRAMEWORK.md states the physical n=3 controls reach ~8 and instructs operators to set ISMIP7_MAP_CLIP=10 manually, but no code, launcher, or restart guard enforces this. A forward run (or a restart of a run originally launched with MAP_CLIP=10 but resumed with default env) silently clips valid theta/phi to |.|<=6, distorting the validated n=3 physics with only a log line. The restart guards check friction law and apparent-MB consistency but not the clip. Consider an n-aware default (e.g. 10 when ISMIP7_N_FLOW=3), or storing the control range / clip value in the MAP checkpoint and guarding on it like friction.
  • ℹ️ antarctica/scripts/inversion_icepack2.py:107 - The new inversion method defaults are not gated on the flow exponent: GAMMA_THETA/GAMMA_PHI changed 1.0 -> 1e4 and ISMIP7_FLUIDITY_PRIOR defaults to 'thermo' for every inversion run from this branch, including n=4 (whose MAP filename carries no n-tag via map_n_tag()) and regularized-Coulomb friction. Regenerating an n=4 MAP from this branch would silently produce a different-method MAP under the same filename the antarctica branch's old-method MAP uses; after the planned merge into icepack/ismip7 upstream this changes the default inversion method for all users. Confirm this is intended or gate the thermo-prior/gamma defaults on n==3.
  • ℹ️ icepack2_tools/thermo_model.py:208 - compute_fluidity_prior returns silently when the under-relaxed Picard loop exhausts max_picard without reaching rtol - the 'thermo prior converged.' message only prints on success, so an unconverged A_prior feeds the inversion with no warning. Emit a loud warning (or raise) when the loop ends without dA < rtol.
  • ℹ️ antarctica/scripts/simulation.py:385 - The legacy constant-baseline fallback is silent: when a checkpoint lacks fluidity_prior, or the load fails via the broad 'except Exception' at line 238, the forward quietly reconstructs A on the constant A0*a4_factor baseline. The loaded path prints the A_prior range but the fallback branch prints nothing, so an n=3 forward accidentally running against the wrong baseline (e.g. a corrupt/partial checkpoint read) is hard to diagnose from logs. Add a PETSc.Sys.Print in the fallback branch noting the legacy baseline is in use.
  • ℹ️ antarctica/scripts/inversion_icepack2.py:387 - Both branches of the friction conditional now compute the identical 'C_w0 = weertman_anchor(H, s, u_obs, m_slide_val, Q)' (lines 387 and 398); only the log messages differ. Hoist the assignment above the if/else and keep the branch-specific prints.
  • ℹ️ icepack2_tools/thermo_model.py:118 - Performance note on the Picard loop: the enthalpy bilinear form (a_adv + a_exch) is independent of A_k - fluidity enters only the RHS strain-heating term - and of with_strain, so all ~2*max_picard fd.solve calls assemble and MUMPS-factorize the same matrix (up to ~120 redundant LU factorizations). Trivial at 32 km but material at production resolutions; a cached firedrake LinearSolver with per-call RHS reassembly would cut prior-setup cost. Left as informational since the module is deliberately a near-verbatim port of the mismip thermo model.

🔧 Fix: n-aware MAP clip default; loud prior/baseline warnings; dedupe anchor
✅ Re-checked - no issues remain.

⚠️ **Test** - 2 issues (1 warning, 1 info)
  • ⚠️ The author-run 32 km real-data validation (thermo prior T 221-272 K, MAP misfit 4815, forward setup misfit 6191) cannot be reproduced in this gated worktree: the required datasets (BedMachine, MEaSUREs velocity, RACMO SMB, ISMIP7 tas climatology, meshes, MAP h5 files) are per-machine and gitignored. The new code paths were instead validated end-to-end on synthetic physically-scaled inputs (all checks passed), and the 8c5f2e2 first-solve fail-loud guard plus the n-aware MAP clip default were verified by code inspection only since triggering them needs the real data/MPI setup. Decide whether the author's documented on-machine validation (N3_FRAMEWORK.md) plus this synthetic evidence is sufficient.
  • ℹ️ antarctica/scripts/inversion_icepack2.py:65 - Pre-existing (not introduced by these commits): import colorcet at the top of antarctica/scripts/inversion_icepack2.py does not resolve in the primary ~/venv-firedrake-2026 environment (colorcet 3.1.0 lives only in the older ~/venv-firedrake). The N3_FRAMEWORK.md run recipe would fail at import in the 2026 venv unless colorcet is installed there or provided via PYTHONPATH. It was stubbed for the import-smoke test since it is a colormap-only plotting dependency.
  • validate_n3_prior.py section 1: compute_fluidity_prior on a synthetic 400x100 km ice stream - Picard convergence (14/18 iters), physical A band, colder-surface-stiffer-ice check, for scalar and spatially-varying T_srf
  • validate_n3_prior.py section 2: Whittle-Matern regularization_form vs analytic sine-field energy (rel err 6e-4) and regularization_gradient_form vs central finite differences (rel err 1e-10)
  • validate_n3_prior.py section 3: load_mean_annual_surface_temperature against a synthetic ISMIP7-layout tas netcdf (monthly mean, nearest-neighbour sampling, NODATA/ocean fill to 260 K)
  • validate_n3_prior.py section 4: CheckpointFile round-trip of fluidity_prior + forward rebuild A = A_prior*exp(phi) (1e-14) and missing-field raise for the legacy-MAP fallback
  • validate_n3_prior.py section 5: import smoke of inversion_icepack2.py and simulation.py, N_FLOW/A4_FACTOR keep-in-sync check, map_n_tag() n=3/_n3 vs n=4/untagged, gamma=1e4 defaults
  • Code inspection of the 8c5f2e2 first-iteration fail-loud guard and the 63f9dbc n-aware ISMIP7_MAP_CLIP default (not executable without real Antarctic data/MPI)
  • git status --porcelain clean after testing; test fixtures removed
✅ **Document** - passed

✅ No issues found.

⚠️ **Lint** - 1 info
  • ℹ️ No linter or formatter is configured in the repository (no pyproject.toml/setup.cfg/.flake8/pre-commit) and none is installed in the Firedrake venv or system Python, so this pass could only run py_compile plus an AST unused-import scan on the changed files. If deeper lint coverage is wanted, add a ruff/flake8 config and install the tool in the project venv.
✅ **Push** - passed

✅ No issues found.

hoffmaao and others added 30 commits May 7, 2026 18:28
The whole antarctica/mesh/ dir was gitignored to keep the large .msh/.h5
files out of git, but that also hid the 155-byte boundary_ids*.json files
that every solver script reads by default (ISMIP7_BNDIDS -> calving/other
physical-line ids for the calving-terminus BC). A fresh clone hit a missing
default path. Switch the rule to 'antarctica/mesh/*' + a '!boundary_ids*.json'
negation (a negation under the dir form is a no-op) and commit the four files.

These encode the gmsh $PhysicalNames split: addPhysicalGroup() auto-numbers
the alternating Calving_/Other_ segments 1,2,3,... so odd=calving, even=other.

Caveat: boundary_ids_buffered.json (55 ids) matches only the 4km/8km buffered
meshes; the finer buffered meshes have 35 ids (use boundary_ids.json). The
buffered sidecar is not yet per-mesh.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…s generator

- forcing.py: load_racmo_smb_climatology() reprojects RACMO ANT11 SMB to
  EPSG:3031 (rasterio) and samples to the mesh via icepack.interpolate
- control/run.py: use RACMO climatology as the SMB baseline (acabf fallback),
  report area-weighted mean SMB
- simulation.py: VAF/mass diagnostics use SI ice density (RHO_I_SI=917); the
  imported icepack rho_I is in MPa-m-yr units and was zeroing the output
- mesh.py / mesh_antarctica.py: loaders take data_dir so they resolve
  antarctica/data instead of <project>/data
- make_boundary_ids.py: generate boundary_ids.json from mesh $PhysicalNames

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The boundary-id sidecar only matches the specific mesh it was generated from
(its segment count is a deterministic function of the BedMachine input and
SIMPLIFY_TOL/SUBSAMPLE). Committing a fixed sidecar silently mismatches any
mesh built from different inputs. Instead, make_boundary_ids exposes an
importable write_boundary_ids(), and mesh_antarctica.py calls it after writing
each mesh so the sidecar always matches the local mesh.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…orcing

Reconciles our local work with the upstream ocean-melt branch.

Conflict resolutions:
- simulation.py: keep upstream's ocean_melt mass sink and h_clamp=0; take
  upstream's _RHO_I_SI/362.5 for the VAF/mass diagnostic (same fix as ours),
  dropping our redundant RHO_I_SI/GT_PER_MM_SLE duplicates.
- control/run.py: RACMO climatology SMB baseline (ours) + per-basin ocean-melt
  callback (theirs); union imports; construct atm lazily in the acabf fallback.
- forcing.py: our load_racmo_smb_climatology alongside their Burgard melt
  parameterization (auto-merged).

boundary_ids policy: sidecars are now generated locally to match the mesh
(mesh_antarctica.py emits one), so .gitignore ignores the whole mesh dir and
upstream's four committed (35-segment) sidecars are dropped — they matched no
mesh reproducible from the repo.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds make_synthetic_ocean_callback: prescribes TF(draft) = clip(tf_max *
|draft|/depth_ref, 0, tf_max) and runs it through the Burgard
quadratic_mixed_slope melt with constant salinity and the default scalar K.
Needs no ocean/meltMIP data, so it works without the Globus download.

Opt-in via ISMIP7_SYNTHETIC_MELT=1 (tunable with ISMIP7_SYNTH_TF_MAX /
ISMIP7_SYNTH_DEPTH_REF); the real per-basin K path remains the default and is
unchanged. Uncalibrated development stopgap, not protocol forcing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Debugging tooling for the diagnostic solver:
- simulation.py: opt-in SNES/KSP convergence monitoring (ISMIP7_SNES_MONITOR),
  routed to a per-run file via ISMIP7_SNES_LOG (ascii:<file>) or stdout, so
  concurrent debug runs don't interleave.
- control/run.py: --snes-monitor / --snes-log CLI flags that set those, plus a
  --restart <checkpoint.h5> option to resume a control run.
- simulation.py: clamp the loaded thickness to h_clamp on restart, so a run
  never resumes below the floor (no-op for the default h_clamp=0).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- --tag (ISMIP7_RUN_TAG): suffix on experiment_name so concurrent/successive
  runs write distinct output files instead of overwriting each other.
- --checkpoint-interval (ISMIP7_CHECKPOINT_INTERVAL): control how often
  thickness+velocity checkpoints are saved (e.g. frequent snapshots for
  debugging). Default 100, so existing behavior is unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Default to antarctica_64000_500_aniso.msh (the mesh behind
inversion_icepack2_500.h5) instead of the 1 km aniso mesh, raise the
memory gate to 128 GB for the ~7M-DOF MUMPS factorization, and mark the
script executable. This is the configuration that produced
inversion_icepack2_rc_500.h5.
ISMIP7_FRICTION=regularized_coulomb makes setup_model() drive the
prognostic system from the inversion_icepack2_rc_<lc>.h5 MAP:

- geometry (thickness/bed/surface) and velocity_obs are taken from the
  checkpoint itself, so the Weertman anchor C_w0 — and hence the meaning
  of the inverted theta — is reproduced exactly; a node-ordering check
  guards the dof copy (also protects the existing Budd path).
- the diagnostic solves build_rc_residual on the LIVE prognostic (h, s):
  grounded gate, Coulomb cap c0*N, and driving stress all track the
  evolving geometry, so the grounding line migrates with exact-zero
  shelf drag and no thickness clamp anywhere (h_clamp_init defaults to
  0 in RC mode; h_visc_floor=10 m handles the ice-free buffer).
- composite alpha defaults to the inversion's 1e-2 in RC mode so the
  forward diagnostic is consistent with the MAP.
- setup_model now reports the initial velocity misfit vs observations
  (the rc_500 checkpoint predates its final report, so this is where we
  learn how converged that MAP is).

Verified at LC=32000 against the completed RC inversion: the forward
reproduces the checkpoint's saved velocity to rel L2 = 3.2e-9, and a
short dt=0.1 clamp-free prognostic runs cleanly.
The acabf-anomaly files are referenced to the ESM's 1960-1989
climatology, and no historical scenario exists in the local data tree —
so the projection driver's anomaly-only SMB was missing the entire
climatological accumulation, while the CTRL driver's climatology lookup
silently fell back to zero SMB. Fix both sides consistently:

- ssp585 driver: force with the full acabf(t) field (smb_anomaly=False,
  new make_forcing_callback knob). Against a CTRL held at the 2015-2029
  acabf mean, projection minus control is the forced signal relative to
  the 2015-centered climate with no anomaly re-referencing.
- control driver: compute the reference climatology from full acabf
  pooled over historical + projection scenarios in an env-configurable
  window (default 2015-2029, the locally available years); raise
  instead of silently running a zero-SMB control.
- both drivers pick up the calibrated per-basin K npz (falling back to
  the 2500 m calibration — 16 basin scalars remapped via the IMBIE2
  8 km grid, mesh-independent); ssp585 default dt is now 0.1 per the
  May dt sweep.
- launch_rc_ctrl_when_ready.sh: resource-gated launcher for the 500 m
  regularized-Coulomb CTRL2015 verification run (dt=0.1, 2015-2025,
  24 ranks), logging under antarctica/results/logs/ so run logs survive
  reboots.

End-to-end tested at LC=32000 through control/run.py with
ISMIP7_FRICTION=regularized_coulomb.
Bring Dan's fixes into the antarctica branch: RACMO2.4p1 SMB
climatology baseline (load_racmo_smb_climatology), SNES/KSP convergence
monitoring with per-run log routing, restart/argparse tooling in the
control driver, the synthetic depth-dependent melt stopgap
(ISMIP7_SYNTHETIC_MELT=1), and the boundary-ids generator
(make_boundary_ids.py; the static boundary_ids*.json leave tracking).

control/run.py resolution composes both sides: RACMO stays the primary
CTRL baseline (area-weighted mean report), the fallback is the pooled
historical+projection full-acabf climatology (anomaly means are wrt
1960-1989 and are not a baseline), and a missing climatology is a hard
error instead of a silent zero-SMB control (ISMIP7_ALLOW_ZERO_SMB=1 to
override). The OI-climatology + per-basin calibrated K path remains the
default ocean forcing.
SMB(t) = RACMO_clim + [aSMB(t) - mean(aSMB over the reference window)]:
the acabf-anomaly is referenced to the ESM's 1960-1989 climatology, so
re-referencing over the CTRL window (2015-2029 of it exists locally)
removes that mismatch, and the RACMO baseline keeps projection and
control on the same absolute SMB. Falls back to full acabf(t) when
RACMO is absent. make_forcing_callback grows an smb_baseline parameter
(per-node array added to the anomaly each step).

Also: run_simulation nsteps now rounds instead of truncating (a 285-yr
dt=0.1 run silently lost its final step), and the RC CTRL launcher
verifies boundary_ids.json exists (untracked on this branch) and pins
ISMIP7_BNDIDS for the run.
front, and an observation mask for the inversion

Investigating the mass gain Dan saw. A new per-step mass audit in
run_simulation attributes dM to SMB - melt - boundary outflux - calving
and exposes any created mass; at 32 km it immediately caught two real
leaks in the pre-existing scheme:

- the -h*div(u*phi) volume term is non-conservative for DG0 (spurious
  h*div(u) mass at thickness jumps; +4700 Gt/yr at the test front). For
  piecewise constants the facet upwind terms alone are the finite-volume
  scheme, so the volume term is now dropped by default.
- the L2 DG0->CG1 projection undershoots negative at steep fronts and
  the h floor then injects mass (+7600 Gt/yr step-1 at 32 km). Replaced
  by a lumped-mass projection (convex combination of adjacent cell
  values: bounded, no floor injection, integral-preserving).

With both fixes the audited budget closes to machine precision
(resid = 0.00, clamp = 0.0-0.2 Gt/yr). ISMIP7_LEGACY_TRANSPORT=1
restores the old scheme for comparison.

ISMIP7_FIXED_FRONT=1 (now set by the RC CTRL launcher) removes ice that
flows past the initially ice-free cells and tallies it as calving flux
- without it a buffered mesh has no calving sink (~1300 Gt/yr in
reality) and must gain mass. Only meaningful with a true-geometry
initial state (RC mode).

The inversion misfit now excludes nodes without velocity observations
(MEaSUREs ERR > 0 defines coverage; icepack zero-fills NODATA, which
read as "observed stationary" and prescribed friction where there is no
constraint). Coverage gaps are small - 0.7% of grounded, 0.5% of
floating area - so this is rigor, not the mass-gain culprit. obs_mask is
saved in inversion checkpoints; ISMIP7_OBS_MASK=0 reverts.
get_thermal_forcing/get_salinity now load one (variable, year) slice
out of its decadal chunk file into an in-memory nearest-neighbour
RegularGridInterpolator over (z, y, x), cached across steps. The
previous pointwise xarray .interp over the open_mfdataset dask graph
cost ~10 minutes per step even at 5k mesh nodes; the new path reads a
year in ~2 s and answers in ~1 ms, and is verified bit-identical to an
xarray nearest-neighbour reference over 2000 random points and drafts.
Nearest-neighbour also matches the OI-climatology CTRL path and the
per-basin K calibration. A 32 km SSP5-8.5 step now costs the same as a
CTRL step (~10 s after warm start, was ~675 s).

ISMIP7_K_SCALE multiplies the melt K in both the projection callback
and the CTRL driver (the calibrated per-basin K integrates 689 vs 865
Gt/yr observed on the 2500 m mesh; 1.26 matches the observed total).
The front mask was derived from the run's starting h, so a restarted
run (the CTRL -> projection chain) would re-mask cells that
legitimately retreated mid-run inside the observed extent and forbid
them from re-advancing. setup_model now exposes the t=0
BedMachine/inversion thickness as ctx["H_init"] (restart_from does not
overwrite it) and run_simulation builds the mask from that. Cold-start
behavior is unchanged (verified identical at 32 km).
Local working-notes patterns belong in each clone's .git/info/exclude
rather than the shared .gitignore, and the README's writeup pointer now
names only the tracked document.
All ESM-forced core experiments (historical 1/2, SSP projections 3-8)
are the same run with different (esm, scenario, period), so the nine
drivers are now thin shims over experiment.run_core_experiment. That
puts every fix from this week's work in force for the whole matrix at
once: RACMO baseline + aSMB re-referenced over one shared pool
(historical + ISMIP7_CLIM_SCENARIO, 2000-2029) so all cores and the
CTRL share a single baseline and the 2014/2015 historical->projection
handoff is seamless; calibrated per-basin K with the 2500 m fallback;
dt=0.1; hard fast-failure (seconds, before the expensive model setup)
when the forcing tree for an (ESM, scenario) is not downloaded, instead
of silently running anomaly-only/zero-SMB.

OCX (core 11) is rebuilt as a genuinely observation-forced run: RACMO
actual-year SMB (1979-2023) + constant OI-climatology ocean with
per-basin K — runnable entirely from local data. The OI-climatology
ocean callback moves from control/run.py into icepack2_tools.forcing
(shared by CTRL and OCX).

Verified at 32 km: the ssp585 shim reproduces the standalone driver's
results exactly; OCX runs with a closed mass budget; missing-data
drivers (historical, ssp370) fail in ~7 s with the download hint.

Co-authored work retained from prior drivers; fracture masks are loaded
when present but still not applied to the thickness update (open
protocol item).
Gate log, run log, and lockfile move from /tmp (wiped on reboot — the
Jun 20 rc_500 inversion log was lost that way) to the repo results
tree, tagged by resolution so a 2500 m and a 500 m gate can coexist.
The AIS_ocean/share_with_modellers tree is gone; data now lives under
CMIP6_test_protocol/AIS mirroring the runtime layout, so downloads land
directly in ISMIP7/AIS/ (no rename step, closing the README caveat).

Auth is now headless-first: the script's own refresh-token cache, then
the globus CLI token store (~/.globus/cli/storage.db keeps a long-lived
transfer refresh token + the per-user confidential client), then an
interactive login that now requests refresh tokens so it is one-time.
The destination endpoint defaults to the Globus Connect Personal id
registered on the machine. Transfers verify checksums.

Manifest updated to what the share actually carries today: the SDBN1
1960-1989 long-term means (the exact reference climatology of the
acabf-anomaly files), ocean ct/sa climatology+bias, the newer 06_nov OI
climatology (1972-2024), the ISMIP7 obs kit, IMBIE v3, grid, and
topography. NOT yet published upstream: historical yearly forcing,
processed ssp126/ssp370, all of MRI-ESM2-0 — and the processed ssp585
tree we hold locally is no longer on the share (treat it as
irreplaceable).

Co-existing raw CMIP output lives in ../CMIPraw for when processing
becomes necessary.
- build_oi_climatology_interpolators grows ISMIP7_OI_VERSION (default
  30_sep, matching the per-basin K calibration; 06_nov selects the
  reorganized share's 1972-2024 re-release). The 06_nov so/thetao files
  currently ship the tf field by mistake (upstream packaging bug,
  reported) - the reader refuses them with an explanatory error instead
  of silently melting with TF-as-salinity.
- load_K_per_basin falls back to the IMBIE v3 basin file when the
  calibration-era v2 is absent (verified identical basinNumber field,
  so the calibrated per-basin K remains valid with either).
- scripts/preflight.py: five-second per-core readiness table checking
  every input each experiment needs (mesh, MAP, boundary ids, K, RACMO,
  OI climatology, atmosphere/ocean coverage over the run period),
  honoring the same env knobs as the runs. Current verdict: cores 7, 9,
  11 READY (core 10 = core 9 while RACMO is the baseline); 1-6, 8
  blocked on unpublished upstream forcing.
Re-walked the whole GHub tree to diagnose what climate forcing is
pullable. The collection is mid-reorganization: CMIPraw is now gone
entirely, AIS/MRI-ESM2-0 does not exist, and AIS/CESM2-WACCM holds only
climatologies + bias ingredients (no scenario dirs, no per-year
forcing). Everything still pullable is already mirrored locally, so the
downloader is a no-op. The processed CESM2-WACCM/ssp585 forcing we run
from (133 GB) is no longer on the share — flag the local copy as
irreplaceable.
The 500 m RC CTRL crashed in setup_model's initial n=1->4 continuation
(DIVERGED_MAX_IT): a fine mesh + a rough mid-optimization MAP outruns
Newton across the 5-step ramp, and unlike the timestep loop the initial
solve had no retry. Now it restores the initial guess and re-ramps with
2x then 4x steps on divergence (ISMIP7_CONTINUATION_STEPS, default 8),
and reports the step count that worked. 32 km reproduces the identical
initial misfit and VAF/mass through the new path (no regression).
On the 500 m mesh each diagnostic solve is costly, so seed the adaptive
continuation finer (16) rather than failing at 8 and re-ramping.
Diagnosis of the 500 m CTRL divergence: the rc_500 MAP (a 20-iter
mid-optimization checkpoint) carries ~700 theta/phi nodes with |.| up to
23, i.e. exp() coefficients ~1e10 that are local singularities the SNES
cannot solve through — not a step-size issue. A converged MAP is O(1)
everywhere (rc_32000 maxes at 1.64). setup_model now clips
theta/phi to |.|<=ISMIP7_MAP_CLIP (default 6) and logs the count,
removing the pathology (0.1% of nodes) without touching real structure.
No-op for converged MAPs. Pairs with the adaptive continuation to
unblock the 500 m cold start pending a proper re-inversion.
Adds a minimum-yield shelf-drag knob to the RC forward and records the
multi-step instability diagnosis. The RC diagnostic velocity blows up
~2x/step in prognostic use (global, hypersensitive to sub-percent
geometry change) — confirmed independent of transport scheme (legacy
and conservative both blow up), independent of the fixed calving front
(on/off identical), and NOT curable by shelf drag (tested up to 50 kPa,
already enough to defeat RC's purpose). Budd ran stably for 10-20 steps
in May, so this is RC-specific: exact-zero shelf drag + composite
viscosity leaves the mixed diagnostic system near-singular as h evolves.
The knob and MAP clip / adaptive continuation stay as robustness
options; the forward prognostic needs Budd (proven stable) or a
monolithic implicit coupling.
Following the pivot away from the prognostically-unstable RC forward:
a resource-gated 2500 m Budd CTRL2015 validation run with the full
conservation pipeline (RACMO SMB, OI ocean + per-basin K, conservative
transport, fixed front, h_clamp_init=0, dt=0.1). Also confirms the
Budd + fixed-front + h_clamp_init=0 config (new since May, which used
h_clamp_init=10); a non-converging run self-terminates in ~10 steps so
gating is safe. The rc_ctrl launcher is superseded for forward use.
hoffmaao added 27 commits July 18, 2026 18:11
- download_forcing.py now mirrors straight into ISMIP7/AIS (drop the
  layout-mismatch caveat + known issue; new remote base path; point at
  the script's status comment for the upstream scenario-data gap)
- boundary_ids sidecars: auto-generated by mesh_antarctica.py /
  make_boundary_ids.py, and mesh/ is now fully gitignored
- friction-tagged MAP checkpoints (ISMIP7_FRICTION, _budd/_rc names)
- control driver restart flags (--tag, --checkpoint-interval,
  ISMIP7_AUTO_RESUME), self-contained (u, M, tau) checkpoints
- timeseries CSV schema: full mass-budget columns incl. amb_gtyr
- env-knob table: checkpoint cadence, APPARENT_MB, FIXED_FRONT,
  LEGACY_TRANSPORT, SNES_TYPE/MAXIT
- known issues: blow-up fixed (audits ON TRACK); remaining item is the
  diagnostic-Newton wall on hard projection geometries
- check_ismip6_track.py audit documented under §6
- COMPOSITE_RHEOLOGY.md defaults updated for the residual friction laws
- data/README reduced to a pointer at the §1 dataset table
The mid-July GHub "shrinking collection" was the reorganization in
flight: the data MOVED to a new top-level /ISMIP7 tree (earlier walks
started inside /ISMIP6/ISMIP7_Prep and never looked up). /ISMIP7/AIS
holds the full scenario matrix for BOTH ESMs (ctrl, ctrlclim,
historical, ssp126, ssp370, ssp534-over, ssp585; per-year SDBN1-8000m
and 2000m atmosphere, decadal-chunk ocean so/tf/thetao, fracture),
plus cmipraw, the real OCX forcing product, SMBmip, submission
templates, AntarcticaObsISMIP7-v1.2, and a FIXED 06_nov OI climatology
(so/thetao v4; the old tree's v3 so/thetao are byte-identical to tf -
the packaging bug we reported).

Add --scenarios [--esm ...] [--scenario ...]: mirrors the minimal
runtime sets (SDBN1-8000m acabf + acabf-anomaly, ocean tf + so,
fracture) per (ESM, scenario) as recursive-dir Globus transfers with
version autodetection and checksum sync (idempotent - re-run to check
completeness or fill gaps). Local CESM2-WACCM/ssp585 verified complete
against the new tree, so the local copy is no longer irreplaceable.
First submission: CESM {historical, ssp126, ssp370} + MRI {historical,
ssp126, ssp370, ssp585} + fracture masks, ~130 GB.
The Jul 18 2500m Budd inversion died ~11 hr in (iter 60, misfit already
down 3.07e4 -> 5.2e3): tlm_adjoint's adjoint_jacobian_solve hit the SNES
iteration cap at rough mid-optimization controls, and only the FORWARD
solve was guarded. Wrap compute_gradient in the same recovery - restore
the backup state and return an inflated objective with a zero gradient
so L-BFGS-B backtracks its line search instead of crashing the run.
Relaunched warm from the periodic checkpoint with this guard in place.
compare_ismip6.py overlays our (projection - CTRL) sea-level contribution
on the Seroussi et al. 2020 computed scalars (~/data/ismip6 local mirror:
per-model computed_ivaf_minus_ctrl_proj files, ivaf m^3, in-file rho).
Pools RCP8.5-class experiments by default (exp01-05, 45 members), prints
ours vs ensemble p5/median/p95/min/max at marker years, exits 0 iff we
end inside the min-max envelope - the sharper companion to
check_ismip6_track.py's observational audit.

First result (32 km, 9.4-yr ssp585 vs balanced CTRL): ours -8.1 mm SLE
at 2024.5 vs ensemble [-1.9, +6.9] - OUTSIDE, on the SMB-gain side.
Consistent with (a) the pending ~9% aSMB unit-inflation finding and
(b) CMIP6 CESM2-WACCM's stronger Antarctic SMB response vs the CMIP5
forcings behind ISMIP6 - quantitative motivation for resolving the
units question before production runs.
run_simulation's step solve now escalates through a rescue ladder before
giving up: direct -> fresh continuation -> trust-region direct ->
trust-region continuation, restoring the last CONVERGED mixed state
between attempts (and at final stop, so checkpoints never hold a
diverged Newton iterate - closes the earlier review note). Based on the
2024.5 ssp585 wall evidence that newtontr converges steps nleqerr
cannot. Validated at 32 km: a healthy 2-yr CTRL never triggers it
(behavior unchanged), and the cold ssp585 wall reproducer now runs to
2025.9 with a demonstrated step-107 trust-region recovery before a
graceful stop (was 2024.4). The deep ssp585 hard era remains open
(monolithic coupling is the structural fix).

core_report.py generates the tracked per-core run record for
antarctica/reports/: run environment, budget rows at marker years, the
check_ismip6_track audit, the compare_ismip6 ensemble overlay for
projections, and provenance (git SHA, log, gitignored CSV path). Each
of the 11 core experiments lands as one commit whose payload is this
report.
Wall localization (32 km historicals, both ESMs stopping at ~1873; same
class as the ssp585 2024.5 wall): the unconvergeable residual is carried
~100% by a handful of floating h=0-60 m front nodes (Amery sector) whose
frozen-iterate speeds reach 1e10-1e11 m/yr - the gia frictionless-front
runaway, but above the h<10 m ocean-drag band, and only at hard-era
geometries.

Mechanism: the gia soft speed limiter is now structurally present
(threshold ISMIP7_U_LIM, default 2e4 m/yr) but INERT - its coefficient is
a live Constant at 0, so the term and Jacobian vanish identically and the
cold-start continuation is unaffected (a built-in limiter provably breaks
it). The step rescue ladder gains two deepest rungs that raise the
coefficient to ISMIP7_K_LIM under trust region for the rescue solve only,
then zero it: physical flow (<< u_lim) never feels the limiter, and it
exists exactly at wall geometries where the runaway nodes need pinning.

Restart fast path (setup_model): re-solve directly at the loaded
converged state with normal tolerances instead of re-ramping n from 1
(the ramp is fatal at wall geometries); if even that fails, keep the
loaded state, set a TIGHT run tolerance (1e-6 x the loaded-state
residual), and hand the hard step to the run's rescue ladder. The first
version of this path let its loose acceptance atol persist, silently
freezing the velocity for the rest of the run (0.2 s "steps") - that
invalidated one resume experiment and is exactly what the tight-scale
rule prevents.

Validated: 1-yr balanced CTRL regression is unchanged (no rescue fires,
limiter inert). The historical cores rerun cold with this machinery next;
their per-core reports record whether the 1873 era clears.
The 1873 hard eras are front-cell-emptying EVENTS in sequence: the
rescue ladder alone fails them at dt=0.1, but a dt=0.025 approach from
the 1870 checkpoint crossed the 1873.3 event with four consecutive
trust-region+limiter recoveries (then walled at the next event) - small
geometry increments let Newton track the branch through each event.

run_simulation's step is now transport-first (advance with the velocity
solved AT the current geometry, then solve at the new geometry - the
same (G, u) sequence as before, with the bonus that checkpoints store a
mutually consistent (h, u) pair). A step whose post-advance solve
exhausts the rescue ladder rewinds ITS OWN advance (h_dg/z entry
buffers) and retries it as dt/4 then dt/16 subcycles with full rescue
solves between substeps (ISMIP7_SUBCYCLES, default 1,4,16). Budget
tallies (outflux/calving/clamp/limit) accumulate over substeps into the
step's row, so the audit residual still closes exactly.

Validated: the 2-yr balanced CTRL is unchanged (flat VAF, resid 0.0,
ON TRACK, no rescue fires). Historical cores rerun next; their reports
record how the 1873+ era sequence resolves.
First core experiment of the matrix run end-to-end: full 1850-2014
window from the balanced init, 13 front-cell-emptying events crossed by
the dt-subcycle rescue ladder. Century-scale behavior is sound: SMB
2520, shelf melt 1371, front discharge 1050 Gt/yr means; dM/dt -314
Gt/yr; dVAF/dt -0.5 mm SLE/yr; budget residual <= 0.1 Gt/yr over 1640
steps. The single audit FAIL is the runaway detector's absolute-peak
clause on an isolated one-step discharge spike during an emptying event
(tallied, budget closed - a different signature from the sustained
exponential growth the clause was built for; left as-is pending a
ruling on refining the detector). Final state (t_yr=2014.0) is the
restart base for the CESM2-WACCM projections (cores 3/5/7).
Second core of the matrix, on the newly mirrored MRI forcing. Completed
in two legs: cold to 1876.9, then a warm resume whose x64 subcycle rung
crossed the 1877 Amery front-emptying event (a single 553 s step) and
cruised to 2014. Inter-ESM contrast established at 32 km: MRI gains
~+88 mm VAF over the window (stronger SMB) where CESM loses ~-87 mm.
Budget residual closed throughout; same isolated-spike audit caveat as
core 1. Final state (t_yr=2014.0) is the restart base for the MRI
projections (cores 4/6/8).
First projection of the matrix to run its full window: 2015-2100 from
the core-2 historical final, zero rescue events end to end (the MRI
forcing never produces a front-emptying era at 32 km within this
window). Budget residual closed throughout; audit table in-report.
Ensemble overlay lands with the CTRL pair (cores 9/10).
Full window in two legs: cold to the 2069.7 front-emptying wall, then a
warm resume that crossed it and ran out the window (17 rescued events
total). By 2100 the CESM ssp370 forcing has driven the system into a
surface-loss regime (net SMB -1987 Gt/yr, melt 5064 Gt/yr); VAF -265 mm
vs 2015. Budget residual closed throughout.
108+ years of ssp585 from the historical final - the configuration that
died at year 9 before the balanced-init/rescue work. 19 rescued events;
a warm resume in the deep-collapse era (net SMB -6659, melt 13,818
Gt/yr, VAF -584 mm vs 2015) buys only +1.3 yr: front-emptying eras are
continuous there and the split scheme saturates. Honest stop state
saved; the monolithic coupling port owns the 2124->2300 stretch.
Three more full 2015-2300 projections:
- Core 5 (ssp126 CESM): two legs across the 2056.4 wall.
- Core 6 (ssp126 MRI): one clean leg, no rescues.
- Core 8 (ssp585 MRI): 114 clean years to the 2129.0 wall, then a warm
  resume through the deep era to 2300 - where the CESM ssp585 twin
  (core 7) saturates at 2124.5, MRI's milder ocean lets the split
  scheme finish the window.
Budget residual closed on all three; per-core reports carry audits and
CTRL-differenced ensemble overlays.
- Core 9 (CTRL CESM): COMPLETE 2015-2300, the balanced control the
  projections difference against.
- Core 10 (CTRL MRI): PARTIAL to 2040.5 (split-scheme-edge wall its
  physical twin core 9 crossed; RACMO baseline makes them near-identical,
  so CTRL behavior is demonstrated by core 9).
- Core 11 (OCX): COMPLETE 1990-2025 obs-constrained; the real OCX
  forcing product (now on the share) is a follow-up swap.
Parallel line to antarctica running the composite viscous rheology at
standard Glen n=3 instead of n=4 Goldsby-Kohlstedt. n=3 is the
ISMIP6/ISMIP7 default and the more directly comparable configuration;
the n=4 branch is left untouched.

The n=4 assumption lived entirely in two env-var defaults, so this is a
small, self-documenting change:
- N_FLOW_DEFAULT 4.0 -> 3.0, A4_FACTOR_DEFAULT 10.0 -> 1.0 as module
  constants in simulation.py and inversion_icepack2.py. A0 =
  rate_factor(260 K) is already the n=3 fluidity, so the composite main
  term is plain Glen with no prefactor rescale (a4_factor=1); at n=4 the
  factor lifts A_3 to A_4 so A_4 tau_c^4 ~ A_3 tau_c^3 at tau_c.
- map_n_tag() gives n=3 MAPs an _n3 filename tag
  (inversion_icepack2_budd_n3_<lc>.h5) so they coexist on disk with the
  untagged n=4 MAPs; empty at n=4 (backward compatible). The forward and
  its inversion must use the same ISMIP7_N_FLOW / ISMIP7_A4_FACTOR.
- Launcher log/lock names n-tagged so n=3 and n=4 inversions don't
  collide on one machine.
- COMPOSITE_RHEOLOGY.md + antarctica/N3_FRAMEWORK.md document the branch.

Verified: syntax on all three scripts; map_n_tag() returns _n3 at the
n=3 default, '' at n=4; launcher dry-run emits budd_inv_n3 names.
Adopt the mismip_time-dependent-da (Recinos/fenics_ice) inversion method
to fix the n=3 theta/phi blow-up. The old n=3 MAP was ill-conditioned
(theta/phi to +-36 in observed ice; forward gave misfit 3.5e5 vs the
MAP's 1798, not forward-consistent) because the fluidity control
phi=log(A/A0) sat on a CONSTANT baseline, so phi had to carry all the
spatial fluidity structure and, without the n=4 a4x10 boost, exploded.

The fix regularizes the DEVIATION from a physical prior mean, not the
amplitude:
- icepack2_tools/thermo_model.py: depth-averaged enthalpy (Stefan) model
  ported from the mismip thermo_model, adapted for Antarctica with a
  spatially-varying surface-temperature BC + an under-relaxed fixed-u
  Picard driver (compute_fluidity_prior).
- icepack2_tools/prior.py: fenics_ice Whittle-Matern prior
  gamma*(theta^2 + L^2|grad theta|^2) - the theta^2 mass term (absent
  from the old gradient-only reg) removes the null space that let smooth
  large controls through.
- forcing.load_mean_annual_surface_temperature: ISMIP7 tas climatology.
- inversion: control phi=log(A/A_prior) on the thermomechanical prior
  mean (A4_base=A_prior, a Function; theta stays log(C/C_w0) on the
  balance anchor); WM reg at a physical gamma=1e4; A_prior saved in the
  MAP.
- forward (simulation.py): loads A_prior from the MAP and reconstructs
  A=A_prior*exp(phi); restart checkpoints carry it.
- antarctica/scripts/thermo_prior.py: standalone A_prior driver + figure.

Validated at 32 km (30-iter inversion): thermo prior converges (T
221-272 K, A 1-884), controls drop to theta [-2.7, 5.7], phi [-6.1,
0.64] (was +-36), misfit descends 7.6e3->5.7e3, and the forward
reproduces (setup_model initial misfit 6.7e3 vs the old 3.5e5; MAP clip
now touches 2 nodes, was ~600). Full-convergence inversion next.
… MAP)

At -n 16 on the tiny 32km mesh (~90 vertices/rank) MUMPS got fragile on
the stiff thermomechanical A_prior and the FIRST adjoint solve failed.
The adjoint/forward guards returned a large objective with a ZERO
gradient, which L-BFGS-B read as convergence at iteration 0 - it quit
and overwrote the good MAP with theta=phi=0 (misfit 1e11). Now a failure
on the first evaluation (iteration_count==0, no successful gradient yet)
raises a clear RuntimeError naming the likely cause (too many ranks for a
small mesh) instead of silently producing garbage. Mid-run failures still
backtrack as before. Fix: run small-mesh inversions at fewer ranks.
The full 32km n=3 Budd inversion with the thermomechanical fluidity prior
+ Whittle-Matern regularization converged (iter 534, misfit 4815,
n=4-comparable). Controls are physical - theta p99abs 2.93 (max 7.8),
phi p99abs 3.78 (max 5.9), vs the old p99 14-17 / max +-36 - and the
forward REPRODUCES the inversion (setup_model initial misfit 6191 vs the
old broken 3.5e5). Document the physical-prior method + run recipe in
N3_FRAMEWORK.md (few ranks for small meshes; ISMIP7_MAP_CLIP=10 at n=3).
The physical-prior inversion changed the fluidity baseline from the
constant a4_factor*A0 to the thermomechanical A_prior stored in the MAP:

- COMPOSITE_RHEOLOGY.md: composite spec now defines A = A_prior*exp(phi)
  with the legacy constant fallback noted, pointing to N3_FRAMEWORK.md
  (the owner of the prior method) instead of duplicating it.
- antarctica/README.md \S4: the default inversion now also needs the \S2
  ISMIP7 tas climatology (plus \S1 RACMO) for the thermo prior solve.
- inversion_icepack2.py header: note the physical-prior controls and
  Whittle-Matern regularization with a pointer to N3_FRAMEWORK.md.
- N3_FRAMEWORK.md: add ISMIP7_L_REG to the env list.

Lint (no linter configured; py_compile + AST unused-import scan on the
changed files): remove imports the change orphaned (grad, ds, dS in the
inversion after the regularization moved to icepack2_tools/prior.py)
plus pre-existing unused ones in the changed files.
@hoffmaao hoffmaao changed the title feat(antarctica): add ISMIP7 simulation framework with configurable n=3 flow law feat(antarctica): n=3 physical-prior inversion on the ISMIP7 simulation framework Jul 26, 2026
@hoffmaao

Copy link
Copy Markdown
Member Author

Superseded by #4, opened from the hoffmaao/Antarctica fork (identical commits at 298f33a). Moving this line to the fork-originated PR workflow.

@hoffmaao hoffmaao closed this Jul 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants