feat(antarctica): n=3 physical-prior inversion on the ISMIP7 simulation framework - #3
Closed
hoffmaao wants to merge 66 commits into
Closed
feat(antarctica): n=3 physical-prior inversion on the ISMIP7 simulation framework#3hoffmaao wants to merge 66 commits into
hoffmaao wants to merge 66 commits into
Conversation
The whole antarctica/mesh/ dir was gitignored to keep the large .msh/.h5 files out of git, but that also hid the 155-byte boundary_ids*.json files that every solver script reads by default (ISMIP7_BNDIDS -> calving/other physical-line ids for the calving-terminus BC). A fresh clone hit a missing default path. Switch the rule to 'antarctica/mesh/*' + a '!boundary_ids*.json' negation (a negation under the dir form is a no-op) and commit the four files. These encode the gmsh $PhysicalNames split: addPhysicalGroup() auto-numbers the alternating Calving_/Other_ segments 1,2,3,... so odd=calving, even=other. Caveat: boundary_ids_buffered.json (55 ids) matches only the 4km/8km buffered meshes; the finer buffered meshes have 35 ids (use boundary_ids.json). The buffered sidecar is not yet per-mesh. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…s generator - forcing.py: load_racmo_smb_climatology() reprojects RACMO ANT11 SMB to EPSG:3031 (rasterio) and samples to the mesh via icepack.interpolate - control/run.py: use RACMO climatology as the SMB baseline (acabf fallback), report area-weighted mean SMB - simulation.py: VAF/mass diagnostics use SI ice density (RHO_I_SI=917); the imported icepack rho_I is in MPa-m-yr units and was zeroing the output - mesh.py / mesh_antarctica.py: loaders take data_dir so they resolve antarctica/data instead of <project>/data - make_boundary_ids.py: generate boundary_ids.json from mesh $PhysicalNames Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The boundary-id sidecar only matches the specific mesh it was generated from (its segment count is a deterministic function of the BedMachine input and SIMPLIFY_TOL/SUBSAMPLE). Committing a fixed sidecar silently mismatches any mesh built from different inputs. Instead, make_boundary_ids exposes an importable write_boundary_ids(), and mesh_antarctica.py calls it after writing each mesh so the sidecar always matches the local mesh. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…orcing Reconciles our local work with the upstream ocean-melt branch. Conflict resolutions: - simulation.py: keep upstream's ocean_melt mass sink and h_clamp=0; take upstream's _RHO_I_SI/362.5 for the VAF/mass diagnostic (same fix as ours), dropping our redundant RHO_I_SI/GT_PER_MM_SLE duplicates. - control/run.py: RACMO climatology SMB baseline (ours) + per-basin ocean-melt callback (theirs); union imports; construct atm lazily in the acabf fallback. - forcing.py: our load_racmo_smb_climatology alongside their Burgard melt parameterization (auto-merged). boundary_ids policy: sidecars are now generated locally to match the mesh (mesh_antarctica.py emits one), so .gitignore ignores the whole mesh dir and upstream's four committed (35-segment) sidecars are dropped — they matched no mesh reproducible from the repo. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds make_synthetic_ocean_callback: prescribes TF(draft) = clip(tf_max * |draft|/depth_ref, 0, tf_max) and runs it through the Burgard quadratic_mixed_slope melt with constant salinity and the default scalar K. Needs no ocean/meltMIP data, so it works without the Globus download. Opt-in via ISMIP7_SYNTHETIC_MELT=1 (tunable with ISMIP7_SYNTH_TF_MAX / ISMIP7_SYNTH_DEPTH_REF); the real per-basin K path remains the default and is unchanged. Uncalibrated development stopgap, not protocol forcing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Debugging tooling for the diagnostic solver: - simulation.py: opt-in SNES/KSP convergence monitoring (ISMIP7_SNES_MONITOR), routed to a per-run file via ISMIP7_SNES_LOG (ascii:<file>) or stdout, so concurrent debug runs don't interleave. - control/run.py: --snes-monitor / --snes-log CLI flags that set those, plus a --restart <checkpoint.h5> option to resume a control run. - simulation.py: clamp the loaded thickness to h_clamp on restart, so a run never resumes below the floor (no-op for the default h_clamp=0). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- --tag (ISMIP7_RUN_TAG): suffix on experiment_name so concurrent/successive runs write distinct output files instead of overwriting each other. - --checkpoint-interval (ISMIP7_CHECKPOINT_INTERVAL): control how often thickness+velocity checkpoints are saved (e.g. frequent snapshots for debugging). Default 100, so existing behavior is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Default to antarctica_64000_500_aniso.msh (the mesh behind inversion_icepack2_500.h5) instead of the 1 km aniso mesh, raise the memory gate to 128 GB for the ~7M-DOF MUMPS factorization, and mark the script executable. This is the configuration that produced inversion_icepack2_rc_500.h5.
ISMIP7_FRICTION=regularized_coulomb makes setup_model() drive the prognostic system from the inversion_icepack2_rc_<lc>.h5 MAP: - geometry (thickness/bed/surface) and velocity_obs are taken from the checkpoint itself, so the Weertman anchor C_w0 — and hence the meaning of the inverted theta — is reproduced exactly; a node-ordering check guards the dof copy (also protects the existing Budd path). - the diagnostic solves build_rc_residual on the LIVE prognostic (h, s): grounded gate, Coulomb cap c0*N, and driving stress all track the evolving geometry, so the grounding line migrates with exact-zero shelf drag and no thickness clamp anywhere (h_clamp_init defaults to 0 in RC mode; h_visc_floor=10 m handles the ice-free buffer). - composite alpha defaults to the inversion's 1e-2 in RC mode so the forward diagnostic is consistent with the MAP. - setup_model now reports the initial velocity misfit vs observations (the rc_500 checkpoint predates its final report, so this is where we learn how converged that MAP is). Verified at LC=32000 against the completed RC inversion: the forward reproduces the checkpoint's saved velocity to rel L2 = 3.2e-9, and a short dt=0.1 clamp-free prognostic runs cleanly.
The acabf-anomaly files are referenced to the ESM's 1960-1989 climatology, and no historical scenario exists in the local data tree — so the projection driver's anomaly-only SMB was missing the entire climatological accumulation, while the CTRL driver's climatology lookup silently fell back to zero SMB. Fix both sides consistently: - ssp585 driver: force with the full acabf(t) field (smb_anomaly=False, new make_forcing_callback knob). Against a CTRL held at the 2015-2029 acabf mean, projection minus control is the forced signal relative to the 2015-centered climate with no anomaly re-referencing. - control driver: compute the reference climatology from full acabf pooled over historical + projection scenarios in an env-configurable window (default 2015-2029, the locally available years); raise instead of silently running a zero-SMB control. - both drivers pick up the calibrated per-basin K npz (falling back to the 2500 m calibration — 16 basin scalars remapped via the IMBIE2 8 km grid, mesh-independent); ssp585 default dt is now 0.1 per the May dt sweep. - launch_rc_ctrl_when_ready.sh: resource-gated launcher for the 500 m regularized-Coulomb CTRL2015 verification run (dt=0.1, 2015-2025, 24 ranks), logging under antarctica/results/logs/ so run logs survive reboots. End-to-end tested at LC=32000 through control/run.py with ISMIP7_FRICTION=regularized_coulomb.
Bring Dan's fixes into the antarctica branch: RACMO2.4p1 SMB climatology baseline (load_racmo_smb_climatology), SNES/KSP convergence monitoring with per-run log routing, restart/argparse tooling in the control driver, the synthetic depth-dependent melt stopgap (ISMIP7_SYNTHETIC_MELT=1), and the boundary-ids generator (make_boundary_ids.py; the static boundary_ids*.json leave tracking). control/run.py resolution composes both sides: RACMO stays the primary CTRL baseline (area-weighted mean report), the fallback is the pooled historical+projection full-acabf climatology (anomaly means are wrt 1960-1989 and are not a baseline), and a missing climatology is a hard error instead of a silent zero-SMB control (ISMIP7_ALLOW_ZERO_SMB=1 to override). The OI-climatology + per-basin calibrated K path remains the default ocean forcing.
SMB(t) = RACMO_clim + [aSMB(t) - mean(aSMB over the reference window)]: the acabf-anomaly is referenced to the ESM's 1960-1989 climatology, so re-referencing over the CTRL window (2015-2029 of it exists locally) removes that mismatch, and the RACMO baseline keeps projection and control on the same absolute SMB. Falls back to full acabf(t) when RACMO is absent. make_forcing_callback grows an smb_baseline parameter (per-node array added to the anomaly each step). Also: run_simulation nsteps now rounds instead of truncating (a 285-yr dt=0.1 run silently lost its final step), and the RC CTRL launcher verifies boundary_ids.json exists (untracked on this branch) and pins ISMIP7_BNDIDS for the run.
front, and an observation mask for the inversion Investigating the mass gain Dan saw. A new per-step mass audit in run_simulation attributes dM to SMB - melt - boundary outflux - calving and exposes any created mass; at 32 km it immediately caught two real leaks in the pre-existing scheme: - the -h*div(u*phi) volume term is non-conservative for DG0 (spurious h*div(u) mass at thickness jumps; +4700 Gt/yr at the test front). For piecewise constants the facet upwind terms alone are the finite-volume scheme, so the volume term is now dropped by default. - the L2 DG0->CG1 projection undershoots negative at steep fronts and the h floor then injects mass (+7600 Gt/yr step-1 at 32 km). Replaced by a lumped-mass projection (convex combination of adjacent cell values: bounded, no floor injection, integral-preserving). With both fixes the audited budget closes to machine precision (resid = 0.00, clamp = 0.0-0.2 Gt/yr). ISMIP7_LEGACY_TRANSPORT=1 restores the old scheme for comparison. ISMIP7_FIXED_FRONT=1 (now set by the RC CTRL launcher) removes ice that flows past the initially ice-free cells and tallies it as calving flux - without it a buffered mesh has no calving sink (~1300 Gt/yr in reality) and must gain mass. Only meaningful with a true-geometry initial state (RC mode). The inversion misfit now excludes nodes without velocity observations (MEaSUREs ERR > 0 defines coverage; icepack zero-fills NODATA, which read as "observed stationary" and prescribed friction where there is no constraint). Coverage gaps are small - 0.7% of grounded, 0.5% of floating area - so this is rigor, not the mass-gain culprit. obs_mask is saved in inversion checkpoints; ISMIP7_OBS_MASK=0 reverts.
get_thermal_forcing/get_salinity now load one (variable, year) slice out of its decadal chunk file into an in-memory nearest-neighbour RegularGridInterpolator over (z, y, x), cached across steps. The previous pointwise xarray .interp over the open_mfdataset dask graph cost ~10 minutes per step even at 5k mesh nodes; the new path reads a year in ~2 s and answers in ~1 ms, and is verified bit-identical to an xarray nearest-neighbour reference over 2000 random points and drafts. Nearest-neighbour also matches the OI-climatology CTRL path and the per-basin K calibration. A 32 km SSP5-8.5 step now costs the same as a CTRL step (~10 s after warm start, was ~675 s). ISMIP7_K_SCALE multiplies the melt K in both the projection callback and the CTRL driver (the calibrated per-basin K integrates 689 vs 865 Gt/yr observed on the 2500 m mesh; 1.26 matches the observed total).
The front mask was derived from the run's starting h, so a restarted run (the CTRL -> projection chain) would re-mask cells that legitimately retreated mid-run inside the observed extent and forbid them from re-advancing. setup_model now exposes the t=0 BedMachine/inversion thickness as ctx["H_init"] (restart_from does not overwrite it) and run_simulation builds the mask from that. Cold-start behavior is unchanged (verified identical at 32 km).
Local working-notes patterns belong in each clone's .git/info/exclude rather than the shared .gitignore, and the README's writeup pointer now names only the tracked document.
All ESM-forced core experiments (historical 1/2, SSP projections 3-8) are the same run with different (esm, scenario, period), so the nine drivers are now thin shims over experiment.run_core_experiment. That puts every fix from this week's work in force for the whole matrix at once: RACMO baseline + aSMB re-referenced over one shared pool (historical + ISMIP7_CLIM_SCENARIO, 2000-2029) so all cores and the CTRL share a single baseline and the 2014/2015 historical->projection handoff is seamless; calibrated per-basin K with the 2500 m fallback; dt=0.1; hard fast-failure (seconds, before the expensive model setup) when the forcing tree for an (ESM, scenario) is not downloaded, instead of silently running anomaly-only/zero-SMB. OCX (core 11) is rebuilt as a genuinely observation-forced run: RACMO actual-year SMB (1979-2023) + constant OI-climatology ocean with per-basin K — runnable entirely from local data. The OI-climatology ocean callback moves from control/run.py into icepack2_tools.forcing (shared by CTRL and OCX). Verified at 32 km: the ssp585 shim reproduces the standalone driver's results exactly; OCX runs with a closed mass budget; missing-data drivers (historical, ssp370) fail in ~7 s with the download hint. Co-authored work retained from prior drivers; fracture masks are loaded when present but still not applied to the thickness update (open protocol item).
Gate log, run log, and lockfile move from /tmp (wiped on reboot — the Jun 20 rc_500 inversion log was lost that way) to the repo results tree, tagged by resolution so a 2500 m and a 500 m gate can coexist.
The AIS_ocean/share_with_modellers tree is gone; data now lives under CMIP6_test_protocol/AIS mirroring the runtime layout, so downloads land directly in ISMIP7/AIS/ (no rename step, closing the README caveat). Auth is now headless-first: the script's own refresh-token cache, then the globus CLI token store (~/.globus/cli/storage.db keeps a long-lived transfer refresh token + the per-user confidential client), then an interactive login that now requests refresh tokens so it is one-time. The destination endpoint defaults to the Globus Connect Personal id registered on the machine. Transfers verify checksums. Manifest updated to what the share actually carries today: the SDBN1 1960-1989 long-term means (the exact reference climatology of the acabf-anomaly files), ocean ct/sa climatology+bias, the newer 06_nov OI climatology (1972-2024), the ISMIP7 obs kit, IMBIE v3, grid, and topography. NOT yet published upstream: historical yearly forcing, processed ssp126/ssp370, all of MRI-ESM2-0 — and the processed ssp585 tree we hold locally is no longer on the share (treat it as irreplaceable). Co-existing raw CMIP output lives in ../CMIPraw for when processing becomes necessary.
- build_oi_climatology_interpolators grows ISMIP7_OI_VERSION (default 30_sep, matching the per-basin K calibration; 06_nov selects the reorganized share's 1972-2024 re-release). The 06_nov so/thetao files currently ship the tf field by mistake (upstream packaging bug, reported) - the reader refuses them with an explanatory error instead of silently melting with TF-as-salinity. - load_K_per_basin falls back to the IMBIE v3 basin file when the calibration-era v2 is absent (verified identical basinNumber field, so the calibrated per-basin K remains valid with either). - scripts/preflight.py: five-second per-core readiness table checking every input each experiment needs (mesh, MAP, boundary ids, K, RACMO, OI climatology, atmosphere/ocean coverage over the run period), honoring the same env knobs as the runs. Current verdict: cores 7, 9, 11 READY (core 10 = core 9 while RACMO is the baseline); 1-6, 8 blocked on unpublished upstream forcing.
Re-walked the whole GHub tree to diagnose what climate forcing is pullable. The collection is mid-reorganization: CMIPraw is now gone entirely, AIS/MRI-ESM2-0 does not exist, and AIS/CESM2-WACCM holds only climatologies + bias ingredients (no scenario dirs, no per-year forcing). Everything still pullable is already mirrored locally, so the downloader is a no-op. The processed CESM2-WACCM/ssp585 forcing we run from (133 GB) is no longer on the share — flag the local copy as irreplaceable.
The 500 m RC CTRL crashed in setup_model's initial n=1->4 continuation (DIVERGED_MAX_IT): a fine mesh + a rough mid-optimization MAP outruns Newton across the 5-step ramp, and unlike the timestep loop the initial solve had no retry. Now it restores the initial guess and re-ramps with 2x then 4x steps on divergence (ISMIP7_CONTINUATION_STEPS, default 8), and reports the step count that worked. 32 km reproduces the identical initial misfit and VAF/mass through the new path (no regression).
On the 500 m mesh each diagnostic solve is costly, so seed the adaptive continuation finer (16) rather than failing at 8 and re-ramping.
Diagnosis of the 500 m CTRL divergence: the rc_500 MAP (a 20-iter mid-optimization checkpoint) carries ~700 theta/phi nodes with |.| up to 23, i.e. exp() coefficients ~1e10 that are local singularities the SNES cannot solve through — not a step-size issue. A converged MAP is O(1) everywhere (rc_32000 maxes at 1.64). setup_model now clips theta/phi to |.|<=ISMIP7_MAP_CLIP (default 6) and logs the count, removing the pathology (0.1% of nodes) without touching real structure. No-op for converged MAPs. Pairs with the adaptive continuation to unblock the 500 m cold start pending a proper re-inversion.
Adds a minimum-yield shelf-drag knob to the RC forward and records the multi-step instability diagnosis. The RC diagnostic velocity blows up ~2x/step in prognostic use (global, hypersensitive to sub-percent geometry change) — confirmed independent of transport scheme (legacy and conservative both blow up), independent of the fixed calving front (on/off identical), and NOT curable by shelf drag (tested up to 50 kPa, already enough to defeat RC's purpose). Budd ran stably for 10-20 steps in May, so this is RC-specific: exact-zero shelf drag + composite viscosity leaves the mixed diagnostic system near-singular as h evolves. The knob and MAP clip / adaptive continuation stay as robustness options; the forward prognostic needs Budd (proven stable) or a monolithic implicit coupling.
Following the pivot away from the prognostically-unstable RC forward: a resource-gated 2500 m Budd CTRL2015 validation run with the full conservation pipeline (RACMO SMB, OI ocean + per-basin K, conservative transport, fixed front, h_clamp_init=0, dt=0.1). Also confirms the Budd + fixed-front + h_clamp_init=0 config (new since May, which used h_clamp_init=10); a non-converging run self-terminates in ~10 steps so gating is safe. The rc_ctrl launcher is superseded for forward use.
- download_forcing.py now mirrors straight into ISMIP7/AIS (drop the layout-mismatch caveat + known issue; new remote base path; point at the script's status comment for the upstream scenario-data gap) - boundary_ids sidecars: auto-generated by mesh_antarctica.py / make_boundary_ids.py, and mesh/ is now fully gitignored - friction-tagged MAP checkpoints (ISMIP7_FRICTION, _budd/_rc names) - control driver restart flags (--tag, --checkpoint-interval, ISMIP7_AUTO_RESUME), self-contained (u, M, tau) checkpoints - timeseries CSV schema: full mass-budget columns incl. amb_gtyr - env-knob table: checkpoint cadence, APPARENT_MB, FIXED_FRONT, LEGACY_TRANSPORT, SNES_TYPE/MAXIT - known issues: blow-up fixed (audits ON TRACK); remaining item is the diagnostic-Newton wall on hard projection geometries - check_ismip6_track.py audit documented under §6 - COMPOSITE_RHEOLOGY.md defaults updated for the residual friction laws - data/README reduced to a pointer at the §1 dataset table
The mid-July GHub "shrinking collection" was the reorganization in
flight: the data MOVED to a new top-level /ISMIP7 tree (earlier walks
started inside /ISMIP6/ISMIP7_Prep and never looked up). /ISMIP7/AIS
holds the full scenario matrix for BOTH ESMs (ctrl, ctrlclim,
historical, ssp126, ssp370, ssp534-over, ssp585; per-year SDBN1-8000m
and 2000m atmosphere, decadal-chunk ocean so/tf/thetao, fracture),
plus cmipraw, the real OCX forcing product, SMBmip, submission
templates, AntarcticaObsISMIP7-v1.2, and a FIXED 06_nov OI climatology
(so/thetao v4; the old tree's v3 so/thetao are byte-identical to tf -
the packaging bug we reported).
Add --scenarios [--esm ...] [--scenario ...]: mirrors the minimal
runtime sets (SDBN1-8000m acabf + acabf-anomaly, ocean tf + so,
fracture) per (ESM, scenario) as recursive-dir Globus transfers with
version autodetection and checksum sync (idempotent - re-run to check
completeness or fill gaps). Local CESM2-WACCM/ssp585 verified complete
against the new tree, so the local copy is no longer irreplaceable.
First submission: CESM {historical, ssp126, ssp370} + MRI {historical,
ssp126, ssp370, ssp585} + fracture masks, ~130 GB.
… fracture lookup, docs sync
The Jul 18 2500m Budd inversion died ~11 hr in (iter 60, misfit already down 3.07e4 -> 5.2e3): tlm_adjoint's adjoint_jacobian_solve hit the SNES iteration cap at rough mid-optimization controls, and only the FORWARD solve was guarded. Wrap compute_gradient in the same recovery - restore the backup state and return an inflated objective with a zero gradient so L-BFGS-B backtracks its line search instead of crashing the run. Relaunched warm from the periodic checkpoint with this guard in place.
…rt guards, forcing caches, preflight
compare_ismip6.py overlays our (projection - CTRL) sea-level contribution on the Seroussi et al. 2020 computed scalars (~/data/ismip6 local mirror: per-model computed_ivaf_minus_ctrl_proj files, ivaf m^3, in-file rho). Pools RCP8.5-class experiments by default (exp01-05, 45 members), prints ours vs ensemble p5/median/p95/min/max at marker years, exits 0 iff we end inside the min-max envelope - the sharper companion to check_ismip6_track.py's observational audit. First result (32 km, 9.4-yr ssp585 vs balanced CTRL): ours -8.1 mm SLE at 2024.5 vs ensemble [-1.9, +6.9] - OUTSIDE, on the SMB-gain side. Consistent with (a) the pending ~9% aSMB unit-inflation finding and (b) CMIP6 CESM2-WACCM's stronger Antarctic SMB response vs the CMIP5 forcings behind ISMIP6 - quantitative motivation for resolving the units question before production runs.
run_simulation's step solve now escalates through a rescue ladder before giving up: direct -> fresh continuation -> trust-region direct -> trust-region continuation, restoring the last CONVERGED mixed state between attempts (and at final stop, so checkpoints never hold a diverged Newton iterate - closes the earlier review note). Based on the 2024.5 ssp585 wall evidence that newtontr converges steps nleqerr cannot. Validated at 32 km: a healthy 2-yr CTRL never triggers it (behavior unchanged), and the cold ssp585 wall reproducer now runs to 2025.9 with a demonstrated step-107 trust-region recovery before a graceful stop (was 2024.4). The deep ssp585 hard era remains open (monolithic coupling is the structural fix). core_report.py generates the tracked per-core run record for antarctica/reports/: run environment, budget rows at marker years, the check_ismip6_track audit, the compare_ismip6 ensemble overlay for projections, and provenance (git SHA, log, gitignored CSV path). Each of the 11 core experiments lands as one commit whose payload is this report.
Wall localization (32 km historicals, both ESMs stopping at ~1873; same class as the ssp585 2024.5 wall): the unconvergeable residual is carried ~100% by a handful of floating h=0-60 m front nodes (Amery sector) whose frozen-iterate speeds reach 1e10-1e11 m/yr - the gia frictionless-front runaway, but above the h<10 m ocean-drag band, and only at hard-era geometries. Mechanism: the gia soft speed limiter is now structurally present (threshold ISMIP7_U_LIM, default 2e4 m/yr) but INERT - its coefficient is a live Constant at 0, so the term and Jacobian vanish identically and the cold-start continuation is unaffected (a built-in limiter provably breaks it). The step rescue ladder gains two deepest rungs that raise the coefficient to ISMIP7_K_LIM under trust region for the rescue solve only, then zero it: physical flow (<< u_lim) never feels the limiter, and it exists exactly at wall geometries where the runaway nodes need pinning. Restart fast path (setup_model): re-solve directly at the loaded converged state with normal tolerances instead of re-ramping n from 1 (the ramp is fatal at wall geometries); if even that fails, keep the loaded state, set a TIGHT run tolerance (1e-6 x the loaded-state residual), and hand the hard step to the run's rescue ladder. The first version of this path let its loose acceptance atol persist, silently freezing the velocity for the rest of the run (0.2 s "steps") - that invalidated one resume experiment and is exactly what the tight-scale rule prevents. Validated: 1-yr balanced CTRL regression is unchanged (no rescue fires, limiter inert). The historical cores rerun cold with this machinery next; their per-core reports record whether the 1873 era clears.
The 1873 hard eras are front-cell-emptying EVENTS in sequence: the rescue ladder alone fails them at dt=0.1, but a dt=0.025 approach from the 1870 checkpoint crossed the 1873.3 event with four consecutive trust-region+limiter recoveries (then walled at the next event) - small geometry increments let Newton track the branch through each event. run_simulation's step is now transport-first (advance with the velocity solved AT the current geometry, then solve at the new geometry - the same (G, u) sequence as before, with the bonus that checkpoints store a mutually consistent (h, u) pair). A step whose post-advance solve exhausts the rescue ladder rewinds ITS OWN advance (h_dg/z entry buffers) and retries it as dt/4 then dt/16 subcycles with full rescue solves between substeps (ISMIP7_SUBCYCLES, default 1,4,16). Budget tallies (outflux/calving/clamp/limit) accumulate over substeps into the step's row, so the audit residual still closes exactly. Validated: the 2-yr balanced CTRL is unchanged (flat VAF, resid 0.0, ON TRACK, no rescue fires). Historical cores rerun next; their reports record how the 1873+ era sequence resolves.
First core experiment of the matrix run end-to-end: full 1850-2014 window from the balanced init, 13 front-cell-emptying events crossed by the dt-subcycle rescue ladder. Century-scale behavior is sound: SMB 2520, shelf melt 1371, front discharge 1050 Gt/yr means; dM/dt -314 Gt/yr; dVAF/dt -0.5 mm SLE/yr; budget residual <= 0.1 Gt/yr over 1640 steps. The single audit FAIL is the runaway detector's absolute-peak clause on an isolated one-step discharge spike during an emptying event (tallied, budget closed - a different signature from the sustained exponential growth the clause was built for; left as-is pending a ruling on refining the detector). Final state (t_yr=2014.0) is the restart base for the CESM2-WACCM projections (cores 3/5/7).
Second core of the matrix, on the newly mirrored MRI forcing. Completed in two legs: cold to 1876.9, then a warm resume whose x64 subcycle rung crossed the 1877 Amery front-emptying event (a single 553 s step) and cruised to 2014. Inter-ESM contrast established at 32 km: MRI gains ~+88 mm VAF over the window (stronger SMB) where CESM loses ~-87 mm. Budget residual closed throughout; same isolated-spike audit caveat as core 1. Final state (t_yr=2014.0) is the restart base for the MRI projections (cores 4/6/8).
First projection of the matrix to run its full window: 2015-2100 from the core-2 historical final, zero rescue events end to end (the MRI forcing never produces a front-emptying era at 32 km within this window). Budget residual closed throughout; audit table in-report. Ensemble overlay lands with the CTRL pair (cores 9/10).
Full window in two legs: cold to the 2069.7 front-emptying wall, then a warm resume that crossed it and ran out the window (17 rescued events total). By 2100 the CESM ssp370 forcing has driven the system into a surface-loss regime (net SMB -1987 Gt/yr, melt 5064 Gt/yr); VAF -265 mm vs 2015. Budget residual closed throughout.
108+ years of ssp585 from the historical final - the configuration that died at year 9 before the balanced-init/rescue work. 19 rescued events; a warm resume in the deep-collapse era (net SMB -6659, melt 13,818 Gt/yr, VAF -584 mm vs 2015) buys only +1.3 yr: front-emptying eras are continuous there and the split scheme saturates. Honest stop state saved; the monolithic coupling port owns the 2124->2300 stretch.
Three more full 2015-2300 projections: - Core 5 (ssp126 CESM): two legs across the 2056.4 wall. - Core 6 (ssp126 MRI): one clean leg, no rescues. - Core 8 (ssp585 MRI): 114 clean years to the 2129.0 wall, then a warm resume through the deep era to 2300 - where the CESM ssp585 twin (core 7) saturates at 2124.5, MRI's milder ocean lets the split scheme finish the window. Budget residual closed on all three; per-core reports carry audits and CTRL-differenced ensemble overlays.
- Core 9 (CTRL CESM): COMPLETE 2015-2300, the balanced control the projections difference against. - Core 10 (CTRL MRI): PARTIAL to 2040.5 (split-scheme-edge wall its physical twin core 9 crossed; RACMO baseline makes them near-identical, so CTRL behavior is demonstrated by core 9). - Core 11 (OCX): COMPLETE 1990-2025 obs-constrained; the real OCX forcing product (now on the share) is a follow-up swap.
Parallel line to antarctica running the composite viscous rheology at standard Glen n=3 instead of n=4 Goldsby-Kohlstedt. n=3 is the ISMIP6/ISMIP7 default and the more directly comparable configuration; the n=4 branch is left untouched. The n=4 assumption lived entirely in two env-var defaults, so this is a small, self-documenting change: - N_FLOW_DEFAULT 4.0 -> 3.0, A4_FACTOR_DEFAULT 10.0 -> 1.0 as module constants in simulation.py and inversion_icepack2.py. A0 = rate_factor(260 K) is already the n=3 fluidity, so the composite main term is plain Glen with no prefactor rescale (a4_factor=1); at n=4 the factor lifts A_3 to A_4 so A_4 tau_c^4 ~ A_3 tau_c^3 at tau_c. - map_n_tag() gives n=3 MAPs an _n3 filename tag (inversion_icepack2_budd_n3_<lc>.h5) so they coexist on disk with the untagged n=4 MAPs; empty at n=4 (backward compatible). The forward and its inversion must use the same ISMIP7_N_FLOW / ISMIP7_A4_FACTOR. - Launcher log/lock names n-tagged so n=3 and n=4 inversions don't collide on one machine. - COMPOSITE_RHEOLOGY.md + antarctica/N3_FRAMEWORK.md document the branch. Verified: syntax on all three scripts; map_n_tag() returns _n3 at the n=3 default, '' at n=4; launcher dry-run emits budd_inv_n3 names.
… stale n=4 comments
Adopt the mismip_time-dependent-da (Recinos/fenics_ice) inversion method to fix the n=3 theta/phi blow-up. The old n=3 MAP was ill-conditioned (theta/phi to +-36 in observed ice; forward gave misfit 3.5e5 vs the MAP's 1798, not forward-consistent) because the fluidity control phi=log(A/A0) sat on a CONSTANT baseline, so phi had to carry all the spatial fluidity structure and, without the n=4 a4x10 boost, exploded. The fix regularizes the DEVIATION from a physical prior mean, not the amplitude: - icepack2_tools/thermo_model.py: depth-averaged enthalpy (Stefan) model ported from the mismip thermo_model, adapted for Antarctica with a spatially-varying surface-temperature BC + an under-relaxed fixed-u Picard driver (compute_fluidity_prior). - icepack2_tools/prior.py: fenics_ice Whittle-Matern prior gamma*(theta^2 + L^2|grad theta|^2) - the theta^2 mass term (absent from the old gradient-only reg) removes the null space that let smooth large controls through. - forcing.load_mean_annual_surface_temperature: ISMIP7 tas climatology. - inversion: control phi=log(A/A_prior) on the thermomechanical prior mean (A4_base=A_prior, a Function; theta stays log(C/C_w0) on the balance anchor); WM reg at a physical gamma=1e4; A_prior saved in the MAP. - forward (simulation.py): loads A_prior from the MAP and reconstructs A=A_prior*exp(phi); restart checkpoints carry it. - antarctica/scripts/thermo_prior.py: standalone A_prior driver + figure. Validated at 32 km (30-iter inversion): thermo prior converges (T 221-272 K, A 1-884), controls drop to theta [-2.7, 5.7], phi [-6.1, 0.64] (was +-36), misfit descends 7.6e3->5.7e3, and the forward reproduces (setup_model initial misfit 6.7e3 vs the old 3.5e5; MAP clip now touches 2 nodes, was ~600). Full-convergence inversion next.
… MAP) At -n 16 on the tiny 32km mesh (~90 vertices/rank) MUMPS got fragile on the stiff thermomechanical A_prior and the FIRST adjoint solve failed. The adjoint/forward guards returned a large objective with a ZERO gradient, which L-BFGS-B read as convergence at iteration 0 - it quit and overwrote the good MAP with theta=phi=0 (misfit 1e11). Now a failure on the first evaluation (iteration_count==0, no successful gradient yet) raises a clear RuntimeError naming the likely cause (too many ranks for a small mesh) instead of silently producing garbage. Mid-run failures still backtrack as before. Fix: run small-mesh inversions at fewer ranks.
The full 32km n=3 Budd inversion with the thermomechanical fluidity prior + Whittle-Matern regularization converged (iter 534, misfit 4815, n=4-comparable). Controls are physical - theta p99abs 2.93 (max 7.8), phi p99abs 3.78 (max 5.9), vs the old p99 14-17 / max +-36 - and the forward REPRODUCES the inversion (setup_model initial misfit 6191 vs the old broken 3.5e5). Document the physical-prior method + run recipe in N3_FRAMEWORK.md (few ranks for small meshes; ISMIP7_MAP_CLIP=10 at n=3).
…rnings; dedupe anchor
The physical-prior inversion changed the fluidity baseline from the constant a4_factor*A0 to the thermomechanical A_prior stored in the MAP: - COMPOSITE_RHEOLOGY.md: composite spec now defines A = A_prior*exp(phi) with the legacy constant fallback noted, pointing to N3_FRAMEWORK.md (the owner of the prior method) instead of duplicating it. - antarctica/README.md \S4: the default inversion now also needs the \S2 ISMIP7 tas climatology (plus \S1 RACMO) for the thermo prior solve. - inversion_icepack2.py header: note the physical-prior controls and Whittle-Matern regularization with a pointer to N3_FRAMEWORK.md. - N3_FRAMEWORK.md: add ISMIP7_L_REG to the env list. Lint (no linter configured; py_compile + AST unused-import scan on the changed files): remove imports the change orphaned (grad, ds, dS in the inversion after the regularization moved to icepack2_tools/prior.py) plus pre-existing unused ones in the changed files.
Member
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Push the n=3 physical-prior ice-sheet inversion work to my personal fork (hoffmaao/Antarctica, a fork of icepack/ismip7) so I can merge it with the icepack ismip7 upstream. The branch antarctica-n3 is the standard-Glen n=3 parallel line to the n=4 antarctica branch. Since the earlier framework commits (through 6da0a92) were already gated as PR #3, the NEW work to focus review on is the last three commits: (1) 6ae0255 - the core n=3 physical-prior inversion method ported from mismip_time-dependent-da (Recinos/fenics_ice): the fluidity control is now phi=log(A/A_prior) on a thermomechanical prior mean (icepack2_tools/thermo_model.py, a depth-averaged enthalpy/Stefan solve adapted for Antarctica with a spatially-varying tas surface-temperature BC and an under-relaxed fixed-velocity Picard loop), the friction control stays theta=log(C/C_w0) on the balance anchor, and the regularization is the fenics_ice Whittle-Matern form gamma*(theta^2+L^2|grad theta|^2) (icepack2_tools/prior.py) at a physical gamma=1e4 - this regularizes the DEVIATION from a physical prior, not the amplitude, which fixed the n=3 blow-up (controls went from +-36 to physical p99~3-4); the forward (simulation.py) loads A_prior from the MAP and rebuilds A=A_prior*exp(phi), with restart checkpoints carrying it; (2) 8c5f2e2 - a guard so a first-iteration solve failure raises instead of silently writing a theta=phi=0 garbage MAP (this happened at -n 16 where the tiny 32km mesh over-decomposed MUMPS; the fix plus running small meshes at fewer ranks resolved it); (3) 87b3d4f - N3_FRAMEWORK.md doc + run recipe. Validated end-to-end at 32km: thermo prior converges (T 221-272K), the converged inversion has physical controls and n=4-comparable misfit 4815, and the forward reproduces it (setup_model initial misfit 6191 vs the old broken 3.5e5). Deliberate choices: the two files intentionally duplicate small map_n_tag/N_FLOW constants with keep-in-sync notes (matching the existing pattern); thermo params (shear_amp=30, duval=181) are the mismip values and slightly over-soften a few margins (a noted, non-blocking tuning item); A_prior is computed inside the inversion on its own mesh to avoid cross-mesh dof-copy issues. Remotes are now origin=hoffmaao/Antarctica (my fork), upstream=icepack/ismip7.
What Changed
phi = log(A/A_prior)on a thermomechanical prior mean (icepack2_tools/thermo_model.py, a depth-averaged enthalpy solve with spatially-varying tas surface temperature and an under-relaxed Picard loop), with fenics_ice-style Whittle-Matern regularization (icepack2_tools/prior.py) at gamma=1e4 penalizing deviation from the prior rather than amplitude; the forward (antarctica/scripts/simulation.py) loadsA_priorfrom the MAP checkpoint and rebuildsA = A_prior*exp(phi), with restart checkpoints carrying it.icepack2_tools(dual friction, forcing readers, regridding, coupled transport), gated CTRL/RC launchers, the shared core-experiment runner, and per-core reports covering 9/11 core experiments at 32 km (N3_FRAMEWORK.mdandCOMPOSITE_RHEOLOGY.mddocument the method and run recipe).ISMIP7_MAP_CLIPdefault is n-aware (10 for n=3, 6 otherwise), and the thermo-prior non-convergence and legacy constant-baseline fallback paths warn loudly (the latter three applied as review auto-fixes in the gate pipeline).Risk Assessment
✅ Low: The fix commit 63f9dbc is small and mechanical, implements all four requested round-1 fixes exactly as instructed (n-aware clip default with doc sync, loud Picard non-convergence warning, legacy-baseline log line, deduplicated anchor) without touching numerical method defaults, and introduces no new issues.</risk_rationale>
Review-only pass per pipeline instructions (no tests run): diffed 87b3d4f..63f9dbc, confirmed the worktree is clean at HEAD, verified each fix against the round-1 instructions (n-aware clip logic and N_FLOW_DEFAULT resolution, for/else semantics of the Picard warning, symmetry of the fallback print, hoisted weertman_anchor with branch-specific prints intact), and checked the new lines for fresh edge cases (float parsing of ISMIP7_N_FLOW, float(A0) on a Constant, dA scope in the for/else).
Testing
Ran a purpose-built synthetic validation harness in the Firedrake 2026 venv covering all new modules and script changes from the three commits: thermo fluidity prior convergence and physicality (with figure), Whittle-Matern prior analytic and gradient correctness, the tas climatology reader against a fake ISMIP7 data tree, MAP fluidity_prior checkpoint round-trip with legacy fallback, and import/constant-sync smoke of both scripts - all 12 checks passed; the real 32 km data run was not reproducible in the gated worktree and the fail-loud guard was verified by inspection.
/tmp/no-mistakes-evidence/01KYEAN0Q31W85QW09Z68D94J0/thermo_prior_synthetic.png)Evidence: Full validation log (12/12 checks passed)
[PASS] scalar-T_srf prior in physical band (A in [5.28, 129.51]) [PASS] varying-T_srf prior in physical band (A in [1.00, 82.96]) [PASS] colder surface -> stiffer ice (lower A_prior) (flank mean A: cold third 1.00 < warm third 21.68) [PASS] R(theta) matches analytic sine-field energy (rel err 6.18e-04) [PASS] gradient form == Gateaux derivative of energy (rel 1.1e-10) [PASS] tas reader: monthly mean + nearest-neighbour sampling (max err 0.500 K) [PASS] tas reader: seasonal cycle averaged out (T in [230.0, 270.0] K) [PASS] tas reader: ocean/NODATA masked to fill_K=260 [PASS] checkpoint carries A_prior; forward rebuild A=A_prior*exp(phi) (max abs err 1.42e-14) [PASS] missing fluidity_prior raises (legacy-MAP fallback path) (RuntimeError) [PASS] N_FLOW/A4_FACTOR defaults in sync (inversion vs forward) (n=3.0, a4=1.0) [PASS] map_n_tag: n=3 default tags MAPs '_n3'; n=4 keeps legacy untagged name [PASS] Whittle-Matern gamma defaults are the physical 1e4 RESULT: ALL CHECKS PASSEDEvidence: Synthetic validation harness (reproducible)
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
⏭️ **Rebase** - skipped
antarctica/scripts/control/run.py- merge conflict rebasing onto origin/mainantarctica/scripts/simulation.py- merge conflict rebasing onto origin/main🔧 **Review** - 6 issues found → auto-fixed ✅
antarctica/scripts/simulation.py:336- Default ISMIP7_MAP_CLIP=6.0 clips legitimate n=3 controls: N3_FRAMEWORK.md states the physical n=3 controls reach ~8 and instructs operators to set ISMIP7_MAP_CLIP=10 manually, but no code, launcher, or restart guard enforces this. A forward run (or a restart of a run originally launched with MAP_CLIP=10 but resumed with default env) silently clips valid theta/phi to |.|<=6, distorting the validated n=3 physics with only a log line. The restart guards check friction law and apparent-MB consistency but not the clip. Consider an n-aware default (e.g. 10 when ISMIP7_N_FLOW=3), or storing the control range / clip value in the MAP checkpoint and guarding on it like friction.antarctica/scripts/inversion_icepack2.py:107- The new inversion method defaults are not gated on the flow exponent: GAMMA_THETA/GAMMA_PHI changed 1.0 -> 1e4 and ISMIP7_FLUIDITY_PRIOR defaults to 'thermo' for every inversion run from this branch, including n=4 (whose MAP filename carries no n-tag via map_n_tag()) and regularized-Coulomb friction. Regenerating an n=4 MAP from this branch would silently produce a different-method MAP under the same filename the antarctica branch's old-method MAP uses; after the planned merge into icepack/ismip7 upstream this changes the default inversion method for all users. Confirm this is intended or gate the thermo-prior/gamma defaults on n==3.icepack2_tools/thermo_model.py:208- compute_fluidity_prior returns silently when the under-relaxed Picard loop exhausts max_picard without reaching rtol - the 'thermo prior converged.' message only prints on success, so an unconverged A_prior feeds the inversion with no warning. Emit a loud warning (or raise) when the loop ends without dA < rtol.antarctica/scripts/simulation.py:385- The legacy constant-baseline fallback is silent: when a checkpoint lacks fluidity_prior, or the load fails via the broad 'except Exception' at line 238, the forward quietly reconstructs A on the constant A0*a4_factor baseline. The loaded path prints the A_prior range but the fallback branch prints nothing, so an n=3 forward accidentally running against the wrong baseline (e.g. a corrupt/partial checkpoint read) is hard to diagnose from logs. Add a PETSc.Sys.Print in the fallback branch noting the legacy baseline is in use.antarctica/scripts/inversion_icepack2.py:387- Both branches of the friction conditional now compute the identical 'C_w0 = weertman_anchor(H, s, u_obs, m_slide_val, Q)' (lines 387 and 398); only the log messages differ. Hoist the assignment above the if/else and keep the branch-specific prints.icepack2_tools/thermo_model.py:118- Performance note on the Picard loop: the enthalpy bilinear form (a_adv + a_exch) is independent of A_k - fluidity enters only the RHS strain-heating term - and of with_strain, so all ~2*max_picard fd.solve calls assemble and MUMPS-factorize the same matrix (up to ~120 redundant LU factorizations). Trivial at 32 km but material at production resolutions; a cached firedrake LinearSolver with per-call RHS reassembly would cut prior-setup cost. Left as informational since the module is deliberately a near-verbatim port of the mismip thermo model.🔧 Fix: n-aware MAP clip default; loud prior/baseline warnings; dedupe anchor
✅ Re-checked - no issues remain.
antarctica/scripts/inversion_icepack2.py:65- Pre-existing (not introduced by these commits):import colorcetat the top of antarctica/scripts/inversion_icepack2.py does not resolve in the primary ~/venv-firedrake-2026 environment (colorcet 3.1.0 lives only in the older ~/venv-firedrake). The N3_FRAMEWORK.md run recipe would fail at import in the 2026 venv unless colorcet is installed there or provided via PYTHONPATH. It was stubbed for the import-smoke test since it is a colormap-only plotting dependency.validate_n3_prior.pysection 1: compute_fluidity_prior on a synthetic 400x100 km ice stream - Picard convergence (14/18 iters), physical A band, colder-surface-stiffer-ice check, for scalar and spatially-varying T_srfvalidate_n3_prior.pysection 2: Whittle-Matern regularization_form vs analytic sine-field energy (rel err 6e-4) and regularization_gradient_form vs central finite differences (rel err 1e-10)validate_n3_prior.pysection 3: load_mean_annual_surface_temperature against a synthetic ISMIP7-layout tas netcdf (monthly mean, nearest-neighbour sampling, NODATA/ocean fill to 260 K)validate_n3_prior.pysection 4: CheckpointFile round-trip of fluidity_prior + forward rebuild A = A_prior*exp(phi) (1e-14) and missing-field raise for the legacy-MAP fallbackvalidate_n3_prior.pysection 5: import smoke of inversion_icepack2.py and simulation.py, N_FLOW/A4_FACTOR keep-in-sync check, map_n_tag() n=3/_n3 vs n=4/untagged, gamma=1e4 defaultsCode inspection of the 8c5f2e2 first-iteration fail-loud guard and the 63f9dbc n-aware ISMIP7_MAP_CLIP default (not executable without real Antarctic data/MPI)git status --porcelainclean after testing; test fixtures removed✅ **Document** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.