Snapshot the fit parameters, so an interrupted fit is not a lost fit

[davidwalter2]

Independent of #153 and #154 — branched from main, one new module plus small
hooks.

Reviewing fitter.py: the minimizer's while True: loop is indented by
four spaces to sit inside a with snapshot_on_signal(...) block, so git pairs
the old and new copies line by line and renders the whole loop as rewritten —
existing breaks and comments included. Nothing in it changed:
git diff -w upstream/main..HEAD -- rabbit/fitter.py is the real change, 33
lines added and 1 removed, and the loop still has the same three break
statements and the same comments, byte for byte.

The problem

Everything a fit has learned lives in one in-memory vector until the output
file is written at the very end. A fit that is killed, hits a wall clock limit,
or dies on the way to its output leaves nothing at all behind, however close
to the minimum it had got. That is cheap for a five-minute fit and expensive for
one that has been running for days.

What this adds

rabbit/snapshot.py writes that vector to disk. A snapshot is deliberately the
smallest thing load_fitresult accepts — the parameter values and their names —
so it resumes through the path that already exists, with no new format and no
change to how results are read:

rabbit_fit.py input.hdf5 -o out/ --externalPostfit snapshot.hdf5           # carry on
rabbit_fit.py input.hdf5 -o out/ --externalPostfit snapshot.hdf5 --noFit   # postfit only

No covariance is stored: it does not exist until the Hessian is computed, and
load_fitresult already treats it as optional. Bin-by-bin parameters are
re-profiled on load. A snapshot is a few hundred kB.

When one is written

trigger	covers
SIGINT / SIGTERM	wall clock limits, and kill on a fit someone decided to stop
the minimizer raises	a failure that would otherwise discard the whole fit
convergence	the stretch between the minimum and the output being closed — on a large model the Hessian, the impacts and the postfit histograms
every N hours	--snapshotInterval, off by default

The first three do not depend on --snapshotInterval. Nothing is written unless
--snapshotFile or --snapshotInterval is given; with an interval and no
filename it defaults to <outdir>/<outname>_snapshot.hdf5.

Three details that are load-bearing

The values written are physical. Under preconditioning the minimizer's
iterate is in internal coordinates, and a snapshot of those would load without
complaint and be silently wrong. Conversion happens when the point is recorded
rather than when it is written, which also closes a window that would otherwise
be minutes wide: during a preconditioner rebuild the stored transform no longer
matches the stored iterate.

The write is atomic. Snapshots exist for processes dying at moments they did
not choose, and that includes mid-write. Writing in place would leave a truncated
file where a good one used to be, making the safety net the thing that destroys
the result. Each snapshot goes to a temporary file in the same directory
(os.replace is only atomic within a filesystem) and is moved into place.

The signal handler writes immediately rather than asking the minimizer to
stop at the next iteration. On a large model an iteration can run for hours, and
a scheduler that sends SIGTERM follows it with SIGKILL long before then — a
cooperative stop would arrive too late to be the safety net this is for. The
handler then restores the previous handler and re-raises, so the process still
dies as the sender intended and with the right exit status; the fit's own
control flow is unchanged.

Notes

rabbit.snapshot imports neither the fitter nor TensorFlow. That is what makes
it usable from a signal handler and testable in a subprocess that starts in
seconds instead of minutes (importing TensorFlow currently costs over two
minutes on a loaded machine).

The diff to fitter.py looks larger than it is: ignoring whitespace it is 33
lines added and 1 removed, the rest being the minimizer loop reindenting into a
with block.

This does not checkpoint the minimizer's internal state — trust radius, Lanczos
basis, preconditioner — so a resumed fit restarts its trust region at the
snapshot point. Near a minimum that is cheap; mid-descent it costs some
re-convergence.

Testing

tests/test_snapshot.py covers the round trip through the real
load_fitresult (so it fails if the layout ever drifts from what
--externalPostfit accepts), that the preconditioner is undone, that a rebuilt
transform does not retroactively remap an older iterate, that a failed write
leaves the previous snapshot intact and no temporary file behind, the interval
logic, and — in a subprocess — that SIGTERM leaves a correct snapshot on disk
and the process still dies by SIGTERM. 74 tests pass on this branch.

🤖 Generated with Claude Code

https://claude.ai/code/session_018rYVNwxQvuUYz5z9wt6ddb