Refactor postprocessing into a single batch-handling path; add standalone postprocessor runner - #128
Refactor postprocessing into a single batch-handling path; add standalone postprocessor runner#128almilder wants to merge 11 commits into
Conversation
… the electron suseptibility
…end the code will fit from right to left instead of throwing an error
…tandalone postprocessor runner Moves saving of fit results from postprocessing into the main fitter, unifies the previously-duplicated batch-creation logic between fitter and postprocess, and makes the postprocessor optional so a run can be restarted or replotted without redoing the fit. Adds run_postprocessor.py so plots can be remade or run separately after a fit completes, with regression coverage for refit_bad_fits and the new runner. Also updates a test_manifest.py mock to match the new process_data signature, and fixes an unrelated pre-existing Windows-only test failure (mlflow artifact_location needs a file:// URI, not a bare Windows path).
joglekara
left a comment
There was a problem hiding this comment.
Requesting changes for these correctness issues:
- Angular replay skips the ang_res_unit conversion used during fitting, so it postprocesses a different lineout range.
- refit_bad_fits passes every index to one_d_loop with num_batches=1, so it optimizes lineout 0 instead of the flagged lineout.
- Replay rebuilds checkpoints from the original config, but angular refinement mutates fe.nvx; refined checkpoints therefore cannot be deserialized.
- Remote replay also assumes defaults.yaml/inputs.yaml, excluding app runs that only have config.yaml.
CPU/GPU checks are green, but these paths need regression coverage.
…actor - refit_bad_fits passed the full sample_indices array to one_d_loop with num_batches=1, so it always refit lineout 0 regardless of which lineout was actually flagged as bad. Now passes just the flagged index. - postprocess_runner's replay never applied the ang_res_unit lineout start/end conversion multirun_angular_optax applies during the original fit, so it postprocessed a different (unconverted) lineout range for angular fits. That conversion, and the nvx/window-length growth from multi-pass angular refinement (config["optimizer"]["num_mins"] > 1), are now factored out of loops.py (apply_ang_res_unit, advance_refinement_shape) and replayed before reconstructing the ThomsonParams skeleton, so a refined checkpoint can actually be deserialized and the correct batch is rebuilt. - run_postprocess_local/_load_merged_config and run_postprocess_remote hardcoded defaults.yaml + inputs.yaml, excluding app-originated runs (runner.run_for_app), which only ever log a single config.yaml. Both now check for config.yaml first and fall back to the defaults/inputs pair. Adds regression coverage for all four: tests/test_inverse/test_loops.py for the two extracted helpers, and new cases in test_postprocess_runner.py for the config.yaml artifact layout (both local and remote). Full test suite: 133 passed, 5 skipped (up from 125 -- 8 new tests). Note: separately investigated a report that plain "angular" (non-"range" lineout type) fitting looks non-functional -- build_angular_batch assumes a full CCD frame that only "angular_full" actually has, and there is no one_d_loop-style batched path for it. This predates the work in this PR (traced back to at least the 2024-08 optax-based fitter, unrelated to the postprocessor refactor) and is left unfixed here as a separate, tracked issue.
|
Thanks for the thorough review — all four are fixed in the latest commit ( Angular replay skips the
Refined checkpoints can't be deserialized. Confirmed: Remote replay assumes Added regression coverage for all four in Separately, while chasing the angular-replay fix I found that plain |
…batch_size run_postprocess rebuilt its normalization sample from the first batch_size raw rows instead of the fitted lineout range, and never reset batch_size to 1 for angular fits, so replayed sigmas/losses/plots could silently diverge from (or crash relative to) the original fit. Also avoid downloading config.yaml twice in run_postprocess_remote. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Summary
Test plan