Coverage nexp masks by martinkilbinger · Pull Request #797 · CosmoStat/shapepipe

martinkilbinger · 2026-07-09T14:10:59Z

Summary

Implements creation of coverage healsparse masks encoding the exposures on which PSF information is avialable.

The current code, implemented in utilities does the following steps:

Read CCD list and download corresponding FITS exposure images headers (header_downloader.py)
Extract CCD corner coordinates using WCS from header (field_corners_extractor.py)
Create healsparse mask (coverage_map_builder.py)

This can be integrated into a workflow. We also need to implement obtaining the list of valid CCDs with PSF model.

Reviewer Checklist

Reviewers should tick the following boxes before approving and merging the PR.

…asses

…o coverage

# Conflicts: # docs/source/pipeline_canfar.md # pyproject.toml

- Extract _expnum_from_path, _parse_header_to_wcs, _megacam_field_corners as module-level helpers. process_single_header (static, for multiprocessing) and the instance methods now share one implementation. - Remove dead get_wcs_from_header and get_megacam_field (duplicated logic that had already been inlined into process_single_header). - Fix re.search(r'(\d+)\.txt') missing its subject string. The new helper also replaces the hard-coded p[end-6:end] exposure-number extraction in run(), so exposure numbers of any length work. - Declare verbose in params_default so the contract is local instead of relying on cs_util.args.parse_options to inject it. Addresses review feedback on #699.

- Replace hsp_map + 0 with hsp_map.copy() (healsparse supports .copy since at least 1.x; more explicit intent). - Declare verbose in params_default; currently populated only by cs_util.args.parse_options auto-injection. - Fix typo: "Install te cs_util" -> "Install the cs_util". Addresses review feedback on #699.

- coverage_plotter, header_downloader: declare verbose explicitly in params_default. - coverage_plotter: comment acknowledging that FootprintPlotter._regions is a private cs_util attribute; refactor to public accessor when cs_util exposes one. - header_downloader: expand the vos:cfis/pitcairn comment to flag it as UNIONS/CFIS-specific and document override. - build_and_plot_coverage_maps.sh: explain the BUILD_NSIDE=131072 choice (matches UNIONS bit-mask pixel scale, ~0.1"), and note the CoverageMapBuilder default of 2048 for lighter use. Addresses review feedback on #699.

Resolve pyproject.toml conflict: the coverage branch forked before develop's uv/pyproject modernization, so take develop's file as the base and layer in the coverage additions — healsparse + skyproj runtime deps, the plot optional-deps group, plot added to the dev extra, and the six coverage/get-ccds console entry points. All other files auto-merged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HvP7uKkfDU2PvWC7YmbuMW

The coverage feature adds healsparse (map building) and skyproj (plotting) to pyproject. healsparse was already resolved transitively via cs_util; this pins skyproj 2.5.0 and records both as shapepipe's own direct dependencies so 'uv lock --check' passes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HvP7uKkfDU2PvWC7YmbuMW

cailmdaley

Thanks Martin. The merge conflict's resolved — I merged develop in (the branch had fallen well behind), reconciled pyproject.toml, and regenerated uv.lock, so it's MERGEABLE again. Structure looks clean; the main thing is a scope question, then a few specific items inline.

Per-exposure vs per-CCD coverage. The mask is currently built per exposure: get_exposures() collapses the per-CCD valid-PSF IDs down to unique exposure numbers, _megacam_field_corners() draws a single polygon over the whole ~1° MegaCam mosaic, and each exposure adds a flat +1 across it. So an exposure with even one valid-PSF CCD stamps +1 over its entire field — including the sky where its other CCDs had no valid PSF. nexp then over-reports PSF depth exactly where CCD-level failures cluster (chip gaps, flagged/edge CCDs). The per-CCD validity is already computed upstream, so the fix is to carry the exp-shdu identity through and draw one polygon per surviving CCD instead of collapsing to exposures.

Two questions so we're reviewing against the right target: (1) do you intend to build the mask at CCD level? (2) does the valid-CCD-with-PSF selection land in this PR, or a follow-up? The description flags it as still-to-do, so I want to make sure we're not measuring against the wrong scope.

I think this wants test coverage before it merges — there's none yet, and the pure parsing functions (exposure-number regex, header→WCS split, the corner convention) are exactly where silent failures would hide, and are cheap to cover. Happy to write those with you, and to help wire the per-CCD path if that's the direction.

A few specific items inline.

— Fable, on behalf of Cail

cailmdaley · 2026-07-11T12:27:46Z

+        exp_all = self.get_exp(patches)
+
+        # Turn exposures into exposure-single-HDU names (CCDs)
+        exp_shdu_all = summary.get_all_shdus(exp_all, n_CCD)


The missing-CCD set is computed at line 226 but never subtracted here, so this returns every CCD as valid — unfiltered. (The v1.3 path does filter on non-nan FWHM, so the two versions disagree.) Likely part of the valid-CCD-with-PSF work still to come; flagging since the current path silently returns unfiltered. Fix is exp_shdu_all -= exp_shdu_missing_all before returning.

cailmdaley · 2026-07-11T12:27:46Z

+        for i, r in enumerate(rows):
+            ra = [r["tlra"], r["trra"], r["brra"], r["blra"]]
+            dec = [r["tldec"], r["trdec"], r["brdec"], r["bldec"]]
+            poly = hsp.Polygon(ra=ra, dec=dec, value=1)


No RA-wrap or pole guard. Corners arrive straight from astropy as degrees in [0, 360), so an exposure straddling RA=0 yields non-monotonic corners (e.g. [359.8, 0.2, ...]) and healsparse fills the ~360° complement, corrupting nexp across that RA stripe; near Dec ≈ +90 (the driver runs dechi=90) the four-corner ordering also degenerates. Worth unwrapping RA to a common branch per polygon and guarding the pole.

cailmdaley · 2026-07-11T12:27:46Z

+
+        # Download from VOSpace
+        try:
+            client.copy(source, dest, head=True)


client.copy writes straight to dest, so an interrupted transfer leaves a partial .txt that the exists(dest) check at line 194 treats as complete on the next run — a resumed job then builds on a truncated header. Download to a temp path and os.rename on success (atomic).

cailmdaley · 2026-07-11T12:27:46Z

@@ -0,0 +1,72 @@
+#!/bin/bash


No set -euo pipefail — a failed build/plot for one version doesn't stop the loop, and the final success message prints unconditionally.

cailmdaley · 2026-07-11T12:27:46Z

 Then, build the healsparse coverage mask file as
 ```bash
-build_coverage_map
+build_coverage_map -i exp_ra_dec_v1.6 -o coverage_v1.6.x.hsp -c 128 -n 131072 -v


-i exp_ra_dec_v1.6, but the extract_field_corners step above is run without -o, so it writes the default exp_ra_dec.txt — this filename is never produced, and the example throws FileNotFoundError if followed verbatim. Align the two.

The five coverage/get-ccds runners fell back to `args = sys.argv` (the full argv, including argv[0]) instead of `sys.argv[1:]`, so under the entry-point smoke test's `python -c "...; run_*()" -h` the parser read the program slot as the integer `-c` option and exited non-zero (`invalid integer value: '-h'`). Slice argv[1:] to match cs_util.args.parse_options' own convention. Also drop the `coverage_pipeline` console script, whose target `coverage_run:run_pipeline` was removed in an earlier commit (dangling entry point -> AttributeError on invocation), and declare `hpgeom` explicitly (imported directly by coverage_map_builder, previously only transitive via healsparse). Together these fix all six entry-point test failures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HvP7uKkfDU2PvWC7YmbuMW

Cover the pure and lightly-fixtured logic: exposure-number parsing, the CCD-list -> unique-exposure reduction, the MegaCam field-corner convention, the multi-HDU header -> WCS split, power-of-two nside validation, and a regression guard for the -h/argv handling of the console runners. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HvP7uKkfDU2PvWC7YmbuMW

cailmdaley · 2026-07-11T13:00:55Z

Update — dug into the failing checks and took care of them so this is green for you.

Merging develop in ran its full suite against the branch, surfacing that all six coverage console scripts failed the entry-point smoke test (tests/unit/test_entrypoints.py::test_console_entrypoint_help_runs). Two causes, both fixed:

The five runners parsed the full sys.argv (including argv[0]) rather than sys.argv[1:], so -h collided with the integer -c option (error: option -c: invalid integer value: '-h'). Fixed to slice argv[1:], matching cs_util.args.parse_options' own convention.
coverage_pipeline pointed at run_pipeline, which had been removed — dropped the dangling entry point (easy to re-add as a real orchestrator later if you'd like one).

Also declared hpgeom explicitly (imported directly by coverage_map_builder, previously only resolving transitively via healsparse).

And I added tests/module/test_coverage.py — unit + property tests for the pure logic: exposure-number parsing, the CCD-list → unique-exposure reduction, the MegaCam field-corner convention, the multi-HDU header → WCS split, power-of-two nside validation, and a regression guard for the -h handling. Happy to extend these as the feature grows.

CI is green now (357 passed). With that, the one substantive thing still open is the design question at the top: whether the mask should be built per-CCD — so PSF validity is honoured per-CCD footprint — rather than per-exposure. That one's yours; glad to help wire it once you've settled the intent (and whether the valid-CCD-with-PSF selection lands here or in a follow-up).

— Fable, on behalf of Cail

martinkilbinger · 2026-07-14T05:43:58Z

Update on workflow, the 0th step is:

Get CCDs with valid PSF: get_ccds_with_psf using CcdPsfHandler (needs updating for v2.0)

And field_corner_extractor.py is indeed not doing the CCD-wise masking, but only looking at the entire MegaCam field, which is wrong. This needs fixing.

cailmdaley · 2026-07-14T11:07:16Z

want me to take care of these issues @martinkilbinger ?

martinkilbinger · 2026-07-15T09:13:55Z

want me to take care of these issues @martinkilbinger ?

yes please :-)

There was an earlier question, not sure whether this has been answered, but I confirm that here we want single-CCD-based masks, to account for available/missing PSF. Exposure-count-based masks, or coverage masks, are done elsewhere.

The v>=1.4 path built the full set of exposure single-HDU IDs but never removed the CCDs whose PSF model is missing, so every candidate CCD was reported as valid. Subtract the missing set so the result is the CCDs that actually have a valid PSF, and report the three counts.

Replace the whole-MegaCam-field corner extraction with per-CCD extraction: for each HDU's WCS, build the 4 corners from the CCD pixel bounds (read from pixel_shape, i.e. NAXIS1/NAXIS2) using pixel edges so the quadrilateral covers the full CCD area. Emit one row per CCD keyed on the '<expnum>-<ccd_idx>' ID. Add an optional --ccd_list/-l option: when given, only CCDs in that file (the output of get_ccds_with_psf) are written, so the coverage mask is built from single CCDs with a valid PSF. Resume still keys the done-set on exposure number, now parsed from the CCD-ID first column. Drop the dead _megacam_field_corners whole-field path; keep _parse_header_to_wcs and the multiprocessing structure.

…ards Parse the new per-CCD row format (string CCD ID in column 0, then 8 float corner columns). Add an RA-wrap guard that unwraps polygons straddling the RA=0 seam onto a common branch, and a pole guard that skips polygons reaching |dec| >= 89 deg with a warning (CCD footprints never reach there). Since the CCDs of one exposure do not overlap, accumulating value 1 per CCD polygon still counts exposures-with-valid-PSF per pixel.

Copy each header to a '.part' file in the same directory and os.rename on success. An interrupted transfer can no longer leave a partial file that resume treats as complete; the temp file is removed on failure.

Fix the console-script name typo (get_ccd_with_psfs -> get_ccds_with_psf) so it matches the docs and script. Harden the build/plot script with 'set -euo pipefail' and document the per-CCD chain (step 0 get_ccds_with_psf feeding extract_field_corners --ccd_list). Update the canfar docs: restore the coverage helpers as shipped, align filenames (ccds_with_psf_<version>.txt, exp_ra_dec_<version>.txt), and describe the per-CCD workflow including step 0.

…ownload Rewrite the coverage tests for the per-CCD design: handler missing-CCD subtraction; per-CCD corner extraction and pixel-shape error; --ccd_list filtering end to end; builder parsing of the new row format; RA-wrap unwrap (a CCD straddling RA=0 fills a small patch, not the complement); pole-guard skip; and atomic-download rename behaviour with a mocked copy. Drop the tests that pinned the removed whole-field convention.

Two correctness fixes in the per-CCD extractor: - Image dimensions: header_downloader fetches fpack tile-compressed headers where NAXIS1/NAXIS2 describe the compressed binary table (byte width, row count), so WCS.pixel_shape reports e.g. (8, 4612) instead of the true (2080, 4612). Read ZNAXIS1/ZNAXIS2 when ZIMAGE is set, falling back to NAXIS1/NAXIS2. _parse_header_to_wcs now returns (wcs, shape) pairs since the WCS drops the Z-keywords; _ccd_corners takes the shape explicitly. - Resume: key the done-set on individual CCD IDs, not exposure numbers. A write interrupted mid-exposure previously left a partial exposure that a -r rerun skipped forever (silent undercount); an expanded --ccd_list rerun skipped exposures whose new CCDs were never written. get_done_ccds reads the present CCD IDs and run() drops only those, so partial exposures are completed and new CCDs added, never duplicating a row.

- header_downloader.get_exposures: wrap the text-format loadtxt in np.atleast_1d so a single-line CCD list (0-d array) does not crash on iteration. - coverage_run: update the extract/build runner docstrings from whole-field 'field corner coordinates' to per-CCD language. - pipeline_canfar docs: add a runnable plot_coverage_map example (the SGC region command) covering the flags build_and_plot_coverage_maps.sh uses.

Extend the coverage tests for the confirmed findings: - fpack-compressed header fixture; assert ZNAXIS dims are recovered and corners span the true CCD width, not the 8-byte compressed NAXIS1. - resume run() path: a new exposure is appended without duplicating existing rows, and a partial exposure (interrupted mid-write) is completed rather than skipped. - the nexp contract on pixel values: two overlapping CCDs from different exposures give value 2 in the overlap and 1 outside. - single-row builder file (atleast_1d/2d guards); single-line get_exposures; RA-wrap strengthened to a positional-parity check (seam CCD vs the same CCD shifted +10 deg agree in pixel count); get_ccds_with_psf subtraction pinned against the real get_all_shdus.

cailmdaley · 2026-07-16T00:02:46Z

Done — the branch now builds single-CCD-based masks. The chain:

get_ccds_with_psf → valid-PSF CCD IDs. The v ≥ 1.4 path now actually subtracts the missing-CCD set (it previously returned every CCD as valid), and -V v2.0 is accepted (see below).

On "needs updating for v2.0": following your #812 note that the patch concept is removed in v2.0, -V v2.0 is now patch-less — the handler reads summary/missing_job_32_all.txt and exp_numbers.txt relative to the run root instead of iterating P1..P9 (a new -p option restores an explicit P1..P<n> layout if ever needed). The job-32 numbering and summary filenames are unchanged on develop, so those are kept as-is — flag if the v2 runs will change the summary conventions themselves.
extract_field_corners --ccd_list <step-1 output> → one row of corners per valid CCD, from each HDU's own WCS (image dims read from ZNAXIS — the .fz headers' NAXIS are the compressed-table dims). This is where the PSF-validity filter lives; without -l all CCDs are extracted.
build_coverage_maps → one polygon per CCD, with RA-wrap and pole guards. CCDs of one exposure don't overlap, so pixel values still count exposures with valid PSF.

Robustness along the way: extractor resume is keyed on individual CCD IDs (interrupted runs and expanded CCD lists complete cleanly), and header downloads are atomic, so partial files can't poison a resumed run.

57 tests cover the chain — including the nexp value contract (overlapping CCDs from two exposures → 2 in the overlap, 1 outside) — and pipeline_canfar.md documents the full flow with aligned filenames. CI green. Over to you for review.

— Claude (Fable) on behalf of Cail.

…v1.6) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MqH9wjbJicVmaQdhZ4kBYr

… map The v2 footprint is still growing (>5000 deg2 per the 2026-07 Paris meeting, new Dec<30 region); the newest patch definitions (auxdir/CFIS/tiles_202510) include P10. Default v2.0 to 10 patches and add an explicit -p/--n_patch override so the handler tracks evolving patch layouts without code changes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MqH9wjbJicVmaQdhZ4kBYr

Martin (PR #812 review): the patch concept is removed in v2.0. For -V v2.0 the handler reads summary/missing_job_32_all.txt and exp_numbers.txt relative to the run root instead of iterating P1..P9; -p restores an explicit P1..P<n> layout when needed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MqH9wjbJicVmaQdhZ4kBYr

…-less) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MqH9wjbJicVmaQdhZ4kBYr

cailmdaley · 2026-07-16T01:16:51Z

@martinkilbinger please take a look, should be good now!

cailmdaley · 2026-07-16T08:53:34Z

i think this closes #836?

martinkilbinger and others added 25 commits January 9, 2026 14:18

removed tmp files

668e05d

Merge branch 'p9' of github.com:martinkilbinger/shapepipe-1 into p9

de045f5

Moved get_ccds script to pipeline runner and library script

38c5819

class to download eposure headers (for coverage)

d6144ad

coverage mask: scripts and plotter; using cs_util

bc43799

coverage code: added Mike Hudson as co-author

487166c

documentation (in pipeline_canfar.md) on coverage mask scripts and cl…

d0d5488

…asses

Merge remote-tracking branch 'origin/coverage' into coverage

bc8b006

CCD PSF Handler: Added version for v1.3

f4d00d0

fixed coverage plots

271ff40

header downloads (coverage): account for existing downloads

7f99f5a

Merge branch 'coverage' of github.com:martinkilbinger/shapepipe-1 int…

77fe037

…o coverage

Merge remote-tracking branch 'upstream/develop' into coverage

be10deb

Merge branch 'develop' into coverage

728f413

canfar-changes

e10111b

Merge remote-tracking branch 'origin/coverage' into coverage

ac94c16

Merge remote-tracking branch 'origin/develop' into coverage

6d10285

# Conflicts: # docs/source/pipeline_canfar.md # pyproject.toml

PR: remove non-functioning coverage pipeline function

fd61120

PR: coverage_map_builder; moved import into try block

fc37cbf

PR: field_corners_extractor.py; fixed exp number extraction

afcbae6

PR: added comment

96cc4f5

PR: added comment

5215788

martinkilbinger requested a review from cailmdaley July 9, 2026 14:10

martinkilbinger self-assigned this Jul 9, 2026

martinkilbinger added the enhancement New feature or request label Jul 9, 2026

cailmdaley and others added 2 commits July 11, 2026 14:05

cailmdaley reviewed Jul 11, 2026

View reviewed changes

cailmdaley and others added 2 commits July 11, 2026 14:46

This was referenced Jul 12, 2026

Update contributors list #809

Merged

Contributors #704

Closed

cailmdaley mentioned this pull request Jul 15, 2026

Build ShapePipe per-CCD failure mask #836

Open

cailmdaley added 10 commits July 16, 2026 01:30

Merge remote-tracking branch 'origin/develop' into coverage

d2bdf28

fix(coverage): download headers atomically via temp-file rename

96cd62b

Copy each header to a '.part' file in the same directory and os.rename on success. An interrupted transfer can no longer leave a partial file that resume treats as complete; the temp file is removed on failure.

cailmdaley and others added 4 commits July 16, 2026 02:21

feat(coverage): accept catalogue version v2.0 (9-patch footprint, as …

15401d9

…v1.6) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MqH9wjbJicVmaQdhZ4kBYr

docs: state the patch convention per catalogue version (v2.0 is patch…

46214c7

…-less) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MqH9wjbJicVmaQdhZ4kBYr

cailmdaley mentioned this pull request Jul 16, 2026

Exposure-level star/defect masking strategy #839

Open

Uh oh!

Conversation

martinkilbinger commented Jul 9, 2026 • edited by cailmdaley Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Reviewer Checklist

Uh oh!

cailmdaley left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cailmdaley Jul 11, 2026

Choose a reason for hiding this comment

Uh oh!

cailmdaley Jul 11, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cailmdaley Jul 11, 2026

Choose a reason for hiding this comment

Uh oh!

cailmdaley Jul 11, 2026

Choose a reason for hiding this comment

Uh oh!

cailmdaley Jul 11, 2026

Choose a reason for hiding this comment

Uh oh!

cailmdaley commented Jul 11, 2026

Uh oh!

martinkilbinger commented Jul 14, 2026

Uh oh!

cailmdaley commented Jul 14, 2026

Uh oh!

martinkilbinger commented Jul 15, 2026

Uh oh!

cailmdaley commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cailmdaley commented Jul 16, 2026

Uh oh!

cailmdaley commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

martinkilbinger commented Jul 9, 2026 •

edited by cailmdaley

Loading

cailmdaley commented Jul 16, 2026 •

edited

Loading