Fix gain_correct and flat_correct on a non-default device - #993
Conversation
gain_correct and flat_correct called xp.asarray on the gain and on the flat normalization without a device, so the scalar landed on the default device and the multiply/divide failed when the image is elsewhere. Also make a plain-number gain a float, because the array API standard does not promote integer arrays with floating-point arrays. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S36ZzAAVXVm32vuTdtCQME
Astropy's _propagate_multiply_divide uses np.abs and np.sqrt, which convert the operands to numpy (and fail outright for arrays on a non-default device). Give the wrapper mixin its own copy of the formula using the array namespace, and drop the two strict xfail markers that covered this. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S36ZzAAVXVm32vuTdtCQME
The flats in test_flat_correct, test_flat_correct_norm_value and test_flat_correct_deviation were created on the default device while the image is on the testing device. test_flat_correct_data_uncertainty gave a numpy uncertainty; use a StdDevUncertainty in the array namespace instead. test_gain.py multiplied a numpy float64 scalar from Quantity.value into the array and was missing the asserts on its checks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S36ZzAAVXVm32vuTdtCQME
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #993 +/- ##
==========================================
+ Coverage 97.44% 97.47% +0.02%
==========================================
Files 9 9
Lines 1761 1781 +20
==========================================
+ Hits 1716 1736 +20
Misses 45 45
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Drop the branches of _propagate_multiply_divide that cannot be reached through the wrapper (it removes the units from the uncertainties before the arithmetic, and result_data is not used by the formulae), and add tests for the two reachable paths that were untested: an uncertainty on the operand only, and correlated uncertainties. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S36ZzAAVXVm32vuTdtCQME
|
Codecov's 5 missing patch lines (from its line-level report for
Local coverage on numpy now reports no misses in that function. Full matrix on 0772348: numpy 546 passed, dask 539, jax 538 (7 xfailed); strict still 41 failed with the same set as before, 5 XPASS (the stale One thing found while writing the correlation test, not fixed here: correlated — Written by Claude at @mwcraig's direction. |
| from_variance=lambda x: x, | ||
| ): | ||
| """ | ||
| Error propagation for multiplication or division. |
There was a problem hiding this comment.
Needs a proper numpy-style docstring
There was a problem hiding this comment.
Agreed — the current docstring is a prose note with no Parameters/Returns. Proposed replacement, following the numpy-style used by propagate above:
"""
Propagate uncertainty for multiplication or division.
This is astropy's
``_VariancePropagationMixin._propagate_multiply_divide`` with the
NumPy calls replaced by their array-namespace equivalents; see the
astropy version for the derivation of the formulae. Unlike astropy's
version this does not convert the uncertainties between units,
because ``_CCDDataWrapperForArrayAPI._arithmetic_wrapper`` removes
the units from the uncertainties before doing the arithmetic.
Parameters
----------
other_uncert : `~astropy.nddata.NDUncertainty`
The uncertainty of the other operand. Its ``array`` and
``parent_nddata.data`` must be in the same array namespace as
``self.array``.
result_data : array-like
Accepted only for signature compatibility with astropy; the
formulae do not use it.
correlation : float or array-like
Correlation coefficient between the two operands, ``0`` for
uncorrelated.
divide : bool, optional
``True`` for division, ``False`` (default) for multiplication.
to_variance : callable, optional
Converts the stored uncertainty array to a variance. Defaults to
the identity, i.e. the uncertainty is already a variance.
from_variance : callable, optional
Converts a variance back to the stored uncertainty type. Defaults
to the identity.
Returns
-------
array-like
The propagated uncertainty array, in the same array namespace and
on the same device as the inputs, in the representation of
``self`` (as determined by ``from_variance``).
"""Not applied on the branch yet; say the word and I'll push it.
— Written by Claude at @mwcraig's direction.
| - Keep the mask of the result of ``Combiner.average_combine``, | ||
| ``median_combine``, ``sum_combine`` and ``combine`` in the array namespace | ||
| and on the device of the data instead of converting it to NumPy. [#992] | ||
| - Fix ``gain_correct`` and ``flat_correct`` for images on a non-default |
There was a problem hiding this comment.
Agreed — at five lines it's the longest entry in the section. Proposed four-line version that keeps all three fixes (the integer-gain cast is real: array-api-strict refuses int_array * float_array, so gain_correct(ccd, 2) failed there before this PR):
- Fix ``gain_correct`` and ``flat_correct`` for images on a non-default
device: put the gain and flat normalization on the device of the data, cast
an integer gain to float, and keep the uncertainty propagation for
multiplication and division out of NumPy. [#993]Not applied on the branch yet.
— Written by Claude at @mwcraig's direction.
Also shorten the changelog entry for the gain/flat device fix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S36ZzAAVXVm32vuTdtCQME
…espace _ArrayAPIPropagationMixin _propagate_add and _propagate_subtract delegated correlated-uncertainty math to astropy _VariancePropagationMixin _propagate_add_sub, whose correlation term (2 * correlation * np.sqrt(this * other)) is hardcoded to NumPy. That is fine when uncertainty_correlation is 0 (the term is never evaluated), but addition/subtraction with a nonzero correlation on strict fails with TypeError: Expected Array or Python scalar; got numpy.ndarray once the NumPy result is added to an array-API array. #993 already fixed the same class of leak for _propagate_multiply_divide; mirror it here by adding an array-namespace _propagate_add_sub to the mixin and calling it instead of the superclass version. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S36ZzAAVXVm32vuTdtCQME
Part of #971 (the
gain_correct/flat_correctdevice=bucket, plus the multiply/divide uncertainty-propagation bucket that sits directly behind it).On the strict job this takes the failure count from 54 to 41 on top of
main(ba758f0), with no new failures. The two strictxfailmarkers intest_ccddata_wrapper_for_array_api.pythat covered the propagation leak started XPASSing and are removed; the 5 remaining XPASSes are the same staletest_cosmicray.pymarkers as onmain.Three commits:
gain_correct/flat_correctcreate the gain and the flat normalization on the device of the data.xp.asarray(gain_value)/xp.asarray(flat_mean)had nodevice=, so the scalar landed on the default device and the multiply/divide failed with "arrays from two different devices" when the image is elsewhere (core.py:968,:1057). Also, a plain-number gain is now made afloatfirst:gain_correct(ccd, gain=3)produced anint64array and the array API standard does not promote that withfloat64data.The wrapper's uncertainty mixin owns
_propagate_multiply_divide. Astropy's version usesnp.abs/np.sqrt, which convert the operands to numpy — silently on a CPU array (the result then fails atxp.sqrt), and with aRuntimeErroron a non-default device. The override is astropy's formula with the numpy calls replaced by the array namespace's, and_propagate_multiply/_propagate_dividenow call it. This is what makesgain_correcton an image with an uncertainty work on a non-default device.Test bodies. The flats in
test_flat_correct,test_flat_correct_norm_valueandtest_flat_correct_deviationwere built on the default device while the image is on the testing device;test_flat_correct_data_uncertaintyused a numpy uncertainty and.all()(not in the standard);test_gain.pymultiplied a numpyfloat64fromQuantity.valueinto the array and was missing theasserts on its twoxp.all(...)checks.The one gain/flat test still failing on strict,
test_ccd_process_gain_corrected, stops at theccd_processbuiltin-boolbucket (core.py:383), which is separate.🤖 Generated with Claude Code
https://claude.ai/code/session_01S36ZzAAVXVm32vuTdtCQME