Skip to content

test(vetting): run the CellProfiler oracle — five edge-intensity rows proved, one disproved - #425

Open
darkclad wants to merge 2 commits into
PolusAI:mainfrom
darkclad:main-cellprofiler-provenance
Open

test(vetting): run the CellProfiler oracle — five edge-intensity rows proved, one disproved#425
darkclad wants to merge 2 commits into
PolusAI:mainfrom
darkclad:main-cellprofiler-provenance

Conversation

@darkclad

Copy link
Copy Markdown

test(vetting): run the CellProfiler oracle — five edge-intensity rows proved, one disproved

9 files, all under tests/. No src/nyx change.

Closes the test_2d_morphology_cellprofiler.h entry in not_covered.md section C, which had been
open since the file was created: six rows read status=vetted, oracle=cellprofiler on the strength
of a tracker entry, with no version, no config and no generator recorded. Nothing in the tree
distinguished a CellProfiler number from a Nyxus one. Running the oracle settles all six — five of
the claims were true, one was not.

The generator

gen_morphology_cellprofiler.py drives the real cellprofiler.modules.MeasureObjectIntensity
(CellProfiler 4.2.8 / cellprofiler-core 4.2.8.1) on the shape2d gtest fixture. Offline only —
CellProfiler is never a CI dependency. All six features come from that one module, one measurement
each.

The comparison is exact rather than approximate because both tools select the same edge pixels.
CellProfiler's edge is skimage.segmentation.find_boundaries(labels, mode="inner"): an object pixel
is an edge pixel unless all four of its N/S/E/W neighbours share its label. On this fixture that is
18 of the 26 ROI pixels, and they sum to 753 against the ROI's 1048 — the pinned
EDGE_INTEGRATED_INTENSITY to the digit, before any tolerance is applied.

Five proved

MASS_DISPLACEMENT, EDGE_MEAN_INTENSITY, EDGE_MAX_INTENSITY, EDGE_MIN_INTENSITY,
EDGE_INTEGRATED_INTENSITY reproduce at 5.2e-8 relative at worst, and that residual is not a
disagreement: CellProfiler measures on [0,1] and stores the image as float32, so a raw value
round-trips as raw/255 → float32 → *255. Their rows now carry tool, version, module, config recipe
(morphology.cellprofiler_edge_intensity) and generator path.

The assertion band tightens from 0.1% to 1e-6 relative, set from that measurement rather than
inherited from the shared snapshot helper. The old band mattered: it was loose enough to admit the
estimator gap below, which is the kind of tolerance that leaves a test unable to fail on the thing it
claims. Mutation-checked — an 8.8e-6 perturbation of the mean fails, and would have passed at 0.1%.

One disproved

EDGE_STDDEV_INTENSITY is not CellProfiler's number. Over the identical 18 pixels the two tools
use different estimators — Nyxus divides the variance by n-1, CellProfiler by n — so they differ
by exactly sqrt(n/(n-1)) = 1.0289915 at n=18:

nyxus        16.769194445558  == sample std     (/n-1)
cellprofiler 16.296727687893  == population std (/n)

A definitional gap, not a tolerance one, and no honest band absorbs it. The row is demoted to
regression with candidate_oracle=cellprofiler (MeasureObjectIntensity) and
flag=estimator-divergence, and its snapshot moves to test_2d_morphology_regression.h so the
_cellprofiler file holds only assertions CellProfiler actually backs. The generator asserts the
ratio as an identity, so neither side can drift unnoticed.

Beyond this PR: Moments4::std() is shared across features, so Nyxus reports the sample estimator
wherever that helper is used, while the tools it is vetted against generally report the population
one. Whether that is intended is a src/nyx decision and is deliberately left open here.

Found on the way: the coverage report had drifted

coverage_report.md is generated from oracle_coverage.csv and says so in its own header, but
nothing checked that it still matched — and it did not. #422 demoted ten GLCM matlab rows to
regression without regenerating, so the committed report claimed glcm 118/118 vetted against
the registry's 108, and a headline 620/758 against an actual 610/758. Regenerating from HEAD's
own registry, with no other edit, reproduces the corrected figures.

So: the ten-feature drop in this diff is that pre-existing drift being corrected, not an effect of
this work — which accounts for exactly one further demotion (610 → 609, morphology 81 → 80).

check_coverage.py --check now compares the rendered text to the file on disk, with a self-test
covering all three cases (fresh passes, registry-moves-on fails, absent report passes). A registry
with no report beside it stays clean, since the self-tests validate ad-hoc registries.

Left alone, deliberately

oracle_coverage.csv line 566 (3ROBUST_MEAN) is malformed: an unquoted comma inside
candidate_oracle ("[P10, P90]") splits the row into 15 fields, shifting flag, source and
notes by one, so that row's flag currently reads P90]). Unrelated to this change and would
muddy the diff — worth its own one-line fix.

Verification

  • gtest: 732/732 (Windows), 728/728 (Linux ASan) — the four-test delta is the DICOM
    HU-loader mechanics tests, absent from a NOEXTRAS=ON build
  • ASan + UBSan: exit 0, zero reports (-fsanitize=address,undefined -fno-sanitize-recover=undefined, RelWithDebInfo, gcc)
  • pytest tests/python/: 86 passed / 1 skipped (7 pre-existing Arrow failures of a tiff-only build)
  • check_test_names.py --check and check_coverage.py --check clean, the latter now including the
    staleness rule
  • generator exits 0: five goldens reproduced, the divergence identity holds
  • Diff entirely under tests/ — no src/nyx change

Commits

  1. test(vetting): run the CellProfiler oracle -- five edge-intensity rows proved, one disproved
  2. test(vetting): fail --check when coverage_report.md no longer matches the registry

Demian Vladi added 2 commits August 13, 2026 11:19
…s proved, one disproved

The six morphology rows reading oracle=cellprofiler carried no evidence: no version, no
config, no generator, so nothing in the tree distinguished a CellProfiler number from a
Nyxus one and the vetted verdict rested on the tracker alone (not_covered.md section C).
gen_morphology_cellprofiler.py closes that by running the real
cellprofiler.modules.MeasureObjectIntensity (4.2.8) on the shape2d fixture.

Five reproduce and are now vetted for real, with tool, version, module, recipe and
generator recorded: MASS_DISPLACEMENT and the mean, max, min and integrated edge
intensities. The agreement is exact rather than approximate because both tools select the
same edge pixels -- CellProfiler's edge is find_boundaries(mode="inner"), so an object
pixel is an edge pixel unless all four of its N/S/E/W neighbours share its label, which on
this fixture is 18 of the 26 ROI pixels summing to 753 against the ROI's 1048. The residual
is 5.2e-8 relative and is entirely CellProfiler storing the image as float32.

The assertion band tightens from 0.1% to 1e-6 relative to match that measurement. The old
band was inherited from the shared snapshot helper and was loose enough to pass the
estimator gap described below, which is the kind of tolerance that makes a test unable to
fail on the thing it claims. Mutation-checked: a 8.8e-6 perturbation of the mean fails.

EDGE_STDDEV_INTENSITY does not reproduce and is demoted to regression. Over the identical
18 pixels Nyxus divides the variance by n-1 (Moments4::std, a helper shared across
features) and CellProfiler by n, so the two differ by exactly sqrt(n/(n-1)) = 1.0289915 at
n=18: 16.769194 against 16.296728. That is a definitional gap, not a tolerance one. Its
row now reads regression with candidate_oracle=cellprofiler and flag=estimator-divergence,
and its snapshot moves to test_2d_morphology_regression.h so the _cellprofiler file holds
only assertions CellProfiler actually backs. The generator checks the ratio as an identity
rather than printing it, so the relationship cannot drift unnoticed in either direction.

Which estimator Nyxus means to report is a src/nyx question and is left open here.
… the registry

coverage_report.md is generated from oracle_coverage.csv and says so in its own header, but
nothing verified that it still matched. It had not: PolusAI#422 demoted ten GLCM matlab rows to
regression without regenerating, so the committed report claimed glcm 118/118 vetted while
the registry held 108, and the headline figure read 620/758 against an actual 610/758.
Regenerating from HEAD's own registry, with no other edits, reproduces the corrected
numbers -- the ten-feature drop in this diff is that pre-existing drift being corrected,
not an effect of the CellProfiler work, which accounts for exactly one further demotion
(610 -> 609, morphology 81 -> 80).

--check now compares the rendered text to the file on disk. A registry with no report
beside it stays clean, since the self-tests validate ad-hoc registries in tmp_path.

The self-test covers both directions: freshly written passes, registry-moves-on fails, and
absent report passes. Mutation-checked against the real tree -- editing one figure in the
report makes --check exit 1.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant