test(vetting): run the CellProfiler oracle — five edge-intensity rows proved, one disproved - #425
Open
darkclad wants to merge 2 commits into
Open
test(vetting): run the CellProfiler oracle — five edge-intensity rows proved, one disproved#425darkclad wants to merge 2 commits into
darkclad wants to merge 2 commits into
Conversation
added 2 commits
August 13, 2026 11:19
…s proved, one disproved The six morphology rows reading oracle=cellprofiler carried no evidence: no version, no config, no generator, so nothing in the tree distinguished a CellProfiler number from a Nyxus one and the vetted verdict rested on the tracker alone (not_covered.md section C). gen_morphology_cellprofiler.py closes that by running the real cellprofiler.modules.MeasureObjectIntensity (4.2.8) on the shape2d fixture. Five reproduce and are now vetted for real, with tool, version, module, recipe and generator recorded: MASS_DISPLACEMENT and the mean, max, min and integrated edge intensities. The agreement is exact rather than approximate because both tools select the same edge pixels -- CellProfiler's edge is find_boundaries(mode="inner"), so an object pixel is an edge pixel unless all four of its N/S/E/W neighbours share its label, which on this fixture is 18 of the 26 ROI pixels summing to 753 against the ROI's 1048. The residual is 5.2e-8 relative and is entirely CellProfiler storing the image as float32. The assertion band tightens from 0.1% to 1e-6 relative to match that measurement. The old band was inherited from the shared snapshot helper and was loose enough to pass the estimator gap described below, which is the kind of tolerance that makes a test unable to fail on the thing it claims. Mutation-checked: a 8.8e-6 perturbation of the mean fails. EDGE_STDDEV_INTENSITY does not reproduce and is demoted to regression. Over the identical 18 pixels Nyxus divides the variance by n-1 (Moments4::std, a helper shared across features) and CellProfiler by n, so the two differ by exactly sqrt(n/(n-1)) = 1.0289915 at n=18: 16.769194 against 16.296728. That is a definitional gap, not a tolerance one. Its row now reads regression with candidate_oracle=cellprofiler and flag=estimator-divergence, and its snapshot moves to test_2d_morphology_regression.h so the _cellprofiler file holds only assertions CellProfiler actually backs. The generator checks the ratio as an identity rather than printing it, so the relationship cannot drift unnoticed in either direction. Which estimator Nyxus means to report is a src/nyx question and is left open here.
… the registry coverage_report.md is generated from oracle_coverage.csv and says so in its own header, but nothing verified that it still matched. It had not: PolusAI#422 demoted ten GLCM matlab rows to regression without regenerating, so the committed report claimed glcm 118/118 vetted while the registry held 108, and the headline figure read 620/758 against an actual 610/758. Regenerating from HEAD's own registry, with no other edits, reproduces the corrected numbers -- the ten-feature drop in this diff is that pre-existing drift being corrected, not an effect of the CellProfiler work, which accounts for exactly one further demotion (610 -> 609, morphology 81 -> 80). --check now compares the rendered text to the file on disk. A registry with no report beside it stays clean, since the self-tests validate ad-hoc registries in tmp_path. The self-test covers both directions: freshly written passes, registry-moves-on fails, and absent report passes. Mutation-checked against the real tree -- editing one figure in the report makes --check exit 1.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test(vetting): run the CellProfiler oracle — five edge-intensity rows proved, one disproved
9 files, all under
tests/. Nosrc/nyxchange.Closes the
test_2d_morphology_cellprofiler.hentry innot_covered.mdsection C, which had beenopen since the file was created: six rows read
status=vetted, oracle=cellprofileron the strengthof a tracker entry, with no version, no config and no generator recorded. Nothing in the tree
distinguished a CellProfiler number from a Nyxus one. Running the oracle settles all six — five of
the claims were true, one was not.
The generator
gen_morphology_cellprofiler.pydrives the realcellprofiler.modules.MeasureObjectIntensity(CellProfiler 4.2.8 / cellprofiler-core 4.2.8.1) on the
shape2dgtest fixture. Offline only —CellProfiler is never a CI dependency. All six features come from that one module, one measurement
each.
The comparison is exact rather than approximate because both tools select the same edge pixels.
CellProfiler's edge is
skimage.segmentation.find_boundaries(labels, mode="inner"): an object pixelis an edge pixel unless all four of its N/S/E/W neighbours share its label. On this fixture that is
18 of the 26 ROI pixels, and they sum to 753 against the ROI's 1048 — the pinned
EDGE_INTEGRATED_INTENSITYto the digit, before any tolerance is applied.Five proved
MASS_DISPLACEMENT,EDGE_MEAN_INTENSITY,EDGE_MAX_INTENSITY,EDGE_MIN_INTENSITY,EDGE_INTEGRATED_INTENSITYreproduce at 5.2e-8 relative at worst, and that residual is not adisagreement: CellProfiler measures on
[0,1]and stores the image asfloat32, so a raw valueround-trips as
raw/255 → float32 → *255. Their rows now carry tool, version, module, config recipe(
morphology.cellprofiler_edge_intensity) and generator path.The assertion band tightens from 0.1% to 1e-6 relative, set from that measurement rather than
inherited from the shared snapshot helper. The old band mattered: it was loose enough to admit the
estimator gap below, which is the kind of tolerance that leaves a test unable to fail on the thing it
claims. Mutation-checked — an 8.8e-6 perturbation of the mean fails, and would have passed at 0.1%.
One disproved
EDGE_STDDEV_INTENSITYis not CellProfiler's number. Over the identical 18 pixels the two toolsuse different estimators — Nyxus divides the variance by
n-1, CellProfiler byn— so they differby exactly
sqrt(n/(n-1))= 1.0289915 at n=18:A definitional gap, not a tolerance one, and no honest band absorbs it. The row is demoted to
regressionwithcandidate_oracle=cellprofiler (MeasureObjectIntensity)andflag=estimator-divergence, and its snapshot moves totest_2d_morphology_regression.hso the_cellprofilerfile holds only assertions CellProfiler actually backs. The generator asserts theratio as an identity, so neither side can drift unnoticed.
Beyond this PR:
Moments4::std()is shared across features, so Nyxus reports the sample estimatorwherever that helper is used, while the tools it is vetted against generally report the population
one. Whether that is intended is a
src/nyxdecision and is deliberately left open here.Found on the way: the coverage report had drifted
coverage_report.mdis generated fromoracle_coverage.csvand says so in its own header, butnothing checked that it still matched — and it did not. #422 demoted ten GLCM
matlabrows toregressionwithout regenerating, so the committed report claimed glcm 118/118 vetted againstthe registry's 108, and a headline 620/758 against an actual 610/758. Regenerating from HEAD's
own registry, with no other edit, reproduces the corrected figures.
So: the ten-feature drop in this diff is that pre-existing drift being corrected, not an effect of
this work — which accounts for exactly one further demotion (610 → 609, morphology 81 → 80).
check_coverage.py --checknow compares the rendered text to the file on disk, with a self-testcovering all three cases (fresh passes, registry-moves-on fails, absent report passes). A registry
with no report beside it stays clean, since the self-tests validate ad-hoc registries.
Left alone, deliberately
oracle_coverage.csvline 566 (3ROBUST_MEAN) is malformed: an unquoted comma insidecandidate_oracle("[P10, P90]") splits the row into 15 fields, shiftingflag,sourceandnotesby one, so that row'sflagcurrently readsP90]). Unrelated to this change and wouldmuddy the diff — worth its own one-line fix.
Verification
HU-loader mechanics tests, absent from a
NOEXTRAS=ONbuild-fsanitize=address,undefined -fno-sanitize-recover=undefined, RelWithDebInfo, gcc)pytest tests/python/: 86 passed / 1 skipped (7 pre-existing Arrow failures of a tiff-only build)check_test_names.py --checkandcheck_coverage.py --checkclean, the latter now including thestaleness rule
tests/— nosrc/nyxchangeCommits
test(vetting): run the CellProfiler oracle -- five edge-intensity rows proved, one disprovedtest(vetting): fail --check when coverage_report.md no longer matches the registry