Skip to content

test(ngldm): assert the 2D NGLDM features against mirp, and fix a 50% tolerance - #432

Open
darkclad wants to merge 1 commit into
PolusAI:mainfrom
darkclad:main-vet-2d-ngldm
Open

test(ngldm): assert the 2D NGLDM features against mirp, and fix a 50% tolerance#432
darkclad wants to merge 1 commit into
PolusAI:mainfrom
darkclad:main-vet-2d-ngldm

Conversation

@darkclad

Copy link
Copy Markdown

The registry has claimed status=vetted, oracle=mirp for all 19 2D NGLDM rows since the tracker was imported, with target_test=test_2d_ngldm_mirp.h - a file that did not exist. This runs that oracle for the first time and creates the file.

mirp reproduces Nyxus to 2.9e-16 on all 17 IBSI NGLDM features, so the new test_2d_ngldm_mirp.h pins them at the SPEC 7 exact tier (rel=1e-9). The claim was right; it just had no test.

The tolerance defect

assert_ngldm_feature_against_golden_values() ended in agrees_gt(aveTotal, reference, 2.), and agrees_gt computes tolerance = ground_truth / frac_tolerance - so a factor of 2 is a +/-50% band. It applied to all 17 IBSI assertions and both regression assertions, i.e. none of them could have detected a doubled or halved feature value. Measured agreement is 0.45% worst case against the IBSI consensus and 2.9e-16 against mirp, so the band was ~110x looser than the data requires.

The helper now takes the tolerance as a parameter and each caller states its own:

  • test_2d_ngldm_ibsi.h at rel=1e-2. Its goldens are the published IBSI consensus values, quoted to three significant figures; the worst residual is NGLDM_GLNU at 0.45% (10.2 published against 10.2464 computed) and every other feature is under 0.2%.
  • test_2d_ngldm_mirp.h at rel=1e-9, seven orders of magnitude above the measured noise.
  • test_2d_ngldm_regression.h at rel=1e-9. Those pin Nyxus' own output to 17 digits, so a drift guard should catch any change at all. The two oracle files are complementary rather than redundant: IBSI fixes the definition, mirp fixes the digits. Tightening broke nothing - the suite goes 773 -> 790 tests, all passing.

NGLDM_GLM and NGLDM_DCM

Demoted to status=regression. mirp exposes no grey-level-mean or dependence-count-mean column because neither is an IBSI NGLDM feature, and test_2d_ngldm_ibsi.h already marked both "--not in IBSI--". No oracle can reproduce them, so their rows stop claiming oracle=mirp.

Registry corrections applied to all 19 rows

  • current_test named test_3d_ngldm_regression.h - a 3D file - for 2D features.
  • config_recipe held the generic "Not mode-specific..." blurb; it now names the new recipe ngldm.ibsi_phantom_2d, which records the four-slice averaging the assertions actually perform and the mirp settings that match it (by_slice, no discretisation, ngldm_distance=1, ngldm_difference_level=0).
  • source=tracker on every row, i.e. the verdict rested on an offline harness run nobody could repeat. All 19 now read audit.
  • target_test cleared, since the file it named now exists and is met.

gen_ngldm_mirp.py re-verifies every golden pinned in the header it feeds and exits non-zero on a mismatch or on a pin it cannot produce (17/17, rel=0). It reads the phantom out of tests/test_data.h through oracles/ibsi_phantom.py rather than carrying a second copy of the voxels.

Test-file hygiene, per the revet.txt checklist:

  • test_2d_ngldm_ibsi.h uses std::string, std::vector and std::unordered_map but included only <unordered_map>; test_2d_ngldm_regression.h uses std::string and included none of them, relying on the ibsi header to drag it in. Both now include what they use.
  • test_2d_ngldm_regression.h's include comment named assert_ngldm_feature_ibsi, which it does not call - it uses the shared assert_ngldm_feature_against_golden_values.
  • The IBSI table carried two commented-out entries pinning -1 for NGLDM_GLM and NGLDM_DCM. A commented-out golden is a trap rather than documentation - uncommenting it pins -1 - so they are replaced by a note at the table head saying why those two features are absent and where they live.
  • The 17 features are asserted by both oracle files, which is deliberate rather than redundant; the generated audit CSV now says so per row, so a later cleanup does not delete one as a duplicate.

TOOLS.md gains the two mirp gotchas this family hit: its per-family numeric settings must be floats (ngldm_distance=1 raises TypeError during settings construction, before any image is read), and its NGLDM column names do not track the Nyxus abbreviations (ngl_ldlge = NGLDM_LDLGLE, ngl_dc_perc = NGLDM_DCP), so they must be mapped by meaning rather than string similarity.

Verified: runAllTests 790/790 on Windows and under ASan+UBSan on Linux; pytest tests/python/ full directory; check_coverage.py --check, check_test_names.py --check and audit/scan_ngldm_coverage.py --check all clean.

… tolerance

The registry has claimed status=vetted, oracle=mirp for all 19 2D NGLDM rows since the tracker was
imported, with target_test=test_2d_ngldm_mirp.h - a file that did not exist. This runs that oracle
for the first time and creates the file.

mirp reproduces Nyxus to 2.9e-16 on all 17 IBSI NGLDM features, so the new test_2d_ngldm_mirp.h
pins them at the SPEC 7 exact tier (rel=1e-9). The claim was right; it just had no test.

The tolerance defect
--------------------
assert_ngldm_feature_against_golden_values() ended in agrees_gt(aveTotal, reference, 2.), and
agrees_gt computes tolerance = ground_truth / frac_tolerance - so a factor of 2 is a +/-50% band. It
applied to all 17 IBSI assertions and both regression assertions, i.e. none of them could have
detected a doubled or halved feature value. Measured agreement is 0.45% worst case against the IBSI
consensus and 2.9e-16 against mirp, so the band was ~110x looser than the data requires.

The helper now takes the tolerance as a parameter and each caller states its own:
- test_2d_ngldm_ibsi.h at rel=1e-2. Its goldens are the published IBSI consensus values, quoted to
  three significant figures; the worst residual is NGLDM_GLNU at 0.45% (10.2 published against
  10.2464 computed) and every other feature is under 0.2%.
- test_2d_ngldm_mirp.h at rel=1e-9, seven orders of magnitude above the measured noise.
- test_2d_ngldm_regression.h at rel=1e-9. Those pin Nyxus' own output to 17 digits, so a drift guard
  should catch any change at all.
The two oracle files are complementary rather than redundant: IBSI fixes the definition, mirp fixes
the digits. Tightening broke nothing - the suite goes 773 -> 790 tests, all passing.

NGLDM_GLM and NGLDM_DCM
-----------------------
Demoted to status=regression. mirp exposes no grey-level-mean or dependence-count-mean column
because neither is an IBSI NGLDM feature, and test_2d_ngldm_ibsi.h already marked both
"--not in IBSI--". No oracle can reproduce them, so their rows stop claiming oracle=mirp.

Registry corrections applied to all 19 rows
-------------------------------------------
- current_test named test_3d_ngldm_regression.h - a 3D file - for 2D features.
- config_recipe held the generic "Not mode-specific..." blurb; it now names the new recipe
  ngldm.ibsi_phantom_2d, which records the four-slice averaging the assertions actually perform and
  the mirp settings that match it (by_slice, no discretisation, ngldm_distance=1,
  ngldm_difference_level=0).
- source=tracker on every row, i.e. the verdict rested on an offline harness run nobody could
  repeat. All 19 now read audit.
- target_test cleared, since the file it named now exists and is met.

gen_ngldm_mirp.py re-verifies every golden pinned in the header it feeds and exits non-zero on a
mismatch or on a pin it cannot produce (17/17, rel=0). It reads the phantom out of tests/test_data.h
through oracles/ibsi_phantom.py rather than carrying a second copy of the voxels.

Test-file hygiene, per the revet.txt checklist:
- test_2d_ngldm_ibsi.h uses std::string, std::vector and std::unordered_map but included only
  <unordered_map>; test_2d_ngldm_regression.h uses std::string and included none of them, relying on
  the ibsi header to drag it in. Both now include what they use.
- test_2d_ngldm_regression.h's include comment named assert_ngldm_feature_ibsi, which it does not
  call - it uses the shared assert_ngldm_feature_against_golden_values.
- The IBSI table carried two commented-out entries pinning -1 for NGLDM_GLM and NGLDM_DCM. A
  commented-out golden is a trap rather than documentation - uncommenting it pins -1 - so they are
  replaced by a note at the table head saying why those two features are absent and where they live.
- The 17 features are asserted by both oracle files, which is deliberate rather than redundant; the
  generated audit CSV now says so per row, so a later cleanup does not delete one as a duplicate.

TOOLS.md gains the two mirp gotchas this family hit: its per-family numeric settings must be floats
(ngldm_distance=1 raises TypeError during settings construction, before any image is read), and its
NGLDM column names do not track the Nyxus abbreviations (ngl_ldlge = NGLDM_LDLGLE, ngl_dc_perc =
NGLDM_DCP), so they must be mapped by meaning rather than string similarity.

Verified: runAllTests 790/790 on Windows and under ASan+UBSan on Linux; pytest tests/python/ full
directory; check_coverage.py --check, check_test_names.py --check and
audit/scan_ngldm_coverage.py --check all clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant