test(ngldm): assert the 2D NGLDM features against mirp, and fix a 50% tolerance - #432
Open
darkclad wants to merge 1 commit into
Open
test(ngldm): assert the 2D NGLDM features against mirp, and fix a 50% tolerance#432darkclad wants to merge 1 commit into
darkclad wants to merge 1 commit into
Conversation
… tolerance The registry has claimed status=vetted, oracle=mirp for all 19 2D NGLDM rows since the tracker was imported, with target_test=test_2d_ngldm_mirp.h - a file that did not exist. This runs that oracle for the first time and creates the file. mirp reproduces Nyxus to 2.9e-16 on all 17 IBSI NGLDM features, so the new test_2d_ngldm_mirp.h pins them at the SPEC 7 exact tier (rel=1e-9). The claim was right; it just had no test. The tolerance defect -------------------- assert_ngldm_feature_against_golden_values() ended in agrees_gt(aveTotal, reference, 2.), and agrees_gt computes tolerance = ground_truth / frac_tolerance - so a factor of 2 is a +/-50% band. It applied to all 17 IBSI assertions and both regression assertions, i.e. none of them could have detected a doubled or halved feature value. Measured agreement is 0.45% worst case against the IBSI consensus and 2.9e-16 against mirp, so the band was ~110x looser than the data requires. The helper now takes the tolerance as a parameter and each caller states its own: - test_2d_ngldm_ibsi.h at rel=1e-2. Its goldens are the published IBSI consensus values, quoted to three significant figures; the worst residual is NGLDM_GLNU at 0.45% (10.2 published against 10.2464 computed) and every other feature is under 0.2%. - test_2d_ngldm_mirp.h at rel=1e-9, seven orders of magnitude above the measured noise. - test_2d_ngldm_regression.h at rel=1e-9. Those pin Nyxus' own output to 17 digits, so a drift guard should catch any change at all. The two oracle files are complementary rather than redundant: IBSI fixes the definition, mirp fixes the digits. Tightening broke nothing - the suite goes 773 -> 790 tests, all passing. NGLDM_GLM and NGLDM_DCM ----------------------- Demoted to status=regression. mirp exposes no grey-level-mean or dependence-count-mean column because neither is an IBSI NGLDM feature, and test_2d_ngldm_ibsi.h already marked both "--not in IBSI--". No oracle can reproduce them, so their rows stop claiming oracle=mirp. Registry corrections applied to all 19 rows ------------------------------------------- - current_test named test_3d_ngldm_regression.h - a 3D file - for 2D features. - config_recipe held the generic "Not mode-specific..." blurb; it now names the new recipe ngldm.ibsi_phantom_2d, which records the four-slice averaging the assertions actually perform and the mirp settings that match it (by_slice, no discretisation, ngldm_distance=1, ngldm_difference_level=0). - source=tracker on every row, i.e. the verdict rested on an offline harness run nobody could repeat. All 19 now read audit. - target_test cleared, since the file it named now exists and is met. gen_ngldm_mirp.py re-verifies every golden pinned in the header it feeds and exits non-zero on a mismatch or on a pin it cannot produce (17/17, rel=0). It reads the phantom out of tests/test_data.h through oracles/ibsi_phantom.py rather than carrying a second copy of the voxels. Test-file hygiene, per the revet.txt checklist: - test_2d_ngldm_ibsi.h uses std::string, std::vector and std::unordered_map but included only <unordered_map>; test_2d_ngldm_regression.h uses std::string and included none of them, relying on the ibsi header to drag it in. Both now include what they use. - test_2d_ngldm_regression.h's include comment named assert_ngldm_feature_ibsi, which it does not call - it uses the shared assert_ngldm_feature_against_golden_values. - The IBSI table carried two commented-out entries pinning -1 for NGLDM_GLM and NGLDM_DCM. A commented-out golden is a trap rather than documentation - uncommenting it pins -1 - so they are replaced by a note at the table head saying why those two features are absent and where they live. - The 17 features are asserted by both oracle files, which is deliberate rather than redundant; the generated audit CSV now says so per row, so a later cleanup does not delete one as a duplicate. TOOLS.md gains the two mirp gotchas this family hit: its per-family numeric settings must be floats (ngldm_distance=1 raises TypeError during settings construction, before any image is read), and its NGLDM column names do not track the Nyxus abbreviations (ngl_ldlge = NGLDM_LDLGLE, ngl_dc_perc = NGLDM_DCP), so they must be mapped by meaning rather than string similarity. Verified: runAllTests 790/790 on Windows and under ASan+UBSan on Linux; pytest tests/python/ full directory; check_coverage.py --check, check_test_names.py --check and audit/scan_ngldm_coverage.py --check all clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The registry has claimed status=vetted, oracle=mirp for all 19 2D NGLDM rows since the tracker was imported, with target_test=test_2d_ngldm_mirp.h - a file that did not exist. This runs that oracle for the first time and creates the file.
mirp reproduces Nyxus to 2.9e-16 on all 17 IBSI NGLDM features, so the new test_2d_ngldm_mirp.h pins them at the SPEC 7 exact tier (rel=1e-9). The claim was right; it just had no test.
The tolerance defect
assert_ngldm_feature_against_golden_values() ended in agrees_gt(aveTotal, reference, 2.), and agrees_gt computes tolerance = ground_truth / frac_tolerance - so a factor of 2 is a +/-50% band. It applied to all 17 IBSI assertions and both regression assertions, i.e. none of them could have detected a doubled or halved feature value. Measured agreement is 0.45% worst case against the IBSI consensus and 2.9e-16 against mirp, so the band was ~110x looser than the data requires.
The helper now takes the tolerance as a parameter and each caller states its own:
NGLDM_GLM and NGLDM_DCM
Demoted to status=regression. mirp exposes no grey-level-mean or dependence-count-mean column because neither is an IBSI NGLDM feature, and test_2d_ngldm_ibsi.h already marked both "--not in IBSI--". No oracle can reproduce them, so their rows stop claiming oracle=mirp.
Registry corrections applied to all 19 rows
gen_ngldm_mirp.py re-verifies every golden pinned in the header it feeds and exits non-zero on a mismatch or on a pin it cannot produce (17/17, rel=0). It reads the phantom out of tests/test_data.h through oracles/ibsi_phantom.py rather than carrying a second copy of the voxels.
Test-file hygiene, per the revet.txt checklist:
TOOLS.md gains the two mirp gotchas this family hit: its per-family numeric settings must be floats (ngldm_distance=1 raises TypeError during settings construction, before any image is read), and its NGLDM column names do not track the Nyxus abbreviations (ngl_ldlge = NGLDM_LDLGLE, ngl_dc_perc = NGLDM_DCP), so they must be mapped by meaning rather than string similarity.
Verified: runAllTests 790/790 on Windows and under ASan+UBSan on Linux; pytest tests/python/ full directory; check_coverage.py --check, check_test_names.py --check and audit/scan_ngldm_coverage.py --check all clean.