Skip to content

A catches(ERR-NNN) marker on a test the defect does not redden is a phantom catcher — the harness adjudicates verifies, not catches #455

Description

@deOliveira-R

Finding (#448 verification round, 2026-09-06)

The test-architect marked a new gate class with @pytest.mark.catches("ERR-083") and then measured, with the battery arm that models ERR-083's own defect (the ℓ ≥ 1 emission dropped inside the finalize), that 9 of 9 rows in that class stay green. The marker was a false claim — the class pins a DIFFERENT property (System B's ψ½ member equals the converged one) — and it was corrected only because the same agent ran the mutation. Nothing in the tree would have caught it:

  • sphinx -W (via the nexus merge) warns on a catches naming an undeclared entry — the entry existed, so it was silent.
  • tests/_harness/audit.py has _phantom_verifies (a verifies naming a label that no longer exists) and no _phantom_catches; and neither checks that the marked test actually reddens under the catalogued defect.
  • mcp__nexus__verification_audit(run=…) adjudicates verifies claims against a coverage capture (corroborated / refuted); errors reports which entries have catchers, not whether the catchers catch.

So an ERR entry's catcher list is an authored, unfalsifiable claim (vv-principles #17's shape at the catalogue tier): a defect can read as "caught by 5 tests" while 1 of the 5 would redden.

Proposal

The entry's catalogued defect already has, for most ERR entries, a mutation the battery memo names. Record it beside the entry (a .. error-entry:: field, e.g. :mutation: <plugin arm id>), and let a harness gate (opt-in, slow tier) run each entry's arm once and assert every catches(ERR) test reddens — an executed adjudication of the catcher relation, the dual of verification_audit(run=) for verifies. Until then, the discipline is procedural: a catches marker is placed only AFTER the mutation arm for that ERR has been seen to redden the test (scratch/_448_verification_plan.md §12.9 records the #448 instance).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    module:teststests/ verification infrastructuretype:improvementEnhancement to existing functionality

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions