Fix the CLI's failure paths, and test them - #26
Merged
Conversation
Closes #22. `evaluate()` was called outside the try/except that turns configuration problems into a sentence and exit 2. So a rulepack asking for a check this build cannot perform — a missing pattern file, an unimplemented check type, the SynthID extra absent — reached the user as a Python traceback and exit 1. Exit 1 is this tool's word for "a rule failed". A pipeline could not tell "your endpoint is not compliant" from "markproof fell over", which is exactly the ambiguity probe_failure_finding() exists to prevent one layer down: the CLI undid the guarantee the engine was careful to give. Evaluation now sits inside the guarded block, and the check-layer exceptions map to exit 2 with a readable sentence — KeyError included, whose default stringification is its quoted argument and therefore a filename with no verb. None of this was caught because cli.py sat at 21% coverage while the library around it was near 90%. The aggregate looked healthy; the distribution was the problem, and what was unprotected is precisely what an adopter depends on. tests/test_cli.py covers, through the real CLI runner: exit codes 0 for clean, 1 for a failed rule, 2 for a run that could not happen, and — the one that matters most — an unreachable endpoint exiting 1 rather than passing quietly. failure paths malformed YAML, a config that is not a mapping, an unknown rulepack (whose error names one that exists), a missing pattern file. Each asserts no traceback reaches the user. reports that none is written without --report-dir, that both files appear with it, and that the report names the endpoint and pins the rulepack digest. signing a valid key round-tripping through verify-report, and the guard that had never been exercised: when signing fails, nothing is written at all. An unsigned file under a name the caller asked to have signed would look like evidence. verify-report a tampered verdict rejected, a non-JSON file and a missing file answered with a sentence. keygen that the private key is not group- or world-readable. cli.py 21% -> 88%; the suite overall 82% -> 89%. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #22. Fourth step of the audit remediation.
The defect
evaluate()sat outside thetry/exceptthat turns configuration problemsinto one sentence and exit 2. So a rulepack asking for a check this build cannot
perform reached the user as a traceback and exit 1.
Exit 1 is this tool's word for a rule failed. A pipeline could not tell
from
— which is exactly the ambiguity
probe_failure_finding()exists to prevent onelayer down. The engine is careful to make "could not be checked" a distinct,
loud outcome; the CLI undid that guarantee one frame up.
The reachable case is not exotic: the README's own example config on a base
install, where
MPF-T-001needs the[synthid]extra.Why it survived
cli.pywas at 21% coverage while the library around it was near 90%. Theaggregate read as healthy; the distribution was the problem, and the untested part
was the one every adopter's pipeline depends on.
What is now tested
tests/test_cli.py, through the real Typer runner rather than by callinginternals:
--report-dir; with it, the report names the endpoint and carries a 64-char rulepack digestverify-reportround trip, and the guard nothing had ever exercised: when signing fails, no file is written at allThe signing guard is the one worth calling out. Writing an unsigned report under a
name the caller asked to have signed would produce something that looks like
evidence and is not; the code was right and unprotected.
One test needed correcting while writing it: the tamper case originally rewrote a
PASStoPASS, which would have passed while proving nothing. It now changes acount and the target name, and asserts the signature breaks.
Numbers
cli.py21% → 88%. Suite overall 82% → 89%. 429 tests, ruff + mypy strictclean.
🤖 Generated with Claude Code