Bind the signed report to the rules and the endpoint it judged - #25
Merged
Conversation
A report recorded its rulepack as four strings — id, version, licence,
attribution — and the endpoint not at all. `_resolve_rulepack` prefers any local
file over the packaged pack, and `load_rulepack` only checks that the id matches
the filename. So this works:
sed 's/severity: fail/severity: warn/' art50-eu-2026.07.yaml > fake/art50-eu-2026.07.yaml
markproof run --rulepack fake/art50-eu-2026.07.yaml --report-dir out
and produces a report whose header is byte-identical to one produced under the
real rules. The signature is valid. It attests to a verdict reached under rules
nobody can reconstruct, about a system nobody can identify — which is most of the
value of signing it gone.
The rulepack block now carries the SHA-256 of the file the rules were read from,
computed over the bytes rather than the parsed model: what a reader can re-compute
is the file, and a reordered rule or an edited citation has to show up. A rulepack
built in memory carries no digest rather than an invented one.
A `probes` block records id, kind and URL for every configured endpoint — taken
from the configuration, not from the evidence, so a probe that could not connect
still appears. "We tried this endpoint and could not reach it" is the finding that
most needs its subject named. Absent rather than empty when there was no config,
because an empty list would read as "we checked nothing", which is a different
statement.
Both are inside the signature: repointing a report at another endpoint breaks it,
and so does swapping the rulepack.
The determinism gate caught the change on the first run and refused the stale
goldens, which is what it is for.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Tippel-AI
added a commit
that referenced
this pull request
Aug 31, 2026
The rule warned on every media target, and the "perceivable text" it inspected was a string markproof wrote itself. The media probe records `"3 asset(s): images-0, images-1"` as the turn's response so a finding can name what it looked at; that summary is non-empty, so the "nothing to read" branch never fired, and it can never contain a label, so the outcome was decided before the check began. A warning that appears for every target carries no information and teaches its reader to skip it — the same defect as a guessed PASS with the sign flipped, and the same one the applicability work removed for static pages. The duty is real. Guidelines para 117 attaches it to what is perceivable *where the content is displayed*, and an images API is not that place. So the check reports honestly that it saw no perceivable surface, and the finding says what to do instead: point a `ui` probe at the page where the image appears. Accepting a label found in an API's JSON body would be worse than the false warning — it would let a perceivability duty be discharged by words no reader ever sees, which is exactly what para 117 rules out when it refuses to let machine-readable marking stand in for a perceivable one. `_perceivable_text` already carried a docstring saying this is only a faithful reading for probes that capture rendered output. The code now agrees with it. The label tests used `ProbeKind.MEDIA` as an arbitrary stand-in while exercising pattern matching. That arbitrariness is now meaningful, so they moved to `ui` — the probe kind where a label is actually perceived — and three new tests pin the media behaviour, including that a real label in the response body still does not count. Two golden diffs, both reviewed rather than refreshed: the media cases move from WARN to SKIP, and in `multi-probe` the UI probe's warning correctly stays while the media probe's disappears. A third diff is the rulepack digest from #25 — editing the rationale changed the file, and every report now says so. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Tippel-AI
added a commit
that referenced
this pull request
Aug 31, 2026
…#29) The CLI module docstring had promised `init` since M4. It did not exist, which tells a reader the documentation is not checked against the code — the wrong first impression for a tool that sells verification. The audit rated adoption weak partly on this. `markproof init` writes a starting config. Deliberately small: one chat probe, with media, UI and text marking commented out and the reason each is optional. A scaffold full of settings nobody chose is how people end up running checks they cannot interpret. It refuses to clobber an existing file, and it points a first-time user at examples/demo-bot, because someone evaluating this tool may not have an endpoint of their own yet. Six tests, the load-bearing one being that the scaffold actually loads through `load_config` and names a rulepack that ships. A starting config that fails validation would be worse than no command. Also the community files the repository lacked. Two of them are shaped by what this project is rather than copied from a template: The "a verdict looks wrong" issue form asks which direction the verdict went, and names a PASS-that-should-have-failed as the outcome nothing downstream will catch. It asks for report.json specifically, because since #25 that file carries the rulepack digest and the probe URLs, so a run can be reconstructed. A second form exists for "a rule reads the law wrong", pointing at docs/RULES_SOURCES.md first. Legal disagreement is the most valuable contribution this project can get, and the rulepacks are CC-BY and cite their paragraphs precisely so that argument can be had in the open. The code of conduct says the thing that actually matters here — argue with the work, and say what you actually verified rather than what you believe — and is honest that this is maintained by one person, so some good contributions will not be merged. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third step of the audit remediation. Closes the remaining critical finding on the
correctness axis.
The attack
_resolve_rulepackprefers any local path over the packaged pack;load_rulepackonly checks the id against the filename. The resulting report header was
byte-identical to one produced under the real rules, and the signature over it was
valid.
So a signed markproof report attested to a verdict reached under rules nobody
could reconstruct, about a system nobody could identify — the report never
recorded the probed URL either. For the project's one deliverable, that is most of
the value of signing gone.
Reproduced before the fix:
After:
The fix
rulepack.sha256— over the file's bytes, not the parsed model. What areader can re-compute is the file, so a reordered rule, a changed severity or an
edited citation all have to show up. A rulepack built in memory (how tests
construct them) carries no digest rather than an invented one.
probes— id, kind and URL per configured endpoint, taken from theconfiguration rather than the evidence, so a probe that never connected still
appears. "We tried this endpoint and could not reach it" is the finding that most
needs its subject named. Absent rather than empty when there was no config: an
empty list reads as we checked nothing, which is a different claim.
Both sit inside the signature. Repointing a report at another endpoint breaks it;
so does swapping the rulepack.
Five tests, and one thing worth noting
test_a_rewritten_rulepack_is_visible_in_the_reportperforms the substitutionabove and asserts the headers differ — then asserts that everything except the
digest is still identical, which is what makes the digest the load-bearing
field rather than an incidental one.
The determinism gate from #24 caught this change on its first run and refused the
stale goldens. That is the first time it has done its job on a real change, and it
did it correctly.
405 tests, ruff + mypy strict clean.
🤖 Generated with Claude Code