Skip to content

docs: say where the tests are and how to reproduce this tree - #3

Draft
open-coder-ai wants to merge 1 commit into
mainfrom
docs/verifiable-provenance
Draft

docs: say where the tests are and how to reproduce this tree#3
open-coder-ai wants to merge 1 commit into
mainfrom
docs/verifiable-provenance

Conversation

@open-coder-ai

@open-coder-ai open-coder-ai commented Aug 31, 2026

Copy link
Copy Markdown
Owner

Why

A Copilot review on the GitHub marketplace submission flagged that our entries "expose externally implemented security enforcement without repository-local integration tests."

The observation is correct, and it will be raised by every marketplace we submit to — so this closes it in the repository rather than in one comment thread.

The gap was documentation, not testing

The policies are tested. Every policy in the catalog ships base/*/evals/suite.yaml, and publish.yml runs chock check and chock check --only evals as a gate before anything is packaged — a policy whose evals fail cannot reach this repository.

None of that was discoverable from here. The word "eval" appeared zero times in this README and in PLUGINS.md. A reviewer following the listing's repository: link landed on a page asserting trustworthiness with no way to check it.

What this adds to the Trust section

  • Where the tests are, and why they're there — the catalog holds policy source; this repo is compiled output. Tests belong with source.
  • A runnable reproduction — clone this tree, the catalog and the framework at the release it was published from, rebuild, diff. Two silent git commands mean byte-identical.
  • An honest caveat — this README is the one file publish.yml never writes, so it alone sits outside the generated-only guarantee. A page about what may be trusted should say where it stands itself.
  • Pinning guidance — tag and full SHA. A tag alone is a movable reference; the SHA is what holds reviewed bytes still.

The recipe was run, not derived

I executed it against this repository before writing it down — a verification procedure nobody has run is exactly the unearned claim this section exists to avoid.

chock-claude-plugins     CLEAN
chock-cursor-plugins     CLEAN
chock-codex-plugins      CLEAN
chock-copilot-plugins    CLEAN

All four reproduce byte-identically from catalog ed2930e + chock v0.7.0.

Checks

  • No generated content touched — README only, so generated-only is unaffected.
  • Bullet style matches this repo's existing Trust section.

Noted while verifying — not fixed here

generated-only.yml checks out open-coder-ai/chock with no ref:, so it verifies against a moving main, while publish.yml builds from a pinned framework_ref. They agree today only because chock's two commits past v0.7.0 are CI-only and don't touch the emitter. The next chock merge that changes emitter output turns this repo's verify red on an untouched tree. Worth its own change; deliberately not bundled with a docs PR.

A review on the GitHub Copilot marketplace submission flagged that these
entries "expose externally implemented security enforcement without
repository-local integration tests". The observation is correct and the
gap is real, but it is a documentation gap rather than a testing one --
and it will be raised by every marketplace we submit to, so it is worth
closing properly rather than answering in a comment thread.

The policies ARE tested. Every policy in the catalog ships an eval suite
at base/<policy>/evals/suite.yaml, and publish.yml runs `chock check` and
`chock check --only evals` as a gate before anything is packaged, so a
policy whose evals fail cannot reach this repository at all. None of that
was discoverable from here: the word "eval" did not appear in this README
or in PLUGINS.md. A reviewer following the listing's repository link
landed on a page that asserted trustworthiness without showing any way to
check it.

So the Trust section now states where the tests live and why they live
there -- the catalog holds the policy source; this repository is compiled
output -- and adds a runnable reproduction: clone this tree, the catalog
and the framework at the release it was published from, rebuild, and diff.
Two silent git commands mean the tree is byte-identical to a fresh build.
The recipe was executed against this repository before being written down,
not derived from reading the workflow, because a verification procedure
nobody has run is exactly the kind of unearned claim this section exists
to avoid.

Also stated plainly: this README is the one file the publisher never
writes, so it alone sits outside the generated-only guarantee. A page
explaining what may be trusted should be honest about its own standing.

Pinning guidance is included because the failure it prevents is concrete.
A marketplace that pins a tag alone follows a movable reference; a tag
plus the full commit SHA is what holds reviewed bytes still.

No generated content is touched, so generated-only is unaffected.

Signed-off-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants