From cc6d5f3b7b6b68a257412c5d90b41fe28dcacd90 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 31 Aug 2026 02:03:38 +0000 Subject: [PATCH] docs: say where the tests are and how to reproduce this tree A review on the GitHub Copilot marketplace submission flagged that these entries "expose externally implemented security enforcement without repository-local integration tests". The observation is correct and the gap is real, but it is a documentation gap rather than a testing one -- and it will be raised by every marketplace we submit to, so it is worth closing properly rather than answering in a comment thread. The policies ARE tested. Every policy in the catalog ships an eval suite at base//evals/suite.yaml, and publish.yml runs `chock check` and `chock check --only evals` as a gate before anything is packaged, so a policy whose evals fail cannot reach this repository at all. None of that was discoverable from here: the word "eval" did not appear in this README or in PLUGINS.md. A reviewer following the listing's repository link landed on a page that asserted trustworthiness without showing any way to check it. So the Trust section now states where the tests live and why they live there -- the catalog holds the policy source; this repository is compiled output -- and adds a runnable reproduction: clone this tree, the catalog and the framework at the release it was published from, rebuild, and diff. Two silent git commands mean the tree is byte-identical to a fresh build. The recipe was executed against this repository before being written down, not derived from reading the workflow, because a verification procedure nobody has run is exactly the kind of unearned claim this section exists to avoid. Also stated plainly: this README is the one file the publisher never writes, so it alone sits outside the generated-only guarantee. A page explaining what may be trusted should be honest about its own standing. Pinning guidance is included because the failure it prevents is concrete. A marketplace that pins a tag alone follows a movable reference; a tag plus the full commit SHA is what holds reviewed bytes still. No generated content is touched, so generated-only is unaffected. Signed-off-by: Claude --- README.md | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/README.md b/README.md index b41e273..2b355ef 100644 --- a/README.md +++ b/README.md @@ -68,6 +68,37 @@ chock init && chock sync --ci their framework sources. - **Best-effort, not a boundary:** guards are pattern-based filters. See [SECURITY.md](https://github.com/open-coder-ai/chock/blob/main/SECURITY.md). +- **Tested upstream, and gated:** every policy ships an eval suite + (`base//evals/suite.yaml`) in the catalog, and the publish workflow runs + `chock check` and `chock check --only evals` before packaging anything — a policy whose + evals fail cannot reach this repository. The tests live in the catalog because the policy + source does; this repository is compiled output. +- **This README is the exception:** it is the one file the publisher never writes, so it + alone sits outside the generated-only guarantee. Everything else here regenerates. + +### Verify it yourself + +Nothing above asks for trust that cannot be checked. This rebuilds the published tree from +source and compares it with what is committed here: + +```bash +git clone https://github.com/open-coder-ai/chock-codex-plugins dist +git clone --branch v0.7.0 https://github.com/open-coder-ai/chock framework +git clone https://github.com/open-coder-ai/chock-catalog catalog +pip install ./framework +chock plugin build --repo catalog --policies-dir base --format codex --out-dir dist +chock marketplace build --dist dist --tree codex +git -C dist diff --exit-code && git -C dist status --porcelain +``` + +Silence from both `git` commands means this repository is byte-identical to a fresh build +from the catalog. `--branch v0.7.0` is the framework release this tree was published from. +`chock-market.lock` records a sha256 per published plugin directory, so one package can be +checked without rebuilding the rest. + +**If you are listing these plugins in a marketplace,** pin both a tag and the full commit +SHA. The tag names the release; the SHA is what holds the reviewed bytes still. + ## License