Skip to content

ci: derive the framework ref from the catalog instead of a typed default - #4

Draft
open-coder-ai wants to merge 2 commits into
mainfrom
ci/derive-framework-ref-from-catalog
Draft

ci: derive the framework ref from the catalog instead of a typed default#4
open-coder-ai wants to merge 2 commits into
mainfrom
ci/derive-framework-ref-from-catalog

Conversation

@open-coder-ai

@open-coder-ai open-coder-ai commented Aug 31, 2026

Copy link
Copy Markdown
Owner

Lead with the surprises

1. The dry-run bug is worse than "the stat misleads a reader" — it made the publish itself a silent no-op.
git diff reports tracked files only, so on a publish whose only change is added files the review step printed nothing and the Publish step's own git diff --quiet gate was clean, printed No change to publish., exited 0, and pushed nothing. Reproduced below with the workflow's own step bodies: origin/main does not move.

2. verify and publish were never checking the same thing. generated-only.yml checked out the framework with no ref: at all — floating main — while publish.yml built from a pinned tag. They agree today only because chock main is two CI-only commits past v0.7.0; I confirmed this repository's published tree reproduces clean under both main (4cd85b3) and v0.7.0 (a1fb5bb), which is the only reason this has not already gone red.

3. My first old-vs-new test rig was contaminated and I had to throw it out. Steps on a runner share one working tree, so the intent-to-add records written by the review step are still in the index when the publish step runs. A rig that reuses one copy silently hands the old publish body the new body's staging, and the first run showed the old body committing correctly. The results below run each variant from a pristine copy with its own bare origin.

What changed

Three pins name the framework and none were related: the catalog's .framework-ref (v0.7.0), publish.yml's framework_ref default (v0.4.0 for three releases before being corrected), and generated-only.yml's framework checkout (no ref at all). Both workflows now derive from chock-catalog/.framework-ref:

  • generated-only.yml checks out the catalog first, reads .framework-ref into a step output, then checks out the framework at that ref. Verify now reproduces exactly what a publish from that catalog ref would produce.
  • publish.yml keeps the dispatch input — a human override is why it exists — but defaults to the sentinel auto, resolved in a step from the catalog ref being published. A sentinel rather than an empty string so the resolved value is visible in the run log and a blank input resolves the same way. The commit message a publish writes now records the resolved ref, not the literal input.
  • The comment that failed is replaced, not restated. It said "Bump this default with each framework release" and was skipped three times; prose does not execute. The new one says what the value means — the emitter version whose output is this repository's content.
  • Task 2: git add -A -N . before the stat, and in the Publish gate for the same reason.
  • Task 3: SECURITY.md, assets/icon.svg, .github/workflows/hol-plugin-scanner.yml.

The change is the same in all four distribution repositories; the four run: bodies touched here are byte-identical across them (compared by sha256 of the extracted script, not by eye).

Evidence

Generated content is untouched — before and after

Mirrors the generated-only job: dist, catalog and framework as siblings, pip install ./framework, this repo's own build lines, then both git diff --exit-code and git status --porcelain (the latter because git diff alone will not show added files — the very bug in Task 2).

Before any change, framework at main (4cd85b3):

### chock version: 0.7.0
### step: marketplace build --check
Marketplace index matches the plugin tree (21 plugins).
  -> exit 0
### step: regenerate
  $ chock plugin build --repo catalog --policies-dir base --format agent-plugins --out-dir dist
Packaged 21 policies (format: agent-plugins)
  $ chock plugin build --repo catalog --policies-dir base --format claude --out-dir dist
Packaged 21 policies (format: claude)
  $ chock plugin build --repo catalog --policies-dir base --format copilot --out-dir dist
Packaged 21 policies (format: copilot)
  $ chock marketplace build --dist dist
Indexed 21 plugins into .claude-plugin/marketplace.json and .github/plugin/marketplace.json
Wrote chock-market.lock: sha256 per published plugin directory
Wrote PLUGINS.md: 21 plugins with their posture
### git diff --exit-code
  -> exit 0
### git status --porcelain
  -> (0 lines)

After the change, framework at v0.7.0 — the ref the new workflow derives:

### chock version: 0.7.0
### step: marketplace build --check
  -> exit 0
### step: regenerate
Packaged 21 policies (format: agent-plugins)
Packaged 21 policies (format: claude)
Packaged 21 policies (format: copilot)
Indexed 21 plugins into .claude-plugin/marketplace.json and .github/plugin/marketplace.json
### git diff --exit-code
  -> exit 0
### git status --porcelain
  -> (0 lines)

So SECURITY.md, assets/icon.svg and the new workflow do not break generated-only.

The derivation actually derives — run against this repository's committed workflows

The resolution step's script is extracted from the committed YAML (not retyped) and executed with bash --noprofile --norc -e -o pipefail, as the runner does, against a catalog directory whose .framework-ref is varied. The value read back is what the step wrote to $GITHUB_OUTPUT:

chock-catalog/.framework-ref contains: 'v0.7.0\n'

[generated-only ] real catalog value             exit=0  resolved ref='v0.7.0'
[generated-only ] changed .framework-ref         exit=0  resolved ref='v9.9.9'
[generated-only ] empty .framework-ref           exit=1  resolved ref=None
[generated-only ] no .framework-ref at all       exit=1  resolved ref=None
[publish        ] default input `auto`           exit=0  resolved ref='v0.7.0'
[publish        ] `auto` + changed .framework-ref exit=0  resolved ref='v9.9.9'
[publish        ] blank input                    exit=0  resolved ref='v0.7.0'
[publish        ] explicit override              exit=0  resolved ref='v0.4.0'
[publish        ] `auto` + no .framework-ref     exit=1  resolved ref=None

Changing .framework-ref changes what gets checked out; a missing or empty file fails loudly instead of silently defaulting; an explicit ref still wins.

The dry run, old vs new, each on a pristine copy

Run in chock-codex-plugins — the step bodies are byte-identical to this repository's, so the behaviour is the same, but the run itself was there and I am not going to claim otherwise. Both bodies extracted from the committed YAML; origin is a local bare repo so the push path runs for real.

### scenario: noop
  ground truth, git status --porcelain: 0 entries
  OLD              review step  exit=0:
      <no output>
  OLD              publish step exit=0: No change to publish.
  OLD              origin/main after: 6b2a748 ci: derive the framework ref ...
  NEW (committed)  review step  exit=0:
      <no output>
  NEW (committed)  publish step exit=0: No change to publish.
  NEW (committed)  origin/main after: 6b2a748 ci: derive the framework ref ...

### scenario: additions_only
  ground truth, git status --porcelain: 1 entries
  OLD              review step  exit=0:
      <no output>
  OLD              publish step exit=0: No change to publish.
  OLD              origin/main after: 1b42c71 pretend this plugin is not published yet
  NEW (committed)  review step  exit=0:
      codex/token-efficiency/.codex-plugin/plugin.json      | 17 +++++++++++++++++
       .../token-efficiency/skills/token-efficiency/SKILL.md | 19 +++++++++++++++++++
       2 files changed, 36 insertions(+)
  NEW (committed)  publish step exit=0: [main b8db043] generated: catalog main (framework v0.7.0)
  NEW (committed)  origin/main after: b8db043 generated: catalog main (framework v0.7.0)

The no-op path is unchanged. The additions-only path goes from "reports nothing and publishes nothing" to "reports 36 insertions and publishes them".

Linters

actionlint v1.7.7 (the version security.yml installs) and zizmor --offline --min-severity low .github/workflows/, run over all five workflows in this repository:

  actionlint exit=0
 INFO zizmor: 🌈 zizmor v1.30.0
No findings to report. Good job! (3 ignored, 17 suppressed)
  zizmor exit=0

Both were clean before the change too, so this neither introduces nor hides a finding. No persist-credentials: false was weakened; the dist checkout in publish.yml remains the documented artipacked exception.

Notes on the Task 3 files

  • SECURITY.md routes a report to where it can be fixed rather than inventing a surface this repository does not have: emitter and packaging defects to chock, policy content to chock-catalog, and only distribution-integrity findings genuinely here — a fix landed on a generated file is overwritten by the next publish. No SLA and no PGP key are invented; it points at chock's advisory route and its timelines.
  • assets/icon.svg is the existing org mark from org-plan's brand/, byte-identical except for explicit width="512" height="512" alongside the existing viewBox, so "512×512" is unambiguous to a consumer that does not read viewBox. No new mark was invented.
  • hol-plugin-scanner.yml runs the scanner on push and pull_request as a report, not a gate, and the workflow comment says why: two of these plugins are flagged HIGH for quoting the very patterns they block, so a gate armed on that would train people to route around it. submission_enabled is pinned false explicitly rather than left to the default, because that input can open submission issues in third-party catalogue repositories and that is an owner decision. The action is pinned by commit f344c80c093464413432b399f38a7ba2f3365ae2, resolved from v1 and equal to tag v1.2.560; v1 has been re-pointed across 560 releases, so @v1 would let the code running here change without a diff in this repository.

What I could not run

  • GitHub Actions itself. Every workflow claim above is the committed step body executed locally under the runner's shell flags, not a runner execution. CI on this PR is the first real run.
  • The claude plugin validate steps in this repository's workflows were not exercised; the change does not touch them.
  • Python version. CI pins 3.12; this container has 3.11.15. The build output was byte-identical to the committed tree on 3.11, but a 3.12-only difference would not have surfaced here.
  • zizmor version. CI runs zizmor-action v0.6.2; locally the CLI is v1.30.0.
  • The scanner workflow has never executed. Its inputs are checked against the action's action.yml at the pinned commit, nothing more.

Generated by Claude Code

@claude
ci: derive the framework ref from the catalog instead of a typed default
387e304 
Three pins name the framework and none of them were related to each other.
The catalog declares v0.7.0 in .framework-ref. publish.yml's framework_ref
input carried a hardcoded default that sat at v0.4.0 across three releases
before it was corrected. generated-only.yml checked out the framework with
no ref at all -- floating main.

So verify has been checking a fixed, published tree against a *moving*
emitter while publish builds from a pinned one. They agree today only
because chock main happens to be two CI-only commits past v0.7.0. The next
merge that changes emitter output turns this check red on a tree nobody
touched, and the failure would read as tampering in the one repository
whose entire guarantee is "this content is what the catalog published".

Both workflows now derive the ref from chock-catalog/.framework-ref:

- generated-only checks out the catalog first, reads .framework-ref into a
  step output, and checks out the framework at that ref. Verify now
  reproduces exactly what a publish from that catalog ref would produce.
- publish keeps the dispatch input -- a human overriding it is the reason
  it exists -- but its default is the sentinel `auto`, resolved in a step
  from the catalog ref being published. A sentinel rather than an empty
  string so the resolved value is visible in the run log and a blank input
  resolves the same way. The commit message the publish writes now records
  the resolved ref rather than the literal input.

The comment that failed is replaced rather than restated. It said "Bump
this default with each framework release" and was skipped three times;
prose does not execute. The new one says what the value means -- the
emitter version whose output *is* this repository's content -- and the
mechanism now makes the correct value the default.

The dry run was hiding every added file. `git --no-pager diff --stat`
reports tracked files only, so a publish's additions are invisible and a
human reading it to decide whether to publish is shown only what
disappears. On the v0.7.0 publish that printed "15 files changed, 14
insertions(+), 2450 deletions(-)" for a rename into a larger file; the
real change was 5677 insertions. `git add -A -N .` first, which records
paths without staging content, so the worktree and the later commit are
unchanged.

The Publish step's own gate had the same defect with a worse consequence:
on a build whose only change is added files, `git diff --quiet` is clean,
so it printed "No change to publish." and exited 0 having published
nothing. Reproduced locally -- exit 0 with an untracked plugin directory
present, exit 1 once intent-to-add is recorded. Staged there too.

Alongside, three hand-maintained files that are hygiene on their own
merits and happen to be what a public plugin catalogue asks for:

- SECURITY.md, which routes a report to where it can be fixed. This
  repository is compiled output, so a fix landed here is overwritten by
  the next publish: emitter and packaging defects go to chock, policy
  content to chock-catalog, and only distribution-integrity findings are
  genuinely this repository's. No SLA and no PGP key are invented; it
  points at chock's advisory route and its timelines.
- assets/icon.svg, the org mark from the brand family, unchanged except
  for explicit width and height so "512x512" is unambiguous to a consumer
  that does not read viewBox.
- .github/workflows/hol-plugin-scanner.yml, running the HOL AI plugin
  scanner on push and pull_request as a report, not a gate. The score
  thresholds stay at the action's non-failing defaults deliberately: the
  scanner flags two of these plugins HIGH for quoting the patterns they
  block, and a gate that fails on a guardrail describing a guardrail only
  teaches people to route around it. submission_enabled is pinned false
  explicitly rather than left to the default, because that input can open
  issues in third-party catalogue repositories. The action is pinned by
  commit, not by the `v1` tag its docs give: `v1` has been re-pointed
  across 560 releases, so `@v1` lets the code running here change without
  a diff in this repository.

None of this touches generated content. Verified per repository by
mirroring the generated-only job locally -- dist, catalog and framework as
siblings, pip install ./framework, the repository's own build lines --
and confirming `git diff --exit-code` and `git status --porcelain` are
both silent, before and after. actionlint and zizmor --offline
--min-severity low are clean.

Signed-off-by: Claude <noreply@anthropic.com>
@open-coder-ai Claude

open-coder-ai commented Aug 31, 2026

Copy link
Copy Markdown
Owner Author

CI has run. Supersedes the "GitHub Actions itself", "Python version" and "zizmor version" bullets under "What I could not run" in the description above, which were written before the first run.

All checks green on the first run: verify, zizmor, actionlint, and the new scan job. redirect skipped, correctly — close-prs.yml guards on fork PRs and this branch is in-repo. verify here also includes this repository's claude plugin validate step, which passed.

The derivation on a real runner, from the verify log on the sibling PR open-coder-ai/chock-codex-plugins#4 (identical step body — same sha256):

Verifying against framework v0.7.0, from chock-catalog/.framework-ref.
##[group]Run actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8
with:
  persist-credentials: false
  repository: open-coder-ai/chock
  ref: v0.7.0
  path: framework
...
HEAD is now at a1fb5bb Merge pull request #82 from open-coder-ai/chore/bump-agentseam-0.1.1
a1fb5bb803b4cfcf1ed61c62b236ba46a4620a4e
...
Successfully set up CPython (3.12.14)
Tree matches a fresh build from the catalog.

The step read .framework-ref, the checkout received a ref: it did not previously have, that resolved to a1fb5bb, and the tree built from it matched — and this repository's own verify job went green on the same mechanism. Note the Python line: the runner used 3.12.14 and the trees still reproduced byte-for-byte, so the 3.11-only local reproduction was not hiding a version difference.

Two caveats stand, and I am not quietly dropping them:

  • publish.yml was never dispatched — publishing is the owner's decision. Its resolution step is a different script from generated-only.yml's (it handles the auto sentinel and the override), so that one is still proven only locally.
  • The scan job passing means the action ran, not that the repository scored well. It is configured as a non-failing report on purpose.

Generated by Claude Code

@claude
ci: drop the HOL plugin scanner workflow
5b2cd5c 
Owner decision 2026-09-01: the scanner's unique value is a semantic check
on shipped plugin content independent of the reproduction check, which
only matters once the catalog takes external contributions or has real
adopters. Today it would run never-exercised third-party code and carry
two permanent HIGH false positives (policies quoting the patterns they
block) — standing noise that teaches readers to ignore the report.
SECURITY.md, the icon, and the framework-ref derivation stay.

Signed-off-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@open-coder-ai @claude