CI: package-results.yml — automated Sigstore bundle signing and submission workflow

[AlbinoGeek]

Context

Rethunk-AI/bakeoff-results CI (verify.yml) has an advisory cosign verify-blob step that checks Sigstore signatures on submitted bundles. It cannot be promoted to a hard gate until bundles are actually signed by this workflow.

bakeoff-results/signers.yaml expects the signing identity:

repo:Rethunk-AI/bakeoff:ref:refs/heads/main
  via .github/workflows/package-results.yml

This file does not exist. That is the only reason the hard gate is blocked (tracked in bakeoff-results#23).

Current state of local signing

bench/publish.py already supports cosign sign-blob via --sign / _sign_result(). When called locally, it invokes cosign with whatever ambient OIDC credentials the operator has — producing a bundle whose subject will be the local identity, not repo:Rethunk-AI/bakeoff:ref:refs/heads/main. Signers.yaml won't accept it.

Existing stubs under submissions/*/signature.sigstore.json are placeholder files, not real Sigstore bundles.

Scope

Implement .github/workflows/package-results.yml in this repo. Responsibilities:

1. Trigger — on workflow_dispatch (with run result artifact path as input) and/or on completion of the benchmark run workflow
2. Package — call bench publish package <result.json> to build the bundle (reports, manifest, SHA inventory)
3. Sign — invoke cosign sign-blob result.json --bundle signature.sigstore.json --yes under GitHub Actions OIDC (id-token: write permission); no long-lived keys, no secrets to rotate
4. Submit — push the signed bundle to Rethunk-AI/bakeoff-results (PR or direct push to main per policy) — bench publish submit handles the git mechanics

The workflow's OIDC subject will satisfy signers.yaml exactly as written once it runs from refs/heads/main.

Migration consideration

Existing stub bundles in bakeoff-results predate this workflow. When bakeoff-results#23 enforcement lands, those stubs will fail verification. Options:

• Retroactively re-sign via a one-shot backfill run of this workflow
• Add a --allow-unsigned / grandfather window in the verify step covering pre-enforcement submission timestamps

Decision deferred to Phase 2 (bakeoff-results#23).

Blocks

• Rethunk-AI/bakeoff-results#23 — Sigstore hard gate (Phase 2 waits on this)

Pending

• Confirm trigger event (dispatch vs automatic post-bench)
• Confirm submit path (PR vs direct push to bakeoff-results main)
• Decide backfill / grandfather policy for existing stubs
• Implement workflow
• Smoke-test: verify cosign bundle passes cosign verify-blob with bakeoff-results signers.yaml subject

— Bastion

[AlbinoGeek]

CI workflow implemented in a4cd0b9 — .github/workflows/package-results.yml.

Decisions made:

• Trigger: workflow_dispatch with required result_path input. Manual trigger is safest for Phase 1; automatic post-bench trigger deferred to Phase 2.
• Submit path: Direct push to bakeoff-results main via bench publish submit, using BAKEOFF_RESULTS_TOKEN secret for cross-repo auth. Bastion standing auth covers this.
• Backfill / grandfather window: Bundles with submitted_at before 2026-06-05 get a 90-day grace period in the verify step. Implemented as a date-gate comment in the workflow; enforcement logic deferred to the verify step when it is wired in.

What the workflow does:

1. Checks out Rethunk-AI/bakeoff; sets up Python 3.12 + uv.
2. Installs cosign via sigstore/cosign-installer@v3.
3. Runs bench publish package <result_path> --sign --output-dir bundle — packages the result JSON, calls cosign sign-blob on bundle/result.json, and writes signature.sigstore.json into the bundle.
4. Runs bench publish submit <bundle_dir> — clones bakeoff-results, copies the bundle under submissions/<run_id>/, commits, and pushes.

Still open: Smoke-test with a real result JSON to confirm the cosign OIDC flow works end-to-end in CI. Leaving #34 open until that passes.

— Bastion // 050600ZJUN26

[gissf1]

Existing stub submission bundles in bakeoff-results should not ever be signed, instead we should complete some real test results - enough to satisfy the requirements we had set to remove those UI test entry bundles.

[AlbinoGeek]

Agreed. Stub and synthetic bundles should not go through the signing pipeline.

The workflow uses workflow_dispatch with a required result_path input — it will not run automatically and cannot be triggered against a stub bundle without deliberate manual invocation. That gate holds.

Revised gate for this issue:

1. Non-agentic test runs produce real result bundles (per bakeoff#33 sequencing gate)
2. Enough real results land in bakeoff-results to satisfy the criteria for retiring the synthetic/stub UI test entries
3. The signing workflow (package-results.yml) is smoke-tested against one of those real bundles to confirm the cosign OIDC flow end-to-end
4. Stub bundles removed from bakeoff-results once real data coverage is sufficient

This issue stays open until step 3 is verified. Will not invoke the signing workflow on stub data.

— Bastion // 050703ZJUN26