feat: post-release artifact smoke test (three layers) + macOS notarization pass-through

[trakhimenok]

Summary

Adds a post-release check to release.yml that verifies a published CLI binary actually runs, so a broken artifact fails the release run instead of shipping silently. Prompted by a real incident: specscore-cli v0.24.0 released fully green, but the Homebrew-cask-installed binary was silently blocked by macOS Gatekeeper for every user.

Root cause (confirmed, corrected mid-investigation): not a build/hang bug in the binary. The raw published release asset runs fine. The Go CLIs in this org are ad-hoc signed and not notarized (codesign -dv → Identifier=a.out, TeamIdentifier=not set). Homebrew's cask installer applies com.apple.quarantine to the installed binary; macOS Gatekeeper then blocks an ad-hoc-signed, unnotarized binary the instant it's quarantined, and this is fleet-wide, not specscore-specific.

Three layers (all under artifact_smoke_test, default on)

1. Run the published GitHub release asset (linux/amd64 + darwin/arm64 by default) under an explicit watchdog timeout. Does not prove a Homebrew-cask user can run it (a plain download carries no quarantine attribute).
2. Static macOS code-signing/notarization assertion (codesign -dv, spctl -a -vv) — no execution, no Gatekeeper risk in the check itself. This is what would have caught v0.24.0. The primary, reliable, deterministic signal.
3. brew install --cask + run the installed binary — auto-detected from the repo's own .goreleaser.yaml, no-op otherwise. The only layer that reproduces Homebrew's quarantine attribute.

Severity, and the taxonomy that governs it (see "Review round 2" below): once a binary is actually found and executed, layer 1 always hard-fails on hang/non-zero-exit/empty-output; layers 2/3 do the same but gated by require_notarized_macos (default false — nothing in this org is notarized yet, so a default hard-fail would red every release fleet-wide on day one). But finding the binary to test in the first place — a matching release asset, a recognizable archive, the binary inside it, a working brew install --cask — is never guaranteed across every repo this workflow will ever be attached to, and failing to find it is not evidence of a broken release. That distinction is the round-2 fix below.

Also included (same underlying fix — makes the thing layer 2 detects actually fixable): forwards five previously-declared-but-unwired secrets (MACOS_SIGN_P12, MACOS_SIGN_PASSWORD, NOTARIZE_ISSUER_ID, NOTARIZE_KEY_ID, NOTARIZE_KEY) into GoReleaser's env, so a repo's notarize.macos block (e.g. ingitdb-cli's, dormant since it was written for exactly this reason) can actually activate.

Review round 2: could-not-test vs. tested-and-broken

First review caught a real conceptual gap: the check was conflating "I could not verify this" with "I tested this and it is broken" — only the second justifies failing a release, and the first was failing it too, fleet-wide, with no opt-in. Confirmed live against a real public release (gh release download --pattern <non-matching> exits 1 and propagates under set -euo pipefail) and fixed:

• Taxonomy applied throughout both smoke-test jobs. "Could not test" — no matching release asset, an unrecognized archive format, no binary found inside it, an unresolvable binary name, or a failed brew install --cask — now logs ::warning:: (naming exactly what was looked for and what was found) and skips (exit 0), always, regardless of require_notarized_macos. "Tested and broken" — found and executed, then hung / exited non-zero / printed nothing — is unchanged: always a hard failure for layer 1, require_notarized_macos-gated for layers 2/3. artifact_smoke_test_command (already configurable, default --version) is the escape hatch for a CLI whose default invocation doesn't support a bare --version.
• Layer 3's brew install --cask step was an unconditional hard fail (set -euo pipefail, no severity gate at all — require_notarized_macos: false gave zero protection). A transient Homebrew/network failure, tap-visibility timing right after GoReleaser pushes the cask manifest, or a cask-name mismatch now warns and skips the rest of the job instead.
• Nondeterministic artifact selection fixed. The old find excluded only .txt, so --pattern "*_linux_amd64.*" could also match a same-named .sig/.sbom/.intoto.jsonl sidecar, with filesystem order deciding which one got picked — flapping pass/fail across identical releases. gh release download now uses explicit --pattern flags for .tar.gz/.tgz/.zip only; sidecars are never even downloaded.
• Timeouts added to every network step. gh release download (120s) and brew tap/brew install --cask (300s) previously had no watchdog, bounded only by GitHub's 6-hour default job timeout — exactly what the binary-invocation watchdog exists to avoid relying on. Each smoke-test job also now has an explicit timeout-minutes (10/15/20) as a backstop.
• Subdirectory-module tagging handled. resolve_artifact_smoke_test now reads tag_prefix; if it looks like a subdirectory-module path (e.g. "ingitdb/v") and no GoReleaser config is found at the repo root, binary-name inference refuses to guess from the repo name (a subdirectory module's name isn't the repo's) and the whole smoke test is skipped with a warning instead. Checked directly against every current consumer of this workflow: none combine that today, but the workflow is reused by repos not enumerated here, so this stays defensive rather than assumed safe.

Re-validated in CI after the rewrite, same method as before (throwaway workflows, removed before this PR, run on real GitHub-hosted runners against the real specscore-cli v0.24.0 release):

• Happy path (download → extract → find binary) still works end to end after the rewrite.
• A deliberately non-matching platform (linux/riscv64, doesn't exist for specscore-cli): download step warns, job is green — confirms finding 1/2's fix.
• A deliberately wrong Homebrew cask name: install step warns, job is green — confirms finding 3's fix.

Validated in CI, not just locally (round 1)

Ran the layer 2/3 logic on a real GitHub-hosted macos-latest runner against the real, known-bad specscore-cli v0.24.0 Homebrew-cask binary:

• Layer 1 (raw asset): PASS, as expected — confirms the artifact itself is fine.
• Layer 2 (codesign/spctl): correctly flags it — TeamIdentifier: not set, spctl rejected (exit 3).
• Layer 3 (brew install --cask + run): did not hang — completed after a ~30s Gatekeeper assessment delay and exited 0. Confirmed residual gap: a GitHub Actions runner has no interactive session, so it can't reproduce the real, indefinite dialog a logged-in user hits. Documented prominently in the workflow comments and README rather than papered over — layer 2 is positioned as the reliable signal, layer 3 as corroborating only.

That same CI run also caught a real bug before it shipped: GitHub Actions' default shell: bash invocation is bash -e {0} regardless of what the script itself sets, so set -uo pipefail alone does not disable errexit — output="$(run_with_timeout ...)" and spctl_out="$(spctl ...)" were aborting the step before the warn-vs-fail branching logic ever ran. Fixed with an explicit set +e in every custom script. Verified clean with actionlint (shellcheck-backed) against all workflow files throughout, including after the round-2 rewrite.

Platform coverage

• linux/amd64 and darwin/arm64 (an Apple Silicon GitHub-hosted runner — the platform the incident shipped on) by default, via artifact_smoke_test_platforms (JSON array of {runner, goos, goarch}, consumed as a job matrix). Extend/trim as needed.
• Windows is not covered by default (out of scope for this pass; artifact_smoke_test_platforms can be extended to add it).

What a consumer sets

Nothing, to get layer 1 (hard-fail once tested) and layers 2/3 (warn-only once tested) automatically — all seven new inputs are optional with backward-compatible defaults. Existing @v1 callers are unaffected until the maintainer advances the v1 tag (see "Releasing this repo" in the README); no @v1 caller needs any YAML change.

To customize:

with:
  # artifact_smoke_test: true
  # artifact_smoke_test_binary: ''                 # '' infers from .goreleaser.y*ml / repo name
  # artifact_smoke_test_command: '--version'
  # artifact_smoke_test_timeout_seconds: 30
  # artifact_smoke_test_homebrew_cask: true
  # require_notarized_macos: false                 # flip once a repo is actually notarized
secrets:
  # MACOS_SIGN_P12: ${{ secrets.MACOS_SIGN_P12 }}   # + 4 more, only if wiring notarization

Binary-name inference reads the repo's own .goreleaser.y*ml (builds[0].binary, falling back to project_name) at the released tag, falling back further to the repo name (stripping a -cli suffix, or the org name for a repo literally named cli) only if that config can't be read. Verified against every current consumer's real .goreleaser.yaml (specscore-cli, ingitdb-cli, datatug-cli, filetug, codegrapher, chatwright/cli, sneat-dev/wb, synchestra-vm, synchestra-servers). One known limitation: a multi-binary repo (synchestra-servers, which builds two binaries) only smoke-tests the first builds[] entry by default — set artifact_smoke_test_binary explicitly there.

For the maintainer: retagging

This repo is consumed via strongo/cicd/.github/workflows/release.yml@v1, a moving tag advanced deliberately (see README, "Releasing this repo"). After merging, existing @v1 callers get none of this until v1 is advanced:

git fetch --tags origin
git tag -f v1 "$(git tag -l 'v1.*.*' | sort -V | tail -1)"
git push -f origin v1

Once advanced, every @v1 consumer's next release run gets layer 1 (hard-fail once tested) and layers 2/3 (warn-only once tested) automatically, with no action required on their part.

Not merging

Per instructions: no merge, no tag. Review and merge (or hold) at your discretion.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_016oLc6NHcbs832y44pANFyB