ci: Add Percy rejection when any shards fail - #5821
Conversation
A dead shard contributes no snapshots, but finalize still seals the build. On main the project auto-approves without review, so that partial set becomes the baseline every later branch is compared against, and the snapshots the dead shard would have produced resurface as diffs on unrelated PRs. Capture the build id at finalize and reject the build when host-test did not succeed, so an incomplete set cannot become a baseline. percy build:reject authenticates with BrowserStack account credentials rather than the project token — @percy/cli-build's fetchCredentials reads BROWSERSTACK_USERNAME and BROWSERSTACK_ACCESS_KEY. Until those secrets are configured the reject step skips itself and a warning step records that an incomplete build was left to auto-approve, rather than failing the job. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2395de8c07
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if: >- | ||
| needs.host-test.result != 'success' | ||
| && steps.finalize.outputs.build_id != '' | ||
| && env.BROWSERSTACK_USERNAME != '' |
There was a problem hiding this comment.
When BROWSERSTACK_USERNAME is configured but BROWSERSTACK_ACCESS_KEY is missing or misnamed, this condition still runs percy build:reject without the required credential. The command then fails the finalize job instead of following the documented behavior of skipping rejection and emitting the configuration warning; require both environment variables before running the reject step, and treat either missing value as unrejected.
Useful? React with 👍 / 👎.
percy build:reject exits non-zero when either the username or the access key is missing, so gating only on the username let a half-configured secret pair run the command and fail the finalize job — on a run that was already red from the shard failure that triggered it. Gate on both, and make the warning step the exact complement of the reject condition so an incomplete build cannot pass through both silently. The warning now reports which of the three preconditions was missing, since a misnamed secret and an unparsable build id produce the same outcome from the outside. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…l-baselines # Conflicts: # .github/workflows/ci-host.yaml
Host Test Results 1 files 1 suites 1h 58m 12s ⏱️ Results for commit 15f7542. |
2 participants
We have a lot of snapshots coming in and out because builds are auto-accepted on
mainbut shards are failing wholesale and their snapshots are dropped. With this, if any shards fail, the build is rejected instead.