🧪 test(ci): assert the changelog-sync guard verdict mapping - #324
Merged
Conversation
The release guard added in a5fdd3d is correct, but nothing tested the inline `run:` body that turns its exit code into a job outcome. Flipping `2) verdict=hold` to `2) verdict=push` -- which would push the exact fbb5196 demotion of `## [0.17.0]` to main -- left the whole suite green, as did replacing the fatal `*) exit 1` arm with a push verdict. Section 6 only proved the guard runs before the push and that the push reads its output, never which verdict each exit code produces. Extract the guard step's script from changelog-sync.yaml and execute it against fixture repositories whose HEAD is untagged and whose CHANGELOG.md is dirty, exactly as `task changelog:write` leaves it. Assert the job-level outcome in both directions: a release-in-flight demotion emits a verdict that is not the one the push step gates on (green, warned, not pushed), a demotion whose tag is visible and a dropped middle section fail the step, and a legitimate [Unreleased] append or body re-render still pushes. The push condition must be a single `== '<literal>'` comparison against the guard step's own id, so the verdict comparison is an evaluation rather than a hard-coded guess.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
hack/test/changelog_sync_release_guard_test.shassertions that pin the verdict mapping of the changelog-sync release guard. Test-only: 1 file, +196 lines, zero Go, zero production change..github/workflows/changelog-sync.yamlis byte-identical tomain.Why the original story was stale
The REL-CHANGELOG-01 story asked for a demotion guard on the changelog-sync push step. That guard already landed at
a5fdd3dc8on 2026-08-07 — sixteen days before the story was filed. Implementing the story as written would have been a no-op re-implementation.What was actually broken: the gate's test suite was vacuous
The landed guard was covered by tests, but those tests never asserted which verdict each guard exit code maps to. Concretely: flipping the workflow's
2) verdict=holdarm toverdict=push— i.e. making the guard push a CHANGELOG that demotes a release section while its tag is in flight, the exact scenario the guard exists to prevent — previously left all 22 checks green.So the guard was landed but unpinned: any future refactor of the case arms could silently invert its behaviour with a fully green suite.
This lane closes that vacuity. The suite now asserts, per exit code, the verdict emitted, whether the push step runs, whether the job stays green or fails, and the annotation surfaced:
verdict=push, push runs, job green (self-healing preserved)verdict=hold, push skipped, job green,::warningnaming the pending tag (no alarm fatigue on every release)::errorannotationIt also asserts the structural invariants the mapping depends on: the guard step runs before the push step, and the push step gates on a single verdict literal.
Verification
An independent reviewer (not the author of this code) ran 12 of its own production mutations against the workflow and reproduced 7 mutants that
main's suite missed and this branch now catches. Verdict: APPROVE, no P0, no P1.Local gates on the rebased head:
bash hack/test/changelog_sync_release_guard_test.sh— rc=0, 31ok -, 0 failuresbash hack/lint-shell.sh— rc=0go test ./test/ci/— oktask changelog:verify— okRisk
None to production behaviour: no workflow, chart, manifest, or Go file is touched. The only effect is that a future regression in the guard's verdict mapping now turns CI red instead of green.