feat(infra): add blue-green and canary release strategy - #475
Open
Mai-Fura wants to merge 1 commit into
Open
Conversation
Introduce zero-downtime releases with a provider-neutral ReleaseRoute contract, blue-green and canary coordinators with health gating and automated rollback, a release pipeline wired to the existing CI, and a runbook documenting the release and rollback process. Closes AetherEdu#412 🤖 Generated with Codebuff Co-Authored-By: Codebuff <noreply@codebuff.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #412
Implements zero-downtime releases for AetherMint: a provider-neutral
ReleaseRoutetraffic-split contract, blue-green and canary coordinator scripts with health gating and automatic rollback, a release pipeline (.github/workflows/deploy.yml) wired to the existing CI, and a runbook documenting the release and rollback process. The key design decision is to keep the platform-neutral pattern established byscripts/failover-region.sh— coordinators are dry-run by default, require--execute+DEPLOY_APPROVED=true, and take injected platform commands — so the same tooling works with any Kubernetes distribution or edge router.Why
Before this change, production releases were an implicit, manual process: the base Kubernetes deployments referenced
latestimage tags and the README said to "replace the image tags ... as part of the release promotion process" with no pipeline, no health gating, and no defined rollback. There was no.github/workflows/deploy.yml, no traffic-split configuration, and no release runbook. This change makes the release process explicit and safe: every release is deployed to an isolated standby color, health-gated before promotion, automatically reverted on failed health checks, and fully documented so operators can ship and roll back with confidence.What was built
infra/release/release-route.yamlReleaseRoutecontract (aethermint.io/v1alpha1): active/standby colors, image repos, per-color versions, canary weights/hold, health-gate thresholds, and the auto-rollback policy. Mirrors the existinginfra/multi-region/edge-routing.yamlcontract style.infra/kubernetes/release/backend-blue.yaml,backend-green.yaml,frontend-blue.yaml,frontend-green.yaml,kustomization.yamllatest), and usesRecreateso a color never mixes versions. Renders viakubectl kustomize(4 Deployments + 4 Services).scripts/deploy-blue-green.sh--execute+DEPLOY_APPROVED=true.scripts/deploy-canary.sh5 25 50 100) with a hold window and health gate per step. On any failure it automatically drops the weight to 0%. Validates weights are numeric, non-decreasing, and end at 100.scripts/rollback-release.shscripts/update-release-state.shrelease-route.yamlsoactiveColorand per-color versions reflect reality after a release or rollback, so the contract never drifts.scripts/test-release.sh.github/workflows/deploy.ymlci-gate(refuses to deploy commits with failing/pending check runs), image build + push to GHCR tagged with the git SHA, release-tooling validation, and the deploy job (guarded toAetherEdu/AetherMint) running the coordinators with injected commands from secrets. Auto blue-green on push tomain; manualworkflow_dispatchfor strategy/version/canary weights. Updates the ReleaseRoute state after each successful promotion.docs/infrastructure/release-runbook.md.github/workflows/ci-pr.yml(modified)Validate Release Toolingjob (bash syntax +scripts/test-release.sh) gated by a newreleasepaths filter, and includes it in theci-statusgate.infra/kubernetes/README.md(modified).env.example(modified)RELEASE_CONFIGandDEPLOY_APPROVEDplus the injected-command env vars.The tests (
scripts/test-release.sh) exercise the same code paths the pipeline runs, including the failure paths: a simulated unhealthy standby triggers the automatic rollback and re-verification of the restored color, and an unhealthy canary step triggers a drop to 0%.Integration changes outside
scripts/.github/workflows/ci-pr.yml— newreleasepaths filter andvalidate-release-toolingjob wired into the existingci-statusgate, so release tooling regressions fail PRs..github/workflows/deploy.yml— new release pipeline (push tomain+workflow_dispatch), withconcurrencyso only one deploy per environment runs at a time..env.example— documentedRELEASE_CONFIG/DEPLOY_APPROVEDand the injected-command variables.infra/kubernetes/README.md— new "Blue-green release mode" section.No existing functionality was removed; the change is additive.
Acceptance criteria coverage
scripts/deploy-blue-green.sh— atomic--switchafter gates;scripts/rollback-release.sh— revert to the standby color;infra/kubernetes/release/— warm standby color kept running)scripts/deploy-canary.sh— progressive--weightsshifts with per-step--verifyand--hold;infra/release/release-route.yaml—canary.weights/holdSeconds/healthCheck)scripts/deploy-blue-green.sh— auto switch-back on any gate failure;scripts/deploy-canary.sh— auto drop to 0%; both verified by the approval-gate tests and the manual failure-path runs below).github/workflows/deploy.yml—ci-gatejob checks the commit's check runs;validate-release-toolingjob;ci-pr.ymlgains thevalidate-release-toolingjob in theci-statusgate)docs/infrastructure/release-runbook.md)Test plan
bash scripts/test-release.sh— 25/25 passing (25 new tests: plan output, config parsing, argument validation, approval gate, simulated full flow)bash -n scripts/deploy-blue-green.sh scripts/deploy-canary.sh scripts/rollback-release.sh scripts/update-release-state.sh scripts/test-release.sh— no syntax errorskubectl kustomize infra/kubernetes/release— renders 4 Deployments + 4 Servicespython3 yaml.safe_load_all) ofrelease-route.yaml, the 5 release manifests, and both workflows — validci-pr.ymlEnv vars / Notes
The pipeline requires these repository secrets before a deploy can run:
DEPLOY_CMD,DEPLOY_VERIFY_CMD,DEPLOY_SWITCH_CMD,DEPLOY_ROLLBACK_CMD(optional),DEPLOY_SHIFT_CMD(canary). Deployment ordering: images are built and pushed to GHCR before the deploy job runs; the deploy job is restricted toAetherEdu/AetherMintso fork runs can never touch production. Thedeployjob targets a GitHubenvironment(defaultproduction) where required-reviewer protection rules can be attached. After a successful promotion the pipeline commits the updatedrelease-route.yamlback tomain; that path is excluded from the workflow's push trigger so it cannot loop. Intentionally untouched: the multi-region failover model ininfra/multi-region/(this release strategy composes with it — release moves traffic between colors, failover between regions).