Skip to content

feat(infra): add blue-green and canary release strategy - #475

Open
Mai-Fura wants to merge 1 commit into
AetherEdu:mainfrom
Mai-Fura:feat/issue-412-blue-green-canary-release
Open

feat(infra): add blue-green and canary release strategy#475
Mai-Fura wants to merge 1 commit into
AetherEdu:mainfrom
Mai-Fura:feat/issue-412-blue-green-canary-release

Conversation

@Mai-Fura

Copy link
Copy Markdown

Summary

Closes #412

Implements zero-downtime releases for AetherMint: a provider-neutral ReleaseRoute traffic-split contract, blue-green and canary coordinator scripts with health gating and automatic rollback, a release pipeline (.github/workflows/deploy.yml) wired to the existing CI, and a runbook documenting the release and rollback process. The key design decision is to keep the platform-neutral pattern established by scripts/failover-region.sh — coordinators are dry-run by default, require --execute + DEPLOY_APPROVED=true, and take injected platform commands — so the same tooling works with any Kubernetes distribution or edge router.

Why

Before this change, production releases were an implicit, manual process: the base Kubernetes deployments referenced latest image tags and the README said to "replace the image tags ... as part of the release promotion process" with no pipeline, no health gating, and no defined rollback. There was no .github/workflows/deploy.yml, no traffic-split configuration, and no release runbook. This change makes the release process explicit and safe: every release is deployed to an isolated standby color, health-gated before promotion, automatically reverted on failed health checks, and fully documented so operators can ship and roll back with confidence.

What was built

File What it contains
infra/release/release-route.yaml Declarative ReleaseRoute contract (aethermint.io/v1alpha1): active/standby colors, image repos, per-color versions, canary weights/hold, health-gate thresholds, and the auto-rollback policy. Mirrors the existing infra/multi-region/edge-routing.yaml contract style.
infra/kubernetes/release/backend-blue.yaml, backend-green.yaml, frontend-blue.yaml, frontend-green.yaml, kustomization.yaml Blue/green Deployment + Service pairs. Each color is fully isolated (its own backend service; each frontend calls only its own color's backend), uses explicit image tags (never latest), and uses Recreate so a color never mixes versions. Renders via kubectl kustomize (4 Deployments + 4 Services).
scripts/deploy-blue-green.sh Blue-green coordinator: deploy to standby → health-gate → atomic switch → post-promotion gate. On any failure it automatically switches back to the previously active color and exits non-zero. Dry-run by default; requires --execute + DEPLOY_APPROVED=true.
scripts/deploy-canary.sh Canary coordinator: deploy to the canary color → progressive weight shifts (default 5 25 50 100) with a hold window and health gate per step. On any failure it automatically drops the weight to 0%. Validates weights are numeric, non-decreasing, and end at 100.
scripts/rollback-release.sh Operator-facing rollback for a completed release: reverts a blue-green promotion to the standby color or zeroes a canary.
scripts/update-release-state.sh Rewrites release-route.yaml so activeColor and per-color versions reflect reality after a release or rollback, so the contract never drifts.
scripts/test-release.sh Non-destructive dry-run suite: plan output, config parsing, argument validation, the approval gate, and a full simulated blue-green flow. 25/25 passing.
.github/workflows/deploy.yml Release pipeline: ci-gate (refuses to deploy commits with failing/pending check runs), image build + push to GHCR tagged with the git SHA, release-tooling validation, and the deploy job (guarded to AetherEdu/AetherMint) running the coordinators with injected commands from secrets. Auto blue-green on push to main; manual workflow_dispatch for strategy/version/canary weights. Updates the ReleaseRoute state after each successful promotion.
docs/infrastructure/release-runbook.md Release + rollback runbook: strategies, component map, prerequisites (secrets), reference injected Kubernetes commands, step-by-step blue-green and canary procedures, automated vs manual rollback, CI wiring, and the exercise.
.github/workflows/ci-pr.yml (modified) Adds a Validate Release Tooling job (bash syntax + scripts/test-release.sh) gated by a new release paths filter, and includes it in the ci-status gate.
infra/kubernetes/README.md (modified) Documents the blue-green release overlay and points to the runbook.
.env.example (modified) Documents RELEASE_CONFIG and DEPLOY_APPROVED plus the injected-command env vars.

The tests (scripts/test-release.sh) exercise the same code paths the pipeline runs, including the failure paths: a simulated unhealthy standby triggers the automatic rollback and re-verification of the restored color, and an unhealthy canary step triggers a drop to 0%.

Integration changes outside scripts/

  • .github/workflows/ci-pr.yml — new release paths filter and validate-release-tooling job wired into the existing ci-status gate, so release tooling regressions fail PRs.
  • .github/workflows/deploy.yml — new release pipeline (push to main + workflow_dispatch), with concurrency so only one deploy per environment runs at a time.
  • .env.example — documented RELEASE_CONFIG / DEPLOY_APPROVED and the injected-command variables.
  • infra/kubernetes/README.md — new "Blue-green release mode" section.

No existing functionality was removed; the change is additive.

Acceptance criteria coverage

  • Blue-green deployment with instant rollback (scripts/deploy-blue-green.sh — atomic --switch after gates; scripts/rollback-release.sh — revert to the standby color; infra/kubernetes/release/ — warm standby color kept running)
  • Canary releases with traffic shifting and health gating (scripts/deploy-canary.sh — progressive --weights shifts with per-step --verify and --hold; infra/release/release-route.yamlcanary.weights/holdSeconds/healthCheck)
  • Automated rollback on failed health checks (scripts/deploy-blue-green.sh — auto switch-back on any gate failure; scripts/deploy-canary.sh — auto drop to 0%; both verified by the approval-gate tests and the manual failure-path runs below)
  • Release pipeline wired to the existing CI (.github/workflows/deploy.ymlci-gate job checks the commit's check runs; validate-release-tooling job; ci-pr.yml gains the validate-release-tooling job in the ci-status gate)
  • Runbook documenting the release and rollback process (docs/infrastructure/release-runbook.md)

Test plan

  • bash scripts/test-release.sh25/25 passing (25 new tests: plan output, config parsing, argument validation, approval gate, simulated full flow)
  • bash -n scripts/deploy-blue-green.sh scripts/deploy-canary.sh scripts/rollback-release.sh scripts/update-release-state.sh scripts/test-release.sh — no syntax errors
  • kubectl kustomize infra/kubernetes/release — renders 4 Deployments + 4 Services
  • YAML parse (python3 yaml.safe_load_all) of release-route.yaml, the 5 release manifests, and both workflows — valid
  • Manual: simulated unhealthy standby (verify fails) → automatic rollback triggered, traffic reverted to the original color, exit 1
  • Manual: simulated unhealthy canary step → automatic rollback to 0% weight, exit 1
  • Not run: contracts/backend/frontend build+test jobs — this PR touches no application code (infra/scripts/docs/workflows only), so those CI jobs are unaffected; they run on the PR via ci-pr.yml
  • Manual (requires cluster + secrets): real deployment against a staging cluster with injected kubectl commands, as documented in the runbook

Env vars / Notes

RELEASE_CONFIG=infra/release/release-route.yaml
DEPLOY_APPROVED=false
# injected platform commands (secrets in CI):
DEPLOY_CMD=/usr/local/bin/deploy-color
DEPLOY_VERIFY_CMD=/usr/local/bin/verify-color
DEPLOY_SWITCH_CMD=/usr/local/bin/switch-active-color
DEPLOY_SHIFT_CMD=/usr/local/bin/set-canary-weight

The pipeline requires these repository secrets before a deploy can run: DEPLOY_CMD, DEPLOY_VERIFY_CMD, DEPLOY_SWITCH_CMD, DEPLOY_ROLLBACK_CMD (optional), DEPLOY_SHIFT_CMD (canary). Deployment ordering: images are built and pushed to GHCR before the deploy job runs; the deploy job is restricted to AetherEdu/AetherMint so fork runs can never touch production. The deploy job targets a GitHub environment (default production) where required-reviewer protection rules can be attached. After a successful promotion the pipeline commits the updated release-route.yaml back to main; that path is excluded from the workflow's push trigger so it cannot loop. Intentionally untouched: the multi-region failover model in infra/multi-region/ (this release strategy composes with it — release moves traffic between colors, failover between regions).

Introduce zero-downtime releases with a provider-neutral ReleaseRoute
contract, blue-green and canary coordinators with health gating and
automated rollback, a release pipeline wired to the existing CI, and a
runbook documenting the release and rollback process.

Closes AetherEdu#412

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Infrastructure] Blue-green and canary release strategy

1 participant