From e37c68f36b44199ed6c5b63e29459bb9a2bd5e49 Mon Sep 17 00:00:00 2001 From: Musiker15 Date: Tue, 18 Aug 2026 16:36:12 +0200 Subject: [PATCH] feat(deploy): roll back to the previous release when the new one won't serve The health check added in #243 turned a silent outage into a loud one, but it still leaves the site down: by the time it runs, pm2 reload has already swapped the process. On 2026-08-17 that would have meant a red workflow and a 503 until someone read it. So keep the previous bundle instead of deleting it. The build moves apps/web/.next aside rather than removing it, and if the port never answers, the deploy puts the old release back: reset to the previous SHA, reinstall against that lockfile, regenerate the client, restore the bundle, reload. The job still exits non-zero, and the error says plainly which commit is live. What this deliberately does not undo is the migration. `prisma migrate deploy` has already run and migrations are forward-only. That is fine for additive changes, since older code ignores columns it does not know about, but a migration that drops or renames something still needs a human. The comment in the workflow says so. pm2 save moved behind the health check as well. Persisting the process list before knowing whether the release works only makes a bad state stickier. Verified by running the script against stubbed git/pnpm/pm2/curl in a sandbox, since the failure path is the one that must not be discovered in production: a healthy deploy keeps the new bundle and exits 0; an unhealthy one with a recoverable predecessor restores the old bundle, reports that the pushed commit is not live and exits 1; and when the rollback cannot come up either, it says that instead of claiming success. --- .github/workflows/deploy.yml | 66 ++++++++++++++++++++++++++++-------- 1 file changed, 51 insertions(+), 15 deletions(-) diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index 8b5d6de..c774317 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -36,6 +36,9 @@ jobs: echo "-> Pull latest" git fetch --all --prune + # Remember where we came from, so a failed release can be undone. + PREV_SHA="$(git rev-parse HEAD)" + echo "Previous release: ${PREV_SHA}" git reset --hard origin/main echo "-> Install dependencies (frozen)" @@ -48,35 +51,68 @@ jobs: pnpm prisma migrate deploy pnpm prisma generate - echo "-> Build (clean .next first to avoid stale standalone chunks)" - rm -rf apps/web/.next + echo "-> Build (keeping the previous bundle as a fallback)" + rm -rf apps/web/.next.prev + if [ -d apps/web/.next ]; then mv apps/web/.next apps/web/.next.prev; fi pnpm run build echo "-> Reload services via PM2 (zero-downtime)" pm2 reload ecosystem.config.cjs --update-env - echo "-> Save PM2 process list" - pm2 save - # PM2 reporting "online" only means the process exists, which is not # the same as it serving. On 2026-08-17 a bundle missing part of # @swc/helpers died on startup without ever binding the port: PM2 # showed online, this workflow reported success, and the site # returned 503 for 40 minutes. Ask the port, not the supervisor. + serving() { + code="" + for i in $(seq 1 20); do + code="$(curl -s -o /dev/null -w "%{http_code}" --max-time 5 http://127.0.0.1:3008/ || true)" + if [ "$code" = "200" ]; then echo "Serving after ${i} attempt(s)."; return 0; fi + sleep 3 + done + echo "Not serving on :3008 (last status: ${code:-none})" + return 1 + } + echo "-> Health check" - code="" - for i in $(seq 1 20); do - code=$(curl -s -o /dev/null -w "%{http_code}" --max-time 5 http://127.0.0.1:3008/ || true) - [ "$code" = "200" ] && { echo "Serving after ${i} attempt(s)."; break; } - sleep 3 - done - if [ "$code" != "200" ]; then - echo "::error::Web process is not serving on :3008 (last status: ${code:-none})" - pm2 logs msk-forms-web --lines 40 --nostream --err || true + if serving; then + rm -rf apps/web/.next.prev + echo "-> Save PM2 process list" + pm2 save + echo "Deploy finished" + exit 0 + fi + + # From here the new release is known bad. Put the previous one back + # rather than leaving the site down until someone reads the log. + # + # Note what this does NOT undo: `prisma migrate deploy` already ran + # and migrations are forward-only. That is tolerable because they are + # additive in practice (older code ignores columns it does not know), + # but a migration that drops or renames something still needs a human. + echo "::error::New release is not serving — rolling back to ${PREV_SHA}" + pm2 logs msk-forms-web --lines 40 --nostream --err || true + + if [ ! -d apps/web/.next.prev ]; then + echo "::error::No previous bundle kept (first deploy?). Leaving the failed release in place." + exit 1 + fi + + git reset --hard "${PREV_SHA}" + pnpm install --frozen-lockfile --prod=false + pnpm prisma generate + rm -rf apps/web/.next + mv apps/web/.next.prev apps/web/.next + pm2 reload ecosystem.config.cjs --update-env + + if serving; then + echo "::error::Rolled back to ${PREV_SHA}. The site is up on the previous release; the pushed commit is NOT live." exit 1 fi - echo "Deploy finished" + echo "::error::Rollback did not come up either — manual intervention required." + exit 1 - name: Notify Discord (optional) if: always()