feat(deploy): roll back to the previous release when the new one won't serve

[Musiker15]

Closes the last item from the 2026-08-17 outage.

Why the health check alone is not enough

#243 made a dead release visible instead of silent, which is most of the value. It does not make it short: by the time the check runs, pm2 reload has already swapped the process, so a red workflow and a 503 arrive together and stay together until a human reads the log.

What changes

The build no longer deletes the previous bundle, it moves it aside:

rm -rf apps/web/.next.prev
if [ -d apps/web/.next ]; then mv apps/web/.next apps/web/.next.prev; fi
pnpm run build

If the port never answers, the deploy restores the previous release: reset to the SHA recorded before the pull, reinstall against that lockfile, regenerate the Prisma client, move the bundle back, reload. The job still exits non-zero, and the annotation says which commit is actually live so nobody assumes the push landed.

On success the fallback is removed again and pm2 save runs. It moved behind the check on purpose, persisting the process list before knowing whether the release works only makes a bad state stickier.

What it deliberately does not do

Undo the migration. prisma migrate deploy has already run and migrations are forward-only. That is tolerable for additive changes, since older code ignores columns it does not know about, but a migration that drops or renames something still needs a person. The workflow says so at the point where it matters.

Verification

The failure path is precisely the one that must not be discovered in production, so it was run against stubbed git, pnpm, pm2 and curl in a sandbox:

case	result
healthy release	new bundle stays, .next.prev removed, pm2 save, exit 0
release dead, previous bundle present	old bundle restored, "the pushed commit is NOT live", exit 1
release dead, rollback also dead	says so instead of claiming success, exit 1

bash -n on the extracted script and a YAML parse of the workflow both pass.

[graphify-labs]

Graphify reviewed this change.

Worth a look — the grounded gate found no coupling regressions or blocking issues, but 1 advisory finding(s) below merit a look before merge.

---

Graphify review — findings

Adds automatic rollback to the deploy.yml SSH script: it captures the pre-deploy SHA, preserves the old .next bundle as .next.prev, and on a failed health check restores the previous commit and bundle, reinstalls, and reloads PM2. Refactors the port health check into a reusable serving() function and defers pm2 save until after the new release is confirmed serving. Note the script explicitly does not roll back Prisma migrations.

Worth a look

• Build failure leaves previous bundle moved out of place — .github/workflows/deploy.yml:54 · Escalate · medium
  • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review Execution auto-disposal is off for this run; enable it (with sandbox isolation) to have Graphify try to confirm or refute this automatically.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 4 functions depend on the 4 functions this change touches.

Health — grade A; no new coupling hotspots.

Verification — 4 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

• verification_scope: 4 function(s) in the blast radius were not formally verified this run