Rollback Playbook

Make rollback a one-command, well-rehearsed action so deploys are reversible in under 5 minutes.

Why

The fastest incident response is a rollback, but only if rollback is a 30-second command you've practiced. If rollback is a multi-step manual process that nobody has run in 3 months, the on-call at 3am will spend 20 minutes reading a wiki, miss a step, and make the outage worse. Every deploy must be reversible by a single command (kubectl rollout undo, vercel --prod rollback, your platform's equivalent), and that command must be tested in staging monthly so the first time someone runs it in anger is not during a SEV1. Teams with one-command rollback report 5-minute MTTR for deploy-caused outages; teams without it report 45-minute MTTR because rollback is a research project under pressure.

When

Every deploy must be reversible. Test rollback monthly in staging; run it in prod at the first sign of a deploy-caused incident.

How

1. Make rollback a single command
   - k8s: kubectl rollout undo deployment/checkout
   - Vercel: vercel --prod rollback
   - Custom: a script that points the LB at the previous image/tag
   - No multi-step wiki; one command, tested

2. Ensure the previous version is still available
   - Keep the last N (≥2) images/tags deployed; don't garbage-collect immediately
   - DB migrations must be backward-compatible (expand-contract pattern) so rollback doesn't lose data

3. Test rollback in staging monthly
   - Deploy vN → deploy vN+1 → rollback to vN → verify vN serves traffic
   - This is a 10-minute drill that keeps the rollback path oiled
   - Track "time to rollback" as a metric; target <2m

4. Decide rollback vs. forward-fix at the start of an incident
   - Deploy-caused + no forward fix ready → rollback immediately
   - Config-caused + forward fix is 5m away → forward-fix
   - Don't debate for 20m; the runbook has the decision tree

5. Make rollback safe for stateful changes
   - DB: use expand-contract migrations (add column, deploy, backfill, remove old — each step reversible)
   - Queue messages: old version must tolerate new message format (or you can't roll back the producer)

Example

# One-command rollback (k8s):
kubectl rollout undo deployment/checkout -n prod
# Verify:
kubectl rollout status deployment/checkout -n prod  # waits for rollout
curl -s https://app.example.com/health  # confirm 200

# Deploy pipeline keeps last 5 images (don't GC immediately):
#   gcr.io/app/checkout:v2.4.1  (current)
#   gcr.io/app/checkout:v2.4.0  (previous — rollback target)
#   gcr.io/app/checkout:v2.3.9  ...
#   gcr.io/app/checkout:v2.3.8  ...
#   gcr.io/app/checkout:v2.3.7  ...

# DB migration (expand-contract, reversible):
#   1. ADD COLUMN discount_code TEXT NULL  (old code ignores it)  ← deploy v2.4.1
#   2. Backfill: UPDATE orders SET discount_code = ... WHERE ...
#   3. App uses discount_code                                  ← deploy v2.4.2
#   4. (later) DROP old column                                 ← deploy v2.5
#   Rollback from step 3 → step 2: app ignores new column, safe
#   Rollback from step 4 → step 3: old column gone, NOT safe → don't roll back past step 3

# Monthly drill (staging):
#   deploy v2.4.3-staging → deploy v2.4.4-staging → rollback → verify v2.4.3 serves
#   Metric: time-to-rollback = 45s ✓ (target <2m)

Common Problems

⚠Rollback is a multi-step wiki, not a command — the on-call spends 20 minutes reading and misses a step, making the outage worse.
⚠Previous image garbage-collected immediately after deploy — rollback target is gone, so the only option is a forward-fix under pressure.
⚠DB migration isn't backward-compatible — rollback would lose data, so it's not an option, and a deploy-caused outage becomes a 2-hour forward-fix.
⚠Rollback never tested — the first time it runs is during a SEV1, and a typo or stale config makes it fail, doubling the outage.

Best Practices

✓Make rollback a single tested command (kubectl rollout undo / vercel --prod rollback); no multi-step wikis — the on-call can't read under pressure.
✓Keep the last ≥2 images/tags deployed; don't garbage-collect immediately, or rollback has no target.
✓Use expand-contract DB migrations so every step is reversible; a non-reversible migration makes rollback impossible and turns a 5m outage into 2h.
✓Drill rollback monthly in staging and track time-to-rollback as a metric; the first run in anger must not be during a SEV1.