Production Debug

Debug a live prod incident safely: read-only first, hypothesis second, never fix in prod.

Why

A prod incident is high-stress and high-stakes: users are losing money/data/trust every minute, and the temptation to 'just try a fix' is overwhelming. But a fix deployed in panic often makes it worse — you don't understand the cause, the fix has a side effect you didn't test, and now you have two incidents. The method is: first, read-only investigation (logs, metrics, traces, DB queries with SELECT only) to form a hypothesis; second, reproduce in staging; third, fix and verify in staging; fourth, deploy. The single most valuable skill is knowing when to roll back instead of forward — if you can't form a hypothesis in 15 minutes, roll back to the last known-good and investigate from a stable state. Production is not your IDE.

When

During a prod incident (user-facing outage or error spike). Takes as long as needed — do not skip the method under pressure.

How

1. Acknowledge and assign — start a timer, page the on-call, open an incident channel
   - Post "investigating" in the status page within 5 min; users tolerate outages, not silence

2. READ-ONLY investigation — form a hypothesis before touching anything
   - Check the dashboards: when did the error rate spike? What deploy is on that boundary?
   - Read the logs: filter by error level + the time window; look for a common request shape
   - Query the DB with SELECT only — never UPDATE/DELETE in prod during an incident
   - Form a hypothesis: "the deploy at 14:32 changed the auth check, breaking logins for SSO users"

3. Decide: roll back or roll forward?
   - If you can't form a hypothesis in 15 min → roll back to the last known-good deploy
   - If the hypothesis is clear and the fix is small → roll forward (after staging)

4. Reproduce in staging and fix there
   - Deploy the same code to staging; trigger the same request; confirm the bug
   - Write the fix; verify the bug is gone and no regression; get a review

5. Deploy the fix and watch the metrics for 15 min
   - Don't declare resolved when the deploy finishes; declare resolved when the error rate returns to baseline
   - Write a postmortem within 48h: timeline, cause, fix, action items — this is how incidents stop recurring

Example

# 1. Acknowledge (within 5 min)
# - open #incident-2024-03-15, page on-call, post status page update

# 2. READ-ONLY investigation
# Error rate spiked at 14:33 — what deployed at 14:32?
git log --since="2024-03-15 14:30" --until="2024-03-15 14:35" --oneline
# abc1234 feat: tighten SSO token validation

# Read the logs (filter by the spike window)
aws logs filter-log-events --log-group prod --start-time 1710517980 --filter-pattern '"level":"error"'
# {"level":"error","msg":"SSO token rejected","reason":"missing aud claim","userId":42}

# Hypothesis: the new validation rejects tokens without an 'aud' claim;
#             older SSO tokens (issued before aud was added) are being rejected.

# 3. Roll back (fix is not obvious, 15 min elapsed)
vercel rollback prod-deploy-abc1234   # back to known-good, users unblocked

# 4. Reproduce in staging + fix
# - deploy abc1234 to staging, log in with an old token → confirmed
# - fix: accept tokens without aud as a migration (warn, don't reject)
# - PR + review + staging verify

# 5. Deploy the forward fix and watch
vercel deploy --prod
# watch the error dashboard for 15 min → returns to baseline → declare resolved

Common Problems

⚠Deploying a 'quick fix' to prod without staging reproduction — the fix breaks something else and now you have two incidents.
⚠Running UPDATE/DELETE in prod during investigation 'to clean up bad data' — you destroy the evidence and can't find the cause.
⚠Spending 45 minutes trying to roll forward when a 30-second rollback would have unblocked users.
⚠No postmortem, so the same incident happens again 3 months later because nobody wrote down the cause.

Best Practices

✓Make rollback a one-command, well-documented operation — if it takes 20 minutes to figure out, you won't do it under pressure.
✓Keep a read-only dashboard bookmarked (error rate, p99 latency, deploy markers) so investigation starts in 10 seconds, not 10 minutes of finding the right view.
✓Require a postmortem within 48h of every incident, with a blameless timeline and concrete action items tracked to closure.
✓Practice the incident method in game-days (scheduled fire drills) — under real stress, you execute what you've rehearsed, not what you've read.