Postmortem Process

Run a structured postmortem within 5 business days that produces a timeline, root cause, and tracked action items.

Why

An incident without a postmortem is a lesson unlearned — the same systemic cause (no deploy gate, no canary, no runbook) survives to produce the next outage, and the team's incident history is a list of unique events instead of a pattern. A structured postmortem within 5 business days (while memory is fresh and data is available) produces a data-driven timeline, a root cause found via 5-whys, and action items with owners and due dates that fix the system. The process matters because it's the mechanism by which an organization learns from its outages: without it, every incident is paid for in customer trust but the learning is wasted. With it, each incident makes the next one less likely, which is the only sustainable way to improve reliability.

When

After every SEV1 and SEV2, within 5 business days of resolution. SEV3 optional; SEV4 none.

How

1. Schedule within 24h of resolution, hold within 5 business days
   - Invite: IC, on-call, change author, SME, facilitator (blameless keeper)
   - 60-90 minutes; everyone comes having reviewed the timeline draft

2. Build the timeline from data before the meeting
   - Facilitator pulls alerts, deploys, chat logs, metric screenshots into a draft timeline
   - Meeting reviews and corrects the timeline; it's data-driven, not memory-driven

3. Find root cause with 5-whys, stopping at a systemic fix
   - "Why did checkout break?" → "null discount threw" → "no test for null" → "no gate requiring the test" → ...
   - Stop at a cause you can fix with a system change (add a test, add a gate), not "be more careful"

4. Write action items: owner, due date, systemic fix
   - Each item fixes the system, not the human; no "be more careful" items
   - Track in the team's issue tracker, not in the postmortem doc

5. Publish and track to completion
   - Postmortem is shared with the whole org (blameless, so no secrecy)
   - Action items are reviewed in sprint planning until all ship; the incident isn't closed until they do

Example

# postmortem-2026-03-15-checkout-down.md
**Severity:** SEV1  **Duration:** 12m  **Impact:** 340 orders lost (~$4.2k)
**Date:** 2026-03-15  **Author:** Alice (facilitator)

## Timeline (from data)
14:03  Deploy v2.4.1 shipped (canary skipped via "small change" override)
14:05  Error rate 0% → 100% (null-discount orders throw)
14:07  Alert CheckoutSLOBurnRateFast fired
14:08  On-call acked, war room opened, IC designated
14:15  Root cause identified (v2.4.1 null-discount bug)
14:16  Rollback decision made and initiated
14:20  Rollback complete, error rate 0%
14:32  Incident declared resolved

## Root cause (5 whys)
1. Checkout broke → v2.4.1 throws on null discount
2. Shipped → canary skipped
3. Canary skipped → "small change" override exists in deploy tool
4. Override exists → no policy preventing it for prod-critical paths
5. → **Fix: remove canary-skip override for /checkout; require SRE approval**

## Action items (systemic, owned, dated)
- [ ] Remove canary-skip for prod-critical paths — @deploy-platform, due 2026-03-22
- [ ] Add integration test: null-discount edge case — @payments, due 2026-03-29
- [ ] Add deploy gate: block if canary error rate >1% — @deploy-platform, due 2026-03-22

## What went well
- Alert fired in 2m, on-call acked in 2m, rollback in 4m — monitoring and runbook worked
## What went wrong
- Canary override allowed a prod-critical deploy to skip canary
## Where we got lucky
- The bug only affected null-discount orders (~5% of traffic); full impact would have been 100%

Common Problems

⚠Postmortem held weeks later — memory has decayed, data has aged out, and the timeline is reconstructed from recollection, which is wrong.
⚠Root cause is "human error" with no systemic fix — the same mistake recurs because only the human changed, not the system that allowed it.
⚠Action items live in the postmortem doc, not the issue tracker — they're invisible to sprint planning and never ship, so the systemic cause survives.
⚠Postmortem is private to the team — the org can't learn from it, and the same systemic gap exists in other teams who have the same tooling.

Best Practices

✓Schedule the postmortem within 24h and hold it within 5 business days; build the timeline from data (logs, alerts, deploys) before the meeting.
✓Use 5-whys to find a systemic root cause and stop at a fix that changes the system (add a test, add a gate), not the human ('be more careful').
✓Track action items in the issue tracker with owner and due date; review in sprint planning until all ship — the incident isn't closed until they do.
✓Publish the postmortem to the whole org (blameless, so no secrecy); cross-team learning is the multiplier that prevents the same gap elsewhere.