Runbook Structure

Structure runbooks with a fixed template: symptom, diagnosis, steps, verification, escalation.

Why

A runbook that's a wall of prose is read by nobody at 3am — the on-call engineer scans for 30 seconds and if they can't find 'what do I do', they wing it. A fixed template (symptom → diagnosis → steps → verification → escalation) makes the runbook scannable: the symptom confirms 'yes this is my alert', the steps are numbered and copy-pasteable, the verification tells them when to stop, and the escalation tells them when to give up. This structure also makes runbooks writable in 20 minutes instead of 2 hours, because the author fills in a form rather than composing an essay. Teams that adopt a template report 80% of runbooks written within a week of the alert, vs. 20% without — the structure removes the writing friction.

When

Before or immediately after creating any alert. Every alert must link to a runbook with this structure.

How

1. Use a fixed template for every runbook
   ## <Alert Name>
   **Symptom:** what the alert means in user terms
   **Diagnosis:** how to confirm this is the right runbook (queries, dashboard links)
   **Steps:** numbered, copy-pasteable commands, each with expected output
   **Verification:** how to confirm the issue is resolved
   **Escalation:** who to page if the steps don't work

2. Write the symptom in user terms, not metric terms
   - "checkout error rate >1%" → "users can't complete purchases"
   - The on-call needs to know the user impact, not just the metric

3. Make steps numbered and copy-pasteable
   - "1. Run: kubectl get pods -n checkout | grep Crash"
   - "2. If pods are crashing, run: kubectl logs <pod> --tail=50"
   - Each step has a command and the expected output — no prose

4. Include verification
   - "Confirm: error rate <0.1% for 5m on the checkout dashboard"
   - This tells the on-call when to stop and declare resolved

5. Link the runbook from the alert
   - Alert annotation: runbook_url; the paging system surfaces it in the page
   - The runbook lives at a stable URL (versioned in git, not a Confluence page that rots)

Example

# runbook/checkout-slo-burn.md
## CheckoutSLOBurnRateFast
**Symptom:** Users are failing to complete purchases; checkout error rate >1%.
**Severity:** SEV1 (revenue-impacting) — page immediately.

**Diagnosis:**
1. Open the checkout dashboard: https://grafana/d/checkout
2. Confirm error rate >1% in the last 5m (if <1%, alert is stale — close it)
3. Check error breakdown by route: which endpoint is failing?

**Steps:**
1. Check for recent deploys:
   ```bash
   kubectl rollout history deploy/checkout -n checkout | head -5
   ```
   If a deploy in the last 30m → likely the cause. Go to step 2.
2. Rollback the last deploy:
   ```bash
   kubectl rollout undo deploy/checkout -n checkout
   ```
   Expected: "deployment rolled back"
3. If no recent deploy → check downstream deps on the service map:
   https://grafana/d/servicemap?focus=checkout
   If a dependency is down → page that team's on-call.

**Verification:**
- Confirm error rate <0.1% for 5m on the checkout dashboard
- Confirm SLO burn rate <1x

**Escalation:**
- If rollback fails → page payments-platform secondary on-call
- If deps are healthy and error persists → page @checkout-sme

Common Problems

⚠Wall of prose instead of numbered steps — the on-call can't scan it at 3am and wings the response, which is slower and riskier.
⚠No verification step — the on-call runs the fix but doesn't know when to stop, so the incident stays 'investigating' for 20 extra minutes.
⚠Runbook in Confluence instead of git — it rots silently, the link breaks, and nobody notices until a real incident finds the 404.
⚠Symptom in metric terms ('error rate >1%') not user terms ('users can't buy') — the on-call doesn't grasp the urgency and under-responds.

Best Practices

✓Use a fixed template (symptom, diagnosis, steps, verification, escalation) for every runbook — scannable at 3am and writable in 20 minutes.
✓Write the symptom in user terms ('users can't buy') not metric terms; the on-call needs the impact, not the query.
✓Make steps numbered with copy-pasteable commands and expected output; no prose — the on-call executes, they don't read essays.
✓Store runbooks in git at a stable URL, not Confluence; link from the alert annotation so the paging system surfaces it in the page.