Skip to content
Back to Operations & SRE

Incident Severity

Declare severity within 5 minutes of detection to set response urgency and escalation.

Why

The first 5 minutes of an incident set the next 5 hours. If severity is undeclared, responders dither about whether to page the manager, whether to start customer comms, whether to rollback — and each dither adds 10 minutes to MTTR. A pre-defined severity table (see severity-mapping) turns this into a 30-second lookup: "checkout error rate >1% = SEV1 = page manager + start comms + rollback." The incident commander declares it once, and every downstream action (war room, comms, postmortem requirement) flows from that declaration. Teams that declare severity within 5 minutes have 3x faster MTTR than teams that "figure it out as we go," because the declaration aligns everyone immediately instead of through 20 minutes of chat.

When

Within 5 minutes of detecting any incident. Re-assess as the impact changes.

How

1. Detect and declare within 5m
   - On-call sees the alert → assesses impact against the severity table → declares
   - If unclear, declare the higher severity (you can downgrade later; don't under-react)

2. The declaration sets everything downstream
   - SEV1 → page IC + manager + comms lead; open war room; start customer comms
   - SEV2 → page IC; open incident channel; no war room unless it escalates
   - SEV3 → ticket; investigate during business hours

3. Re-assess at every impact change
   - Error rate drops from 5% to 0.1% → downgrade SEV1 to SEV2
   - Spreads to a second region → upgrade SEV2 to SEV1
   - State the change in the incident channel with a reason

4. Assign an Incident Commander (IC)
   - IC is not debugging — they're coordinating: severity, comms, escalation, timeline
   - The responder who declared it becomes IC until explicitly handed off

5. Close the incident with a severity-accurate follow-up
   - SEV1/SEV2 → postmortem required within 5 business days
   - SEV3 → postmortem optional
   - Severity drives the scrutiny, not the other way around

Example

# Incident declaration (in #incidents Slack):
14:07  @oncall: SEV1 declared — checkout error rate 100% (alert: CheckoutSLOBurnRateFast)
       Impact: all checkout attempts failing, ~$500/min revenue loss
       IC: @oncall  |  Comms: @commslead  |  War room: zoom-link
14:09  @oncall: rollback initiated (runbook: rollback-checkout)
14:12  @oncall: rollback complete, error rate 0%
14:15  @oncall: SEV1 downgraded to SEV2 — monitoring for recurrence for 30m
14:45  @oncall: SEV2 resolved — no recurrence. Postmortem scheduled 2026-03-16.

# Severity table (pre-defined, looked up in 30s):
#   checkout error >1% for 5m → SEV1 (revenue down)
#   checkout error 0.1-1%      → SEV2 (degraded)
#   search down                → SEV2 (browse works)
#   email delayed >30m         → SEV3 (no revenue impact)

Common Problems

  • ⚠Severity undeclared for 20+ minutes — responders debate in chat while the outage grows; declare the higher severity and downgrade later.
  • ⚠IC is also debugging — no one is coordinating comms, escalation, or timeline, so the response is chaotic and customer comms are forgotten.
  • ⚠Severity never re-assessed — an incident that dropped to 0.1% error rate keeps a war room open for 2 hours, wasting everyone's time.
  • ⚠No severity table, so declaration is a judgment call at 3am — different engineers declare different severities for the same incident.

Best Practices

  • ✓Declare severity within 5 minutes using a pre-defined table; when unclear, declare high and downgrade — under-reacting is the costly error.
  • ✓Assign a dedicated IC who coordinates (severity, comms, escalation) but does not debug — coordination and debugging compete for the same brain.
  • ✓Re-assess severity at every impact change and state it in the incident channel — an open war room for a resolved incident wastes responder time.
  • ✓Tie severity to follow-up: SEV1/SEV2 require a postmortem, SEV3 does not — the scrutiny matches the impact, not the responder's guilt.