Escalation Policy

Define who gets paged, in what order, and when to escalate if the first responder doesn't acknowledge.

Why

An alert that pages someone but has no escalation path is a gamble — if that person is asleep, driving, or on a bad network, the outage sits unattended until they notice. An escalation policy turns a single point of failure into a chain: page the primary, if no ack in 5 minutes page the secondary, if no ack in 15 minutes page the manager. This guarantees that every SEV1 reaches a conscious human within a bounded time, which is the whole point of on-call. Without escalation, your MTTA (mean time to acknowledge) is unbounded, and a 5-minute outage becomes a 45-minute outage because the primary was in the shower.

When

When you set up your first on-call rotation. Revisit whenever a team's size or geography changes.

How

1. Define severity → response time → escalation chain
   - SEV1 (revenue down): ack <5m, resolve target <1h; primary → secondary → manager → director
   - SEV2 (degraded): ack <30m; primary → secondary
   - SEV3 (minor): next business day; primary only, no escalation

2. Set ack and resolve timeouts per severity
   - SEV1: page primary; if no ack in 5m → page secondary; if no ack in 15m → page manager
   - Each escalation includes the original alert + who was already paged (no "who's on?" confusion)

3. Use a paging system (PagerDuty/Opsgenie) not Slack/DM
   - Paging systems handle schedules, ack, escalation, and follow-the-sun automatically
   - Slack messages get buried; a page overrides silent mode and repeats until acked

4. Include a "stakeholder notification" channel for SEV1
   - Comms lead gets paged in parallel to do customer comms (see comms-protocol)
   - Manager is notified so they can unblock the responder if needed

5. Test the escalation path quarterly
   - Trigger a test SEV1 in staging; confirm it reaches the right people in the right order
   - A broken escalation path discovered during a real SEV1 is a double outage

Example

# PagerDuty escalation policy (terraform)
resource "pagerduty_escalation_policy" "payments_sev1" {
  name = "payments-sev1"
  num_loops = 2  # re-page primary twice before giving up
  rule {
    escalation_delay_in_minutes = 5
    target { type = "user_reference"; id = pagerduty_user.payments_primary.id }
  }
  rule {
    escalation_delay_in_minutes = 10  # 5 + 10 = 15m to secondary
    target { type = "user_reference"; id = pagerduty_user.payments_secondary.id }
  }
  rule {
    escalation_delay_in_minutes = 15  # 15 + 15 = 30m to manager
    target { type = "user_reference"; id = pagerduty_user.payments_manager.id }
  }
}
# SEV1 alert → service → this policy → primary paged immediately
# If primary doesn't ack in 5m → secondary paged
# If secondary doesn't ack in 10m → manager paged
# Stakeholder channel: #incidents gets a parallel Slack notification for comms lead

Common Problems

⚠Paging via Slack or DM instead of a paging system — messages get buried under chat, don't override silent mode, and have no ack/escalation logic.
⚠Escalation chain ends at the primary with no secondary — if the primary is unreachable, the outage waits until they notice, unbounding MTTA.
⚠Escalating to a manager who can't fix the problem — the manager's role is to unblock, not to debug; the chain should reach a technical secondary first.
⚠Never testing the escalation path — a typo in a user ID or a deleted account silently breaks escalation, discovered only during a real SEV1.

Best Practices

✓Use a dedicated paging system (PagerDuty/Opsgenie) with ack timeouts and automatic escalation — never rely on Slack or DM for SEV1.
✓Make the escalation chain reach a technical secondary before a manager; the manager's job is to unblock the responder, not to debug the outage.
✓Include a parallel stakeholder notification for SEV1 so customer comms start immediately, not after the responder is done debugging.
✓Test the escalation path quarterly with a staged SEV1 — confirm each step reaches the right person in the configured time window.