Skip to content
Back to Operations & SRE

Incident Commander

Designate an Incident Commander who coordinates, decides, and communicates — not the person debugging.

Why

Without an Incident Commander (IC), everyone in the channel is debugging and nobody is coordinating: the same log gets read by three people, the rollback decision waits 10 minutes because nobody owns it, and customer comms never go out because everyone assumes someone else is doing it. The IC's job is explicitly not to debug — it's to hold the big picture, assign roles, make the call on rollback vs. fix-forward, and ensure the outside world (customers, stakeholders) is informed. This single role cuts MTTR by 30-50% because decisions get made instead of deferred, and the debuggers can focus without context-switching to write the next customer update. Google, Amazon, and every mature ops org use the IC model because it works.

When

For every SEV1 and SEV2. The IC is designated at incident start, before any debugging begins.

How

1. Designate the IC in the first message of the incident channel
   - "Alice is IC, Bob is debugging, Carol is comms" — state it explicitly
   - The IC does not debug; they coordinate, decide, and communicate

2. IC's responsibilities
   - Hold the timeline: what we know, what we've tried, what's next
   - Make the rollback vs. fix-forward call (don't let it default to "keep debugging")
   - Assign roles: debugger, comms lead, scribe, stakeholder liaison
   - Decide when to escalate (more people, manager, vendor support)

3. Run the incident channel with structure
   - IC posts status updates every 10-15m: "current state, what we're trying, ETA"
   - Debugger posts findings as they happen (no siloed investigation)
   - Scribe records decisions and timeline for the postmortem

4. Declare the incident over explicitly
   - IC confirms: service recovered, SLO no longer burning, monitoring stable for 10m
   - Then hand off to postmortem scheduling; don't let the channel drift to "I think we're done"

5. Rotate the IC role across the team
   - Everyone should be trained to be IC; the IC muscle is different from the debugger muscle
   - Run game days so the first time someone is IC isn't during a real SEV1

Example

# Incident channel (#inc-2026-03-15-checkout)
14:07  Alert fires → on-call acks, opens #inc-2026-03-15-checkout
14:08  IC designated: "Alice is IC. Bob, you're debugging — start with the runbook.
       Carol, you're comms — draft the customer status page. I'll coordinate."
14:10  IC: "Status: checkout 100% error, started 14:05. Bob investigating deploy v2.4.1.
       Customer comms in 5m. Next update 14:25."
14:15  Bob: "Found it — v2.4.1 throws on null discount. 340 orders affected."
14:16  IC: "Decision: rollback. Bob, initiate rollback. Carol, post 'identified, rolling back'."
14:18  IC: "Stakeholders: paging @manager for visibility, not for action."
14:20  Bob: "Rollback complete, error rate 0%."
14:22  IC: "Monitoring for 10m to confirm stable before declaring resolved."
14:32  IC: "Incident resolved. Postmortem tomorrow 10am. Channel archived."
# IC never typed a debug command; they coordinated, decided, and communicated.

Common Problems

  • ⚠Everyone debugs, nobody coordinates — three people read the same log, the rollback decision waits 10m, and customer comms never go out.
  • ⚠IC is also the debugger — they can't hold the big picture while deep in a stack trace, so decisions defer and the incident drifts.
  • ⚠No explicit IC designation — the role is implicit and shifts as people join/leave, so nobody knows who's actually coordinating.
  • ⚠Incident never explicitly declared over — the channel drifts to silence, but nobody confirms recovery or schedules the postmortem.

Best Practices

  • ✓Designate the IC in the first incident-channel message, explicitly and by name; the IC coordinates and decides, they do not debug.
  • ✓IC posts status updates every 10-15m and makes the rollback/fix-forward call explicitly — don't let it default to 'keep debugging'.
  • ✓Train everyone to be IC via game days; the first time someone runs an incident should not be a real SEV1.
  • ✓Declare the incident over explicitly: confirm recovery, 10m of stable monitoring, then schedule the postmortem — don't let the channel drift.