Skip to content
Back to Operations & SRE

War Room

Convene a focused war room for SEV1 with the right people, a clear scope, and a time limit.

Why

A SEV1 with revenue down needs faster decisions than an async Slack thread can produce — five people typing in parallel generate noise, not progress, and the rollback decision waits 15 minutes for everyone to weigh in. A war room (video call or dedicated channel with IC + debugger + comms + subject-matter expert) forces synchronous decisions: the SME says 'rollback is safe', the IC calls it, the rollback starts in 30 seconds instead of 15 minutes. The trap is the war room that grows to 20 people and becomes a spectator sport — everyone watches, nobody decides, and the noise slows the actual responders. The fix is a strict invite list and a scribe who keeps the signal-to-noise high.

When

For SEV1 incidents that need cross-team coordination or fast decisions. End the war room when the incident is resolved.

How

1. Open the war room with a strict invite list
   - Required: IC, debugger, comms lead, SME for the affected system
   - Optional (notify, don't invite): manager, stakeholders — they get status updates, not a seat
   - Cap at ~5 people; beyond that, noise dominates signal

2. Start with a structured kickoff
   - IC states: current impact, what we know, what we're trying, who's doing what
   - Scribe starts the timeline; every decision and finding is recorded

3. Run it synchronous and timeboxed
   - Video call for SEV1 (decisions in seconds, not minutes); dedicated channel for SEV2
   - IC posts status every 10-15m; if nothing has changed, that's a signal to try something new
   - No silent observers — if you're in the room, you have a role

4. Keep the signal-to-noise high
   - Scribe summarizes: "Decision: rollback. Reason: v2.4.1 broke null-discount. Owner: Bob."
   - Debugger posts findings as they happen; IC doesn't let investigation go silent for >10m
   - Spectators get status updates, not a seat in the room

5. Close the war room explicitly
   - IC declares resolved, confirms 10m stable monitoring, schedules postmortem
   - Don't let it drift open — an open war room drains the team for hours after the incident

Example

# War room (Zoom #inc-2026-03-15, SEV1 checkout down)
14:08  IC opens: "SEV1, checkout 100% error since 14:05, ~$350/min revenue impact.
       Bob debugging, Carol on comms, Dave is SME for checkout. Scribe: Alice."
14:10  Bob: "v2.4.1 deployed at 14:03, error rate jumped 14:05. Correlation strong."
14:12  Dave (SME): "v2.4.1 added null-discount handling. If a cart has no discount,
       the new code throws. Rollback is safe — no DB migrations in this deploy."
14:13  IC: "Decision: rollback. Bob, initiate. Carol, post 'identified, rolling back'."
14:15  Bob: "Rollback initiated, v2.4.0 deploying."
14:20  Bob: "v2.4.0 live, error rate 0%."
14:22  IC: "Monitoring 10m. Carol, prepare 'resolving' update for 14:38."
14:32  IC: "Stable. War room closed. Postmortem tomorrow 10am. Great work, team."
# 5 people, 25 minutes, one clear decision path. No spectators, no noise.

Common Problems

  • ⚠War room grows to 20+ people — spectators add noise, slow decisions, and the actual responders can't find the signal in the chat scroll.
  • ⚠No scribe — decisions and findings aren't recorded, so the postmortem reconstructs from memory and the timeline is wrong.
  • ⚠Async Slack thread instead of synchronous call for SEV1 — decisions take 15 minutes instead of 30 seconds because nobody is present at the same time.
  • ⚠War room stays open for hours after resolution — drains the team and normalizes 'war room' as a state instead of a timeboxed tool.

Best Practices

  • ✓Cap the war room at ~5 people (IC, debugger, comms, SME, scribe); notify stakeholders, don't invite them — spectators add noise.
  • ✓Use a synchronous video call for SEV1 so decisions happen in seconds; reserve async channels for SEV2.
  • ✓Assign a scribe to record every decision and finding in real time; the timeline is the postmortem's spine.
  • ✓Close the war room explicitly when the incident resolves; don't let it drift open and drain the team for hours.