Alert Fatigue
Reduce alert volume so each page is meaningful — pruning, aggregation, and noise budgets.
Why
When an engineer gets 40 pages a shift, they stop reading them carefully — they ack, glance, and assume it's another false alarm. This is the exact psychological state in which a real SEV1 gets ignored for 30 minutes, because it's buried in a stream of low-signal pages that trained the engineer to treat all alerts as noise. Alert fatigue is not an annoyance; it is a direct cause of extended outages. The fix is ruthless pruning: every alert that fires without user-visible action is either deleted or converted to a ticket, and the target is <5 pages per on-call shift. Teams that track "pages per shift" and prune to that target catch real outages fast because each page is taken seriously.
When
When pages per on-call shift exceeds 5, or when an engineer admits to acking without reading. Review monthly.
How
1. Measure the baseline - "Pages per shift" over the last 30 days, per team and per service - "Action rate" = pages that led to a real action / total pages - If action rate <70%, you have fatigue 2. Classify every firing alert from the last 30 days - ACTED: led to investigation/fix → keep - NOISE: fired but was expected (deploy, known flaky) → silence or add 'for' window - REDUNDANT: duplicate of another alert → aggregate or delete - NO-RUNBOOK: nobody knows what to do → write runbook or delete 3. Apply the fix per class - NOISE: add a longer 'for' window, or silence during deploy windows - REDUNDANT: aggregate (e.g., "5 services down" → 1 alert, not 5) - NO-RUNBOOK: delete if you can't write a runbook in 30 minutes - Target: <5 pages/shift, >90% action rate 4. Add a noise budget - Each service gets a monthly "alert budget" (e.g., 10 pages/month) - Exceeding it triggers a mandatory alert review with the team - This makes fatigue a measured, owned problem, not a vibe 5. Make pruning a recurring ritual - Monthly alert review in the team retro: "what fired, what did we do, what do we delete" - Delete is the default; keeping an alert requires justification
Example
# Alert audit script — classify last 30 days of pages
# Output: per-alert action rate, candidates for deletion
import json
pages = json.load(open('pagerduty-export-30d.json'))
by_alert = {}
for p in pages:
by_alert.setdefault(p['alert_name'], []).append(p)
print(f"{'alert':40} {'count':>5} {'acted':>5} {'rate':>5}")
for name, ps in sorted(by_alert.items(), key=lambda x: -len(x[1])):
acted = sum(1 for p in ps if p.get('action_taken'))
rate = acted / len(ps)
flag = "DELETE?" if rate < 0.5 else ""
print(f"{name:40} {len(ps):>5} {acted:>5} {rate:>5.0%} {flag}")
# Sample output:
# HighCPU 42 8 19% DELETE?
# CheckoutSLOBurnRateFast 3 3 100%
# DiskSpaceLow 18 2 11% DELETE? (auto-resolves)
# DBConnectionFailed 7 7 100%
# Action: delete HighCPU + DiskSpaceLow → pages/shift drops from 12 to 4Common Problems
- ⚠Treating alert pruning as a one-time cleanup instead of a monthly ritual — alerts accrete with every new feature, so fatigue returns within a quarter.
- ⚠Keeping alerts because 'we might need them someday' — an alert that never fires is free, but one that fires without action is expensive; delete ruthlessly.
- ⚠No measurement of pages-per-shift or action rate — without numbers, fatigue is a vibe, and teams can't justify the pruning work to management.
- ⚠Aggregating nothing — 5 services going down fires 5 pages; one aggregated alert with a list of victims is one page and far less fatiguing.
Best Practices
- ✓Track pages-per-shift and action rate as KPIs; target <5 pages/shift and >90% action rate, and review monthly in the team retro.
- ✓Make deletion the default and keeping the exception — an alert stays only if someone justifies it with a runbook and a user-impact story.
- ✓Aggregate correlated alerts (5 services down → 1 alert with a victim list) to collapse noise without losing signal.
- ✓Give each service a monthly alert budget; exceeding it forces a review, making fatigue a measured, owned problem rather than a vibe.