Runbook Trigger

Link every alert to a runbook and auto-trigger executable runbooks where the first step is deterministic.

Why

An alert without a runbook is a page that costs 15-20 minutes of "what does this mean?" before any investigation starts. Linking the alert to a runbook collapses that to 30 seconds of reading, and if the runbook's first steps are deterministic (restart the pod, failover the DB, clear the cache), you can auto-execute them and cut MTTR by the time the human even reads the page. The runbook trigger is the bridge between monitoring and action: it turns "CPU is high on the cache tier" from a vague symptom into "here's the one-click script that clears the bloated keys, and here's what to check if that doesn't work." Teams that wire runbook triggers report 40-60% MTTR reductions because the common incidents resolve themselves before the human finishes logging in.

When

After you have alerts and runbooks. Add auto-trigger when a runbook's first 1-2 steps are safe, deterministic, and reversible.

How

1. Ensure every alert links to a runbook
   - Alert annotation: runbook_url; the paging system surfaces it in the page
   - No runbook = no alert (enforce in code review of alert rules)

2. Structure the runbook for automation
   - Step 1-2: deterministic, safe, reversible (restart, clear cache, failover read replica)
   - Step 3+: requires judgment (investigate logs, decide rollback) — human-only
   - Mark which steps are auto-executable

3. Wire auto-trigger for the safe steps
   - Alert fires → runbook step 1-2 execute automatically (via webhook → job)
   - Human is paged in parallel; by the time they ack, step 1-2 may have resolved it
   - If auto-steps fail → escalate severity (the simple fix didn't work)

4. Log every auto-execution
   - What ran, what it changed, success/failure, correlation to the alert
   - This is your audit trail and your "did auto-remediation help?" dataset

5. Measure the impact
   - "Incidents resolved by auto-remediation" / total → target 30-50% for common alerts
   - "Auto-remediation made it worse" rate → must be <1%; if higher, demote to human-only

Example

# Alert rule with runbook + auto-trigger webhook
- alert: CacheHitRateLow
  expr: cache_misses / (cache_hits + cache_misses) > 0.5
  for: 10m
  labels: { severity: ticket, team: platform, runbook: "https://rb/cache-low-hit" }
  annotations:
    auto_remediate: "https://ops/api/runbook/cache-clear-bloated"

# ops/api/runbook/cache-clear-bloated (auto-executable, safe, reversible):
#   1. Identify top 10 keys by evictions (read-only, safe)
#   2. Check if they match known bloated-key patterns (safe)
#   3. If yes → DEL those 10 keys (reversible: they're a cache, they'll repopulate)
#   4. Log: which keys, before/after hit rate, correlation_id
#   5. If hit rate recovers → resolve alert; else → escalate to human
# Human runbook (step 3+): https://rb/cache-low-hit
#   - If auto-clear didn't help → check if a new deploy changed access patterns
#   - If yes → rollback (runbook: rollback-platform)

Common Problems

⚠Auto-remediation that isn't reversible — a script that deletes data or drops tables can't be undone; restrict auto-trigger to safe, reversible actions.
⚠Auto-triggering on SEV1 without a human in the loop — if the auto-fix is wrong, you've made the outage worse and faster; always page a human in parallel.
⚠No audit log of auto-executions — when an outage gets worse, you can't tell if the auto-remediation caused it; log every action with a correlation ID.
⚠Runbooks that are stale docs instead of executable scripts — the link points to a Confluence page that was last edited 2 years ago and no longer matches the system.

Best Practices

✓Enforce "no runbook = no alert" in code review so every alert has an actionable link; this alone cuts MTTR by 15-20 minutes per incident.
✓Only auto-trigger safe, reversible steps (restart, clear cache, failover read replica); never auto-trigger data-destructive actions.
✓Always page a human in parallel with auto-remediation; if the auto-fix fails or makes it worse, the human is already investigating.
✓Log every auto-execution with a correlation ID and measure "auto-remediation made it worse" rate; if >1%, demote that runbook to human-only.