Executable Runbook

Make runbooks executable scripts, not just docs — the on-call runs the runbook, they don't read it.

Why

A runbook that says 'run this kubectl command, then this SQL query, then this curl' is three copy-paste steps, each of which can be pasted wrong, run against the wrong cluster, or fail silently. An executable runbook is a script that does all three with the right parameters, validates each step's output, and aborts if something unexpected happens. This cuts MTTR by the time the human spends typing and second-guessing, and it makes the runbook testable — you can run it in staging as part of the quarterly audit and know it works, instead of hoping the docs are right. The script is also self-documenting: the code is the runbook, so it can't rot independently of the docs. Teams that executable-ize their top 5 runbooks report 50% faster incident response because the on-call runs one command instead of reading and typing 10.

When

For the top 5 most-fired runbooks first. Don't executable-ize everything — focus on the runbooks that run most often.

How

1. Pick the top 5 runbooks by fire frequency
   - These are the ones where execution speed matters most
   - A runbook that fires once a year can stay as docs; one that fires weekly should be a script

2. Convert each step to a script function with validation
   - Each step: run command, check expected output, abort if unexpected
   - Use the same language as your ops tooling (bash, Python, Go)

3. Make the script idempotent and safe
   - Re-running it shouldn't break anything (in case the first run partially succeeded)
   - Destructive steps (delete, restart) require a --confirm flag
   - Dry-run mode by default: --apply to execute

4. Test the script in staging as part of the audit
   - Run with --dry-run → confirm it would do the right thing
   - Run with --apply → confirm it actually resolves the issue
   - This is the runbook test; it's now code, so test it like code

5. Wire the script to the alert (auto-trigger for safe steps)
   - Alert fires → script runs step 1-2 automatically (see runbook-trigger)
   - Human runs the full script if auto-trigger didn't resolve it
   - The script logs everything for the postmortem

Example

#!/usr/bin/env python3
# runbook/checkout-slo-burn.py — executable runbook
import subprocess, sys, click

@click.command()
@click.option('--apply', is_flag=True, help="Execute changes (default: dry-run)")
def main(apply):
    # Step 1: check for recent deploys
    r = subprocess.run(["kubectl", "rollout", "history", "deploy/checkout", "-n", "checkout"],
                       capture_output=True, text=True)
    if "v2.4.1" in r.stdout and "30m" in r.stdout:  # recent deploy
        print("Found recent deploy v2.4.1 — likely cause")
        if apply:
            subprocess.run(["kubectl", "rollout", "undo", "deploy/checkout", "-n", "checkout"], check=True)
            print("Rolled back. Verifying...")
            if error_rate() < 0.001:  # verification
                print("✅ Error rate <0.1% — resolved")
            else:
                print("❌ Error rate still high — escalate to @checkout-sme")
                sys.exit(1)
        else:
            print("[dry-run] would rollback deploy/checkout")
    else:
        print("No recent deploy — check downstream deps on servicemap")

if __name__ == "__main__":
    main()
# On-call runs: python3 runbook/checkout-slo-burn.py --apply
# Auto-trigger (alert webhook): runs with --apply for safe steps only

Common Problems

⚠Script isn't idempotent — re-running it after a partial success makes things worse; require idempotency and a dry-run mode.
⚠No dry-run mode — the on-call can't safely test the script before executing, so they either skip it or run it blind and hope.
⚠Script rots independently of the docs — the runbook.md says one thing, the .py does another; keep them as one (code is the runbook).
⚠Executable-izing everything — a runbook that fires once a year isn't worth scripting; focus on the top 5 by frequency.

Best Practices

✓Executable-ize the top 5 runbooks by fire frequency; a yearly runbook can stay as docs, a weekly one should be a script.
✓Make the script idempotent with a dry-run default and --apply to execute; destructive steps require --confirm.
✓Treat the script as the runbook (code is docs) so they can't rot independently; test it in staging as part of the quarterly audit.
✓Wire safe steps to auto-trigger from the alert; the human runs the full script only if auto-remediation didn't resolve it.