Error Budget

Track the error budget as a shared scoreboard that governs deploy velocity and reliability investment.

Why

The error budget is the shared currency that ends the war between product (ship faster) and SRE (be more reliable). Product wants velocity, which consumes budget; SRE wants reliability, which preserves it. Without a budget, this is a political fight won by whoever has more political capital. With a budget, it's a scoreboard: if you're under budget, product ships freely; if you're over, deploys freeze and the team invests in reliability until the budget recovers. This converts a chronic organizational conflict into a self-governing system where the data decides, not the loudest person in the room. Teams that adopt error budgets report 2-3x deploy velocity because the safe headroom is now quantified and usable.

When

Immediately after defining an SLO. The budget is (1 - SLO) × window; track it continuously and surface it to the whole team.

How

1. Compute the budget from the SLO
   - SLO 99.9% over 28d → budget = 0.1% × 40320m = 40.3 minutes
   - This is the total downtime allowed in the window

2. Track budget consumption continuously
   - remaining = budget - downtime_so_far_this_window
   - burn_rate = (downtime_last_1h) / (budget / 28d / 24h)  # how fast you're consuming
   - Surface on a dashboard the whole team sees, updated every 5m

3. Define the policy that consumes the budget
   - burn < 0.5x → "green": deploy freely, allow risky changes, experiment
   - burn 0.5-1x → "yellow": normal deploys, no experiments
   - burn > 1x → "red": freeze feature deploys, only reliability fixes, page SRE

4. Make budget violations visible and automatic
   - When burn > 2x → auto-create a ticket "deploy freeze until burn < 1"
   - When budget exhausted → block the deploy pipeline (enforce, don't just warn)

5. Reset and learn at window end
   - At 28d: did you finish under budget? → loosen SLO or ship more next window
   - Over budget? → tighten process, invest in reliability, keep the freeze longer

Example

# Error budget dashboard (Prometheus + Grafana)
# SLO: 99.9% over 28d → budget = 40.3m
budget_total_minutes = 40.3
downtime_this_window = sum_over_time( (1 - sli) [28d] ) * 28d_in_minutes
remaining = budget_total_minutes - downtime_this_window
burn_rate = (1 - sli_over_1h) / (1 - 0.999)  # >1 = consuming faster than allowed

# Grafana stat panels:
#   "Budget remaining: 12.4m / 40.3m"  (color: green >50%, yellow 10-50%, red <10%)
#   "Burn rate (1h): 2.3x"  (color: red if >2)
#   "Burn rate (6h): 0.8x"  (color: yellow if >1)

# Deploy gate (CI check):
if burn_rate_1h > 2:
    fail("error budget burning >2x — deploy frozen until burn < 1x")
# This makes the budget a hard gate, not a suggestion that product can override

Common Problems

⚠Budget exists but isn't enforced — deploys continue even when over budget, so the number is decoration and the product/SRE war continues.
⚠Only SRE sees the budget — product teams don't know they're consuming it, so they can't self-govern; surface it on a dashboard everyone sees.
⚠Budget window too short (1 day) — noise dominates and the team is constantly flipping between freeze and ship; 28 days smooths this.
⚠No policy for what happens when budget is burned — the number is tracked but nothing changes, so it's just a metric, not a governance tool.

Best Practices

✓Enforce the budget as a CI deploy gate, not a dashboard suggestion — when burn >2x, the pipeline blocks; this makes the policy real.
✓Surface the budget on a dashboard the whole company sees, not just SRE — product teams self-govern when they can see their consumption.
✓Use a 28-day window to smooth noise; shorter windows flip between freeze and ship constantly, longer windows delay action.
✓Define explicit policies per burn-rate band (green/yellow/red) so the response is automatic, not a meeting every time the budget moves.