Guardrail Metric

Metrics that must not degrade during an experiment —the 'do no harm' checks that prevent a winning feature from shipping if it breaks something else.

Why

A feature that lifts the primary metric by 10% and crashes the checkout for 2% of users is not a win —it is a revenue loss disguised as a metric win. Guardrail metrics are the defense against this: they are the metrics that the experiment must not harm, checked alongside the primary metric, with their own thresholds and stop conditions. Without guardrails, the team optimizes the primary metric at the expense of everything else —page load time creeps up, error rate rises, a secondary funnel breaks —and each 'win' ships a small regression that compounds. The guardrails force the team to state, in advance, what the feature must not break, and to verify it before shipping. This is the difference between 'we increased clicks by 10%' and 'we increased clicks by 10% without harming revenue, load time, or error rate' —the second statement is a ship decision; the first is a number.

When

Defined during test design, monitored throughout the test, checked at the decision point. Pre-defined for the entire testing program (the 'standard guardrails').

How

1. Define the standard guardrails that apply to every test on your surface:
   - page load time (P95 < baseline + 100ms)
   - error rate (< baseline + 0.5pp)
   - core conversion (must not decrease > 1% relative)
   - revenue/ARPU (must not decrease > 1% relative)
2. Add experiment-specific guardrails for features that touch sensitive areas:
   - a checkout change -> payment error rate, refund rate
   - a search change -> zero-result rate, search latency
   - an onboarding change -> time-to-first-action, support ticket rate
3. Set a threshold for each guardrail: the maximum acceptable degradation. "Error rate must not increase by more than 0.5pp" is a threshold; "error rate must not increase" is not (it forbids any noise)
4. Set a stop condition: if a guardrail breaches by > 2x the threshold, stop the test immediately (do not wait for the full duration) —this limits user harm
5. Monitor guardrails daily during the test, not just at the end —a breach on day 2 should stop the test on day 2, not on day 7 when the test ends
6. At decision time, check all guardrails: ship only if primary metric wins AND no guardrail breaches its threshold. A guardrail breach veto the ship even if the primary metric is highly significant
7. Document the guardrail check in the experiment result: "Primary: +10% (p<0.01). Guardrails: load time +12ms (within 100ms), error rate +0.1pp (within 0.5pp), revenue -0.3% (within 1%). Decision: SHIP."
8. Review the standard guardrail set quarterly —as the product grows, new sensitive areas emerge (e.g., a new payment method needs its own error-rate guardrail)

Example

# Guardrails for experiment #447 (CTA color)
Standard guardrails (apply to all checkout tests):
  - page_load_p95:     baseline 1.2s  threshold +100ms  stop +200ms
  - error_rate:        baseline 0.8%  threshold +0.5pp  stop +1.0pp
  - checkout_conversion: baseline 64%   threshold -1% rel  stop -2% rel
  - revenue_per_user:  baseline $4.20 threshold -1% rel  stop -2% rel

Experiment-specific guardrails:
  - (none; CTA color does not touch payment or search)

Daily monitoring (day 2 of 4):
  page_load_p95:     1.21s  (+10ms)   OK (within +100ms)
  error_rate:        0.79%  (-0.1pp)  OK
  checkout_conversion: 64.3% (+0.3pp)  OK
  revenue_per_user:  $4.22  (+0.5%)   OK
  -> continue test

Day 3 alert:
  page_load_p95:     1.45s  (+250ms)  BREACH (stop threshold +200ms)
  -> STOP TEST, investigate (likely a CSS regression in BlueCta)
  -> do NOT wait for day 4, do NOT ship even if CTR is up

Decision (after fix and re-run):
  Primary CTR: +6.2% (p=0.003)  -> WIN
  Guardrails:
    page_load_p95:     +8ms    (within +100ms)  OK
    error_rate:        +0.0pp  (within +0.5pp)  OK
    checkout_conversion: +0.1pp (within -1% rel) OK
    revenue_per_user:  +0.2%   (within -1% rel) OK
  -> SHIP (primary wins, all guardrails within threshold)

Common Problems

⚠No guardrails at all —the team ships a 10% CTR win that increases page load time by 400ms, and overall revenue drops because users abandon the slow page; the 'win' was a loss, but there was no guardrail to catch it.
⚠Guardrail thresholds set to zero ('must not increase at all') —any noise triggers a stop, the test never completes, and the team disables guardrails to get results, which is worse than having them.
⚠Checking guardrails only at the end of the test —a guardrail breaches on day 2 and is left running until day 7, harming users for 5 days because nobody looked; guardrails must be monitored daily.
⚠Guardrail breach that does not veto the ship —the team sees the primary metric is highly significant and ships despite a guardrail breach, rationalizing 'the guardrail is probably noise'; this defeats the entire purpose of guardrails.

Best Practices

✓Define a standard guardrail set for each surface (checkout, search, onboarding) that applies to every test —this saves design time and ensures no test forgets to check load time or error rate.
✓Set thresholds that allow noise but catch real harm —'+100ms' allows measurement noise; '+0ms' forbids it. The threshold should be the smallest degradation that would be a real problem, not zero.
✓Monitor guardrails daily with automated stop conditions —if a guardrail breaches 2x its threshold, stop the test automatically; this limits user harm to hours, not days, and it removes the temptation to 'wait and see'.
✓Make guardrail breach a hard veto on shipping —if any guardrail breaches, the feature does not ship, period; the team can re-design and re-test, but they cannot ship a known harm, and this rule must be enforced by the experiment platform, not by human willpower.