Real-Time Monitoring

Why

When

How

1. List the metrics where minutes matter: error rate, p95/p99 latency, checkout conversion rate, payment success rate, API availability —these need real-time monitoring; LTV, retention, NPS do not (they change over weeks)
2. For each metric, define the baseline: the normal value and its natural fluctuation —error rate is 0.1% with fluctuation to 0.2%, so the alert threshold must be above 0.2% to avoid noise
3. Choose the window: 1-minute rolling for fast detection (error rate, availability), 5-minute for stability (conversion rate, which is noisy at 1-minute granularity due to low counts)
4. Set the threshold with both absolute and relative conditions: error rate > 1% AND > 3x baseline —the absolute condition catches high-severity events, the relative condition catches anomalous changes from a low baseline
5. Define the alert routing: P1 (revenue-impacting, page on-call immediately), P2 (degraded, Slack #on-call), P3 (warning, email digest) —the severity determines the channel and the response time
6. Add a canary check on every deploy: for 5 minutes after deploy, compare the error rate and conversion rate to the pre-deploy baseline; if degraded, auto-rollback —this catches deploy-caused incidents in 5 minutes
7. Set the alert cooldown: after an alert fires, suppress duplicates for 15 minutes —without this, a 30-minute incident generates 30 pages and the on-call engineer cannot think
8. Include runbook links in every alert: the alert should link to the runbook (what to check, how to roll back, who to escalate to) so the on-call engineer can respond in 2 minutes, not 20
9. Track alert quality: every week, review which alerts fired, which were true positives, which were false, and tune the thresholds —alert fatigue is the enemy, and a 90% false-positive rate means the team ignores all alerts
10. Do NOT monitor slow-changing metrics in real-time: LTV, retention, NPS change over weeks; monitoring them in real-time wastes compute and produces alerts that fire at 2am for a 0.01% change that could wait until morning

Example

# Real-time monitoring: checkout and payments
Metrics monitored (minutes matter):
  - checkout conversion rate (5-min window)
  - payment success rate (1-min window)
  - API p95 latency (1-min window)
  - error rate 5xx (1-min window)

Baselines (last 30 days, excluding incidents):
  checkout conversion: 3.2% (fluctuation 2.8%-3.6%)
  payment success:     99.7% (fluctuation 99.4%-99.9%)
  API p95 latency:     180ms (fluctuation 150-220ms)
  error rate 5xx:      0.08% (fluctuation 0.05%-0.15%)

Alert thresholds (absolute AND relative):
  checkout conversion: < 2.5% AND < 80% of baseline -> P1 (page)
  payment success:     < 98%   AND < 99% of baseline -> P1 (page)
  API p95 latency:     > 500ms AND > 2x baseline      -> P2 (Slack)
  error rate 5xx:      > 1%    AND > 5x baseline      -> P1 (page)

Canary check (on every deploy):
  - 5-minute post-deploy comparison
  - if checkout conversion < 2.5% or error rate > 1%: auto-rollback
  - rollback time: 90 seconds (terraform apply + health check)

Alert routing:
  P1: PagerDuty -> on-call engineer (5-min response SLA)
      + Slack #incidents (all engineers notified)
  P2: Slack #on-call (30-min response)
  P3: Email digest (next morning)

Runbook (linked in every alert):
  1. Check recent deploys (last 30 min) -> if suspicious, roll back
  2. Check error logs -> identify the failing endpoint
  3. Check downstream services (Stripe, DB) -> status page
  4. If Stripe: check https://status.stripe.com
  5. If DB: check connection pool, slow query log
  6. Escalate to @payments-eng if payment-specific
  7. Post incident update in #incidents every 15 min

Alert quality (Sep 2026):
  Total alerts: 47
  True positive: 39 (83%)  <- target > 80%
  False positive: 8 (17%)   <- tune thresholds to reduce
  Mean time to detect: 3.2 min (target < 5 min)
  Mean time to resolve: 18 min (target < 30 min)

Common Problems

• ⚠Monitoring everything in real-time and producing alert fatigue —200 alerts per week of which 90% are false means the team ignores all alerts, including the real incident; monitor only metrics where minutes matter.
• ⚠Setting thresholds without a baseline —error rate fluctuates to 0.15% normally, and alerting at > 0.1% produces constant noise; the threshold must be above the natural fluctuation to be a signal.
• ⚠Alerting on slow-changing metrics in real-time —LTV and retention change over weeks; monitoring them in real-time wastes compute and fires 2am alerts for 0.01% changes that could wait until morning.
• ⚠No canary check on deploys —a deploy breaks checkout at 2pm and the team detects it at 6pm via a customer report; a 5-minute canary check would have auto-rolled back and saved 4 hours of lost conversions.
• ⚠Alerts without runbook links —the on-call engineer receives a page, has no idea what to check, and spends 20 minutes reading code; a runbook link in the alert cuts response time to 2 minutes.

Best Practices

• ✓Monitor only metrics where minutes matter (error rate, latency, checkout conversion, payment success) in real-time —slow-changing metrics (LTV, retention, NPS) should be reviewed weekly; monitoring everything in real-time produces alert fatigue and the team ignores real incidents.
• ✓Set alert thresholds using both absolute and relative conditions (error rate > 1% AND > 3x baseline) —the absolute condition catches high-severity events, the relative catches anomalous changes from a low baseline, and both are needed to avoid false positives.
• ✓Add a canary check on every deploy that compares post-deploy metrics to the baseline and auto-rolls back if degraded —this catches deploy-caused incidents in 5 minutes instead of 4 hours, and the auto-rollback removes the human decision delay.
• ✓Include a runbook link in every alert so the on-call engineer can respond in 2 minutes —the runbook specifies what to check, how to roll back, and who to escalate to, and without it the engineer spends 20 minutes reading code.
• ✓Track alert quality weekly (true-positive rate, false-positive rate, mean time to detect) and tune thresholds —a > 20% false-positive rate means the team is ignoring alerts, and the thresholds need tightening until the signal-to-noise ratio is > 80%.