Synthetic Monitoring

Run scripted transactions against production from external probes to catch outages before users do.

Why

Real-user monitoring tells you what happened; synthetic monitoring tells you what is happening right now, from a known vantage point, on a schedule you control. A synthetic check that logs in, adds an item to cart, and checks out every 60 seconds will catch a broken payment gateway in 60 seconds, whereas RUM might not surface it until enough real users hit it to move a p95. Synthetics are also the only way to monitor the critical paths that low traffic makes noisy in RUM — your 3am checkout flow might see 2 real users an hour, but a synthetic sees it every minute. They are the cheapest way to get an SLI for uptime that maps directly to "is the site up."

When

After a service reaches production. Prioritize the top 3-5 user journeys (login, search, checkout, signup). Add a check whenever a new critical path ships.

How

1. Identify the 3-5 critical user journeys
   - Login, search, add-to-cart + checkout, signup, key API read
   - These are the paths whose failure = "the site is down" to a user

2. Script each journey as a deterministic transaction
   - Use Playwright/Selenium; headless Chrome; fixed test account
   - Assert on the final step (e.g., "order confirmation page reached")
   - Keep runtime <30s so a 1-min cadence catches outages within 2 cycles

3. Run from multiple geographies and providers
   - 3+ probes (e.g., US-east, EU, Asia) so a single-region blip isn't a false alarm
   - Use a third-party (Pingdom/UptimeRobot/Datadog Synthetics) so the probe isn't in your own cloud

4. Alert on failure, not on slow
   - 2 consecutive failures from 2+ regions = page (avoids single-probe noise)
   - Latency degradation = a separate, lower-urgency ticket (don't page on slow)

5. Tag each check with the journey and owning team
   - The alert routes to the team that owns the journey, not a generic "ops" queue
   - Include a runbook link and the last working screenshot in the alert

Example

// synthetics/checkout.ts — Playwright synthetic check
import { test, expect } from '@playwright/test';
test('checkout journey', async ({ page }) => {
  await page.goto('https://app.example.com/login');
  await page.fill('#email', '[email protected]');
  await page.fill('#password', process.env.SYNTH_PASS!);
  await page.click('button[type=submit]');
  await expect(page).toHaveURL(/dashboard/);
  await page.click('text=Add to cart');
  await page.click('text=Checkout');
  await page.fill('#card', '4242 4242 4242 4242'); // test card
  await page.click('text=Pay');
  await expect(page).toHaveText(/Order confirmed/);
});
// Run every 60s from 3 regions via Datadog Synthetics:
//   cadence: 60s, locations: ["us1","eu1","ap1"], alert: 2 fails in 2 regions
// Alert payload includes: journey=checkout, team=payments, runbook=https://rb/checkout-down

Common Problems

⚠Running synthetics from inside your own VPC — a regional cloud outage takes your probe down too, so you miss the outage you're trying to detect.
⚠Paging on a single probe failure — one flaky browser or DNS hiccup pages the on-call at 3am; require 2+ regions failing before paging.
⚠Using a shared test account that gets rate-limited or locked out — the synthetic starts failing for test-hygiene reasons, masking real outages.
⚠Scripts that drift from the real UI (stale selectors) — the synthetic fails on a harmless button rename, creating alert fatigue and eroding trust.

Best Practices

✓Run probes from a third-party outside your cloud provider, in 3+ regions, so a single-region outage is detected, not silenced.
✓Version synthetic scripts in Git alongside the app and run them in CI on every deploy — a selector change that breaks the synthetic fails the deploy.
✓Alert only on 2+ consecutive failures from 2+ regions; route single-probe failures to a low-urgency cleanup queue to preserve on-call sanity.
✓Include a screenshot on failure in the alert payload — a picture of the 500 page saves the on-call engineer 5 minutes of reproduction.