Log Alerting

Alert on log patterns — specific errors, error rates, and absence-of-expected-logs — not just metrics.

Why

Metrics tell you the error rate is 0.5%, but logs tell you why — and some failures don't move the metrics at all. A 'DB connection refused' error that happens once per minute doesn't breach the 1% error-rate SLO, but it's a leading indicator that the DB pool is exhausted and a SEV1 is 10 minutes away. Alerting on the log pattern (the specific error message) catches this early, when you can fix it (raise the pool size) before it becomes an outage. Log alerts also catch the 'absence of expected logs' failure mode: if the cron job that sends invoices hasn't logged 'invoices sent' in 24 hours, something is wrong, even though no error was logged. This is a failure mode that metrics can't detect (there's no metric for 'cron didn't run'), but logs can (the expected log didn't appear). Together, metric + log alerts cover the full failure space.

When

When you have structured logs and aggregation. Add log alerts alongside metric alerts for the same services.

How

1. Alert on specific error messages (known failure modes)
   - "DB connection refused" → page (DB pool exhausted, SEV1 imminent)
   - "OOMKilled" → page (pod OOM, will restart and lose in-flight requests)
   - These catch failures that don't move the error-rate metric

2. Alert on error rate by service (from logs, not just metrics)
   - rate({service="checkout", level="ERROR"}[5m]) > 1/s → page
   - Log-based error rate catches errors that the metric pipeline misses (e.g., logged but not metric'd)

3. Alert on absence of expected logs (dead-man's switch)
   - "invoices sent" log should appear every 24h → alert if absent for 25h
   - "health check OK" log should appear every 1m → alert if absent for 5m
   - This catches silent failures (cron didn't run, worker died without erroring)

4. Use multi-window alerts to avoid flapping
   - Short window (1m) for fast detection, long window (5m) for auto-resolve
   - Alert fires if error rate >1/s for 1m AND >0.5/s for 5m

5. Route log alerts to the right responder
   - "DB connection refused" → DB team on-call, not app on-call
   - "OOMKilled" → platform team (resource issue), not app team
   - The alert annotation specifies the responder; don't default everything to one team

Example

# Log alerts (Loki + Alertmanager):
groups:
  - name: checkout-log-alerts
    rules:
      # 1. Specific error message (known failure mode)
      - alert: CheckoutDbConnectionRefused
        expr: count_over_time({service="checkout", msg=~"DB connection refused"}[5m]) > 0
        for: 1m
        labels: { severity: page, team: db-platform }
        annotations:
          runbook_url: "https://rb/checkout-db-failover"

      # 2. Error rate by service (from logs)
      - alert: CheckoutErrorRateHigh
        expr: rate({service="checkout", level="ERROR"}[5m]) > 1
        for: 2m
        labels: { severity: page, team: payments }

      # 3. Absence of expected logs (dead-man's switch)
      - alert: InvoiceCronNotRunning
        expr: count_over_time({service="billing", msg="invoices sent"}[25h]) == 0
        labels: { severity: page, team: billing }
        annotations:
          summary: "Invoice cron hasn't logged 'invoices sent' in 25h — likely dead"

      # 4. Multi-window (fast detect, slow resolve)
      - alert: CheckoutErrorMultiWindow
        expr: |
          rate({service="checkout", level="ERROR"}[1m]) > 2
          and
          rate({service="checkout", level="ERROR"}[5m]) > 0.5
        labels: { severity: page, team: payments }

# Coverage:
# - Metric alerts: error rate, latency, SLO burn (the "what")
# - Log alerts: specific errors, dead-man's, log-based rate (the "why" and the silent failures)
# - Together: full failure space covered

Common Problems

⚠Only metric alerts, no log alerts — a 'DB connection refused' that happens once/min doesn't breach the SLO but is a leading indicator; log alerts catch it early.
⚠No dead-man's switch — the cron silently stops and nobody notices for days because no error was logged; alert on the absence of the expected 'invoices sent' log.
⚠All log alerts route to one team — 'DB connection refused' pages the app team who can't fix the DB; route by alert annotation to the right responder.
⚠Single-window alerts — flap on/off every 30s as the error rate oscillates; use multi-window (1m detect, 5m resolve) for stable alerts.

Best Practices

✓Alert on specific error messages (DB connection refused, OOMKilled) that are known failure modes but don't move the error-rate metric.
✓Alert on absence of expected logs (dead-man's switch) to catch silent failures — cron didn't run, worker died without erroring.
✓Route log alerts to the right team via annotations (DB error → DB team, OOM → platform team); don't default everything to one team.
✓Use multi-window alerts (1m detect, 5m resolve) to avoid flapping; combine metric and log alerts to cover the full failure space.