Log Correlation

Correlate logs with traces and metrics using trace_id to see the full request path during incidents.

Why

A checkout failure involves 4 services: gateway, cart, checkout, payment — and the error is in payment, but the user's request started at gateway. Without correlation, the on-call sees a checkout ERROR log, queries payment, sees nothing (the payment call hasn't been logged yet), and concludes 'checkout is broken' — wrong. With trace_id correlation, the on-call sees the checkout ERROR, clicks the trace_id, and sees the full path: gateway → cart → checkout → payment (500) → DB (timeout). The root cause is the DB timeout, visible in 3 clicks. This is the three pillars of observability working together: metrics tell you the error rate is high, logs tell you the error message, traces tell you the full path and where the latency is. Teams that correlate all three report 70% faster root-cause identification because they see the whole request, not one service's view of it.

When

When you have structured logs, distributed tracing, and metrics. Wire the trace_id into all three.

How

1. Generate a trace_id at the edge and propagate it everywhere
   - Gateway generates trace_id on incoming request → puts in request context
   - All downstream calls (HTTP, gRPC, DB) include trace_id in headers/metadata
   - All logs include trace_id as a field; the trace span includes the log links

2. Link logs to traces and traces to logs
   - Each log entry has trace_id and span_id; the trace span links back to the logs
   - In the UI: click a log → see its trace; click a trace → see its logs
   - This is built into OpenTelemetry, Jaeger, Datadog, Honeycomb

3. Link metrics to traces via exemplars
   - A histogram bucket can include an exemplar: a trace_id for a request in that bucket
   - "p99 latency is 2s" → click the exemplar → see the trace of a 2s request → see where the time went
   - This turns a metric from a number into a debuggable instance

4. Build a correlation workflow for incidents
   - Alert fires (metric) → on-call opens the alert → sees the trace_id of a failing request
   - Click trace_id → see the full path, the failing service, the error log
   - All in 3 clicks, no grepping, no guessing which service to query next

5. Use correlation for non-incidents too (performance)
   - "p99 latency is 2s on checkout" → exemplar → trace → see the DB query taking 1.8s
   - Correlation isn't just for errors; it's for any question about a specific request

Example

# Correlation flow (incident):
1. Alert fires: CheckoutErrorRateHigh (metric: error rate 5%)
2. On-call opens alert → sees exemplar trace_id: "abc123"
3. Click trace_id → Jaeger shows the full trace:
   gateway (200, 50ms)
     └─ cart (200, 30ms)
         └─ checkout (500, 1200ms)
             └─ payment (500, 1180ms)
                 └─ db.query (timeout, 1150ms)  ← root cause
4. Click the db.query span → see the log:
   {trace_id:"abc123", service:"payment", msg:"DB query timeout",
    query:"SELECT * FROM charges WHERE...", latency_ms:11500}
5. Root cause in 4 clicks: DB query timeout on charges table.
   → Run the DB runbook → index the charges table → resolved.

# Exemplar in a metric (Prometheus):
histogram_quantile(0.99, checkout_latency_seconds_bucket)
#   = 2.0s
# With exemplar:
#   = 2.0s  (exemplar trace_id: "xyz789")
# Click "xyz789" → see the trace of a 2s request → see the DB query taking 1.8s
# The metric told you "p99 is slow"; the exemplar showed you why.

Common Problems

⚠Logs without trace_id — the on-call sees the checkout error but can't link to the payment log; they query each service separately and correlate by timestamp (slow, wrong).
⚠Traces without log links — the trace shows the path but not the error message; the on-call has to grep logs by timestamp to find the error.
⚠Metrics without exemplars — 'p99 is 2s' is a number; without an exemplar trace_id, the on-call can't see which request was slow or why.
⚠Correlation only for errors — the same correlation finds performance bottlenecks (slow DB query in a 2s request); use it for any per-request question.

Best Practices

✓Generate trace_id at the edge and propagate to all logs, traces, and downstream calls; OpenTelemetry does this automatically.
✓Link logs to traces (trace_id + span_id in every log) and metrics to traces (exemplars); click a metric → see a trace, click a trace → see logs.
✓Build the incident workflow around correlation: alert → exemplar → trace → failing span → error log → root cause, in 3-4 clicks.
✓Use correlation for performance, not just errors; 'p99 is 2s' + exemplar → trace → slow DB query is the same flow for any per-request question.