Log Sampling

Sample high-volume non-error logs to control cost while keeping 100% of errors and warnings.

Why

A service handling 10,000 requests/s that logs every request at INFO produces 600,000 logs/min — that's 30B logs/day, which costs $30k/month in storage and makes queries slow. But you don't need every INFO log; you need all ERRORs (100%, no sampling — an error is rare and important), all WARNs (100%, they're signals of degradation), and a sample of INFOs (1-10%, enough to see traffic patterns and debug specific requests via trace_id). This is tail-based sampling: sample by level, not randomly. The cost drops 90% (you keep 10% of INFOs) while keeping 100% of the logs that matter for incidents. The key rule: never sample errors. If you sample ERRORs at 1%, you'll miss 99% of the errors and the on-call will see a clean dashboard while the service is failing.

When

When log volume exceeds budget or query latency. Start with 100% retention; sample only when cost or speed forces it.

How

1. Never sample ERROR or WARN logs
   - These are rare and important; sampling them defeats the purpose of logging
   - Configure: level=ERROR → keep 100%; level=WARN → keep 100%

2. Sample INFO logs by rate or by trace_id
   - Rate: keep 10% of INFO logs (random sampling)
   - Trace-based: keep 100% of traces that have an ERROR, sample the rest (tail-based)
   - Tail-based is better — it keeps the full trace for failed requests

3. Use probabilistic sampling with a consistent hash
   - Hash the trace_id; keep if hash % 100 < sample_rate
   - This ensures all logs for a kept trace_id are kept (no partial traces)

4. Adjust the sample rate by service
   - High-volume service (checkout): 1% INFO
   - Low-volume service (admin): 100% INFO
   - The admin service's logs are cheap to keep and useful for debugging

5. Monitor the sampling rate and adjust
   - Track "logs kept / logs emitted" per service
   - If a service's error rate rises, temporarily raise its INFO sample rate to 100% for diagnosis

Example

# Sampling config (Loki/Fluentd):
sampling:
  - match: { level: "ERROR" }
    keep: 1.0          # 100% — never sample errors
  - match: { level: "WARN" }
    keep: 1.0          # 100% — never sample warnings
  - match: { level: "INFO", service: "checkout" }
    keep: 0.01         # 1% — high-volume service
    strategy: "trace_hash"  # consistent by trace_id
  - match: { level: "INFO", service: "admin" }
    keep: 1.0          # 100% — low-volume, cheap to keep

# Tail-based sampling (OpenTelemetry Collector):
processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }  # keep 100% of error traces
      - name: sample-rest
        type: probabilistic
        probabilistic: { sampling_percentage: 1 }  # 1% of the rest

# Result:
# - All ERROR/WARN logs kept (100%) → incidents are fully visible
# - 1% of checkout INFO logs kept → cost drops 90%, traffic patterns still visible
# - 100% of admin INFO logs kept → cheap, useful for debugging
# - Any trace with an ERROR → 100% of that trace's logs kept (tail-based)

Common Problems

⚠Sampling errors — a 1% error sample means 99% of errors are invisible; the on-call sees a clean dashboard while the service fails.
⚠Random sampling without trace awareness — half a trace's logs are kept, so the on-call sees the start but not the failure, and can't reconstruct the request.
⚠100% retention of everything — costs $30k/month and queries are slow; sample the high-volume INFOs and keep 100% of what matters.
⚠Same sample rate for all services — the high-volume checkout service drowns the aggregator while the low-volume admin service is over-sampled.

Best Practices

✓Never sample ERROR or WARN logs (keep 100%); sample only INFO, and use tail-based sampling to keep 100% of traces with errors.
✓Use consistent-hash sampling by trace_id so all logs for a kept trace are kept (no partial traces that confuse the on-call).
✓Adjust sample rate per service: 1% for high-volume (checkout), 100% for low-volume (admin); monitor 'logs kept / emitted' and adjust.
✓Temporarily raise sampling to 100% during incidents or rising error rates; sample for cost, not for signal loss.