Structured Logging

Emit logs as JSON with fields (timestamp, level, service, trace_id, message), not unstructured text.

Why

Unstructured logs ('checkout failed for user 12345 at 14:05') are human-readable but machine-hostile — you can't filter by service, group by trace_id, or alert on a field. When an incident hits, the on-call greps 10,000 lines of text, finds 3 relevant, and misses the 4th that explains the root cause. Structured logs ({ts, level, service, trace_id, user_id, msg, error}) are queryable: 'show me all ERROR logs for trace_id=abc in the last 5m' returns 3 lines in 50ms, and you can alert on 'error.rate > 1/s by service'. The trace_id is the multiplier — it links logs across services, so a checkout failure shows the cart-service log, the payment-service log, and the DB query, all in one query. Teams that switch to structured logging report 60% faster root-cause identification because the logs are queryable, not greppable.

When

From day one of a new service. Retrofit structured logging to existing services during the next refactor.

How

1. Choose a structured logging library and format
   - JSON is the standard; most log aggregators (Loki, Datadog, ELK) parse it natively
   - Use the library's structured API, not string interpolation: logger.info({service, trace_id, msg})

2. Include these fields in every log
   - timestamp (ISO 8601, UTC), level (DEBUG/INFO/WARN/ERROR), service, trace_id, span_id
   - Add request-specific fields: user_id, route, status_code, latency_ms
   - The trace_id is non-negotiable — it's how you link logs across services

3. Propagate the trace_id across service boundaries
   - Incoming request → generate trace_id (or accept from header) → put in context
   - Outgoing requests (HTTP, gRPC, DB) → include trace_id in headers/metadata
   - This is distributed tracing; OpenTelemetry does it automatically

4. Log at the right level and volume
   - INFO: request start/end, business events (order placed, payment failed)
   - WARN: degraded but functional (cache miss, retry succeeded)
   - ERROR: request failed, needs attention; include the error and stack
   - DEBUG: detailed diagnostics, off in prod by default

5. Don't log sensitive data
   - No passwords, tokens, PII, or full request bodies in prod logs
   - Use a log sanitizer that redacts known sensitive fields

Example

// Bad: unstructured text
console.log("checkout failed for user " + userId + " at " + new Date());

// Good: structured JSON
import { logger } from "./logger";
logger.error({
  service: "checkout",
  trace_id: req.traceId,      // propagated from the gateway
  span_id: generateSpanId(),
  user_id: req.userId,
  route: "/checkout",
  status_code: 500,
  latency_ms: 1240,
  msg: "checkout failed",
  error: err.message,
  stack: err.stack,
});
// Output (JSON, one line):
{"ts":"2026-03-15T14:05:01Z","level":"ERROR","service":"checkout",
 "trace_id":"abc123","span_id":"def456","user_id":12345,
 "route":"/checkout","status_code":500,"latency_ms":1240,
 "msg":"checkout failed","error":"null discount","stack":"..."}

// Query in Loki/Datadog (find all logs for this trace):
{trace_id="abc123"}  → returns checkout, cart, payment, db logs in one query
// Alert on error rate by service:
rate({level="ERROR",service="checkout"}[5m]) > 1  → pages checkout on-call

Common Problems

⚠String-interpolated logs — human-readable but unqueryable; the on-call greps 10k lines and misses the root cause.
⚠No trace_id — logs from cart, checkout, and payment services can't be linked; the on-call queries each service separately and correlates by timestamp.
⚠Logging sensitive data — passwords or PII in prod logs creates a compliance and security incident on top of the technical one.
⚠Everything at INFO level — no signal; the on-call can't filter to what matters, so they grep and hope.

Best Practices

✓Emit logs as JSON with timestamp, level, service, trace_id, span_id, and request fields; use the library's structured API, not string interpolation.
✓Propagate trace_id across all service boundaries (HTTP, gRPC, DB) so logs link across services; OpenTelemetry does this automatically.
✓Log at the right level (INFO for events, WARN for degraded, ERROR for failures with stack) and never log sensitive data (passwords, tokens, PII).
✓Query logs by field (trace_id, service, level) in a log aggregator; alert on structured fields (error.rate by service), not grepped text.