APM Setup

Install Application Performance Monitoring to get distributed traces, service maps, and per-request profiling.

Why

Metrics tell you a service is slow; APM tells you why. A distributed trace follows a single request across every service, database, and queue it touches, so when p95 latency spikes you can see that 80% of the time was spent in a single Postgres query inside the inventory service — not "somewhere in the checkout flow." The service map gives you a live dependency graph that updates as code ships, which is the only reliable way to understand blast radius during an incident. Without APM, a microservices team operates blind to cross-service latency, and every incident becomes a multi-team finger-pointing exercise instead of a 5-minute trace read.

When

When you have 2+ services that talk to each other in production. Solo monoliths can start with metrics + logs; the moment a second service appears, you need traces to see across the boundary.

How

1. Pick an APM that matches your stack
   - OSS: OpenTelemetry SDK → Jaeger/Tempo collector (self-hosted) or Grafana Cloud
   - Commercial: Datadog, New Relic, Dynatrace, Honeycomb — pick by language support + budget

2. Instrument at the boundaries
   - Auto-instrumentation for HTTP frameworks, DB drivers, message queues (most APMs do this via one agent)
   - Add custom spans for business logic: "validate_cart", "apply_discount" — not just "handle_request"
   - Propagate trace context (W3C traceparent header) across every hop, including async queues

3. Configure sampling
   - Head-based: sample 1-10% at the entry point (deterministic, cheap, misses rare errors)
   - Tail-based: keep 100% of errors + slow traces, sample the rest (expensive but catches anomalies)
   - Start with 5% head-based + 100% of errors; move to tail-sampling when you can afford it

4. Build the service map and use it
   - Confirm it matches your mental model; missing edges = missing instrumentation
   - During incidents, check the map first to see blast radius before diving into traces

5. Add trace-to-log correlation
   - Inject trace_id into log lines so a trace links to the exact logs for that request
   - This is the bridge that turns "slow" (trace) into "why" (log) in one click

Example

// Node + OpenTelemetry auto-instrumentation
// tracing.js (loaded with --require ./tracing.js)
const opentelemetry = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
new opentelemetry.NodeSDK({
  traceExporter: new opentelemetry.ConsoleSpanExporter(), // → OTLP collector in prod
  instrumentations: [getNodeAutoInstrumentations()],
}).start();

// Custom span for business logic
const tracer = opentelemetry.trace.getTracer('checkout');
app.post('/api/checkout', async (req, res) => {
  const span = tracer.startSpan('checkout.validate_cart');
  try { await validateCart(req.body); }
  finally { span.end(); }
  // ... DB query auto-instrumented as a child span
});
// In Jaeger: search service=checkout, sort by duration desc
// → see the slow trace, expand spans, find the Postgres query taking 800ms

Common Problems

⚠Forgetting to propagate trace context across async boundaries (queues, batch jobs) — you get a new trace per hop instead of one end-to-end trace, losing the cross-service view.
⚠Sampling too aggressively (0.1%) to save cost — you miss the rare slow trace that is exactly what you need during an incident; keep 100% of errors.
⚠Only auto-instrumenting HTTP/DB with no business-logic spans — traces show "Postgres took 800ms" but not which business rule caused the query; add custom spans.
⚠APM agent overhead inflating latency — a heavy agent on every request adds 5-10ms; benchmark and tune sampling if p95 rises after install.

Best Practices

✓Propagate W3C traceparent headers across every hop including async queues and batch jobs — a broken trace chain is worse than no trace because it misleads.
✓Use tail-based sampling that keeps 100% of errors and slow traces while sampling normal traffic — this catches anomalies without breaking the budget.
✓Add custom spans for business logic, not just framework calls — "apply_discount" is actionable; "postgres.query" is not when you have 200 queries.
✓Correlate traces with logs via trace_id injection — the one-click jump from a slow span to the exact log line is what makes APM pay for itself.