API Latency Optimization

Reduce the time from request to response, focusing on the p95 and p99, not the average.

Why

API latency is the time your user spends waiting, and it compounds — a page that makes 5 API calls with 200ms each takes 1s to load, and a mobile user on a slow connection waits even longer. The average latency is misleading because it hides the slow tail: an average of 100ms with a p99 of 2s means 1% of users wait 20x longer, and those are the users who see timeouts and errors. Optimizing latency is optimizing the tail — the p95 and p99 — because the fast requests are already fast, and the slow ones are where users suffer. Every 100ms of API latency drops conversion by ~1% (Amazon's finding), so latency is directly a revenue metric.

When

After establishing a latency baseline with APM. Optimize the slowest endpoints first (Pareto — 20% of endpoints cause 80% of latency). Re-optimize whenever a query or feature change could regress latency.

How

1. Measure latency with APM, broken down by endpoint
- Datadog, New Relic, or OpenTelemetry + Jaeger
- Look at p50, p95, p99 per endpoint — not just the average
- Identify the slowest endpoints and the slowest percentiles

2. Profile the slow request to find where time is spent
- Distributed tracing shows the span breakdown: DB, cache, external API, compute
- Is it a slow query? A missing index? An N+1? A synchronous external call?
- Fix the dominant span first — optimizing a 10ms span when the DB takes 500ms is wasted effort

3. Optimize the database (usually the biggest span)
- Add indexes for the columns in WHERE, JOIN, and ORDER BY
- Eliminate N+1 queries: use eager loading (includes/join) or batch loading
- Use connection pooling — a new connection per request adds 50-100ms of handshake
- Consider read replicas for read-heavy endpoints

4. Add caching for repeated responses
- Cache the full response in Redis/Memcached with a TTL (e.g., product catalog: 5 min)
- Use HTTP caching headers (Cache-Control, ETag) for CDN-level caching
- Cache at the application level for expensive computations (memoization)
- Invalidate on write — stale cache is a correctness bug

5. Parallelize independent operations
- If the endpoint calls 3 independent services, call them with Promise.all, not sequentially
- 3 × 100ms sequential = 300ms; 3 × 100ms parallel = 100ms
- Use async/await throughout — never block the event loop with synchronous I/O

Example

// Before: N+1 query, sequential external calls — 800ms total
async function getOrderDetails(orderId: string) {
  const order = await db.query("SELECT * FROM orders WHERE id = $1", [orderId]); // 50ms
  const items = await db.query("SELECT * FROM order_items WHERE order_id = $1", [orderId]); // 50ms
  // N+1: one query per item to get product info
  const products = await Promise.all(items.map(i =>
    db.query("SELECT * FROM products WHERE id = $1", [i.product_id]) // 50ms × N
  ));
  const shipping = await fetchShippingStatus(orderId); // 200ms external API
  const tracking = await fetchTrackingInfo(orderId);   // 200ms external API
  return { order, items, products, shipping, tracking };
}

// After: batch query, parallel external calls — 250ms total
async function getOrderDetails(orderId: string) {
  const [order, items, products] = await Promise.all([
    db.query("SELECT * FROM orders WHERE id = $1", [orderId]),           // 50ms
    db.query("SELECT * FROM order_items WHERE order_id = $1", [orderId]), // 50ms
    // Single query with JOIN instead of N+1
    db.query(`
      SELECT p.* FROM products p
      JOIN order_items oi ON p.id = oi.product_id
      WHERE oi.order_id = $1
    `, [orderId]),                                                        // 50ms
  ]);
  // Parallel instead of sequential: 200ms instead of 400ms
  const [shipping, tracking] = await Promise.all([
    fetchShippingStatus(orderId),  // 200ms
    fetchTrackingInfo(orderId),    // 200ms (parallel)
  ]);
  return { order, items, products, shipping, tracking };
}

Common Problems

⚠Optimizing the average latency and declaring victory while the p99 is still 5 seconds — 1% of users still see timeouts, and those are the support tickets and abandoned carts.
⚠Adding a cache without an invalidation strategy — the cache serves stale data after a write, users see incorrect information, and the bug is intermittent and hard to reproduce.
⚠Parallelizing calls that are not actually independent (one depends on the other's result) — the parallel call fails because its input is not ready, and the error is confusing.
⚠Adding an index but not measuring the improvement — the index helps one query but slows writes, and the net latency may be unchanged or worse.

Best Practices

✓Measure p95 and p99 per endpoint in production with APM, and set SLOs (e.g., p95 < 200ms) — the tail is where users suffer, and the average hides it.
✓Use distributed tracing to find the dominant span (usually the database) and optimize that first — a 500ms query dwarfs a 10ms serialization optimization.
✓Eliminate N+1 queries with a single JOIN or batch loading — this is the most common latency bug and the easiest to fix, often cutting latency in half.
✓Parallelize independent I/O with Promise.all (or asyncio.gather) — sequential calls to independent services waste time that parallelization eliminates for free.