Metrics Design

Choose the right metrics (RED, USE, RUM) and instrument them consistently across services.

Why

You cannot operate what you cannot measure, but most teams measure the wrong things. Vanity metrics (request count, signup count) go up and to the right and tell you nothing about health. The RED method (Rate, Errors, Duration) for services and USE (Utilization, Saturation, Errors) for resources give you a complete, orthogonal view of system health with just three signals each. Picking a standard method up front means every service emits the same shape of metrics, so dashboards and alerts are composable instead of bespoke per team. Teams that skip this end up with 500 unrelated Grafana panels and still miss outages because nobody wired the right signal to an alert.

When

Before launching any service to production. Revisit when adding a new service or changing a critical dependency. If you have >10 ad-hoc dashboards and no shared template, you are overdue.

How

1. Pick a method per layer
   - Service layer: RED — Rate (req/s), Errors (error rate), Duration (latency p50/p95/p99)
   - Resource layer: USE — Utilization (% busy), Saturation (queue depth), Errors (device errors)
   - User layer: RUM — page load, Core Web Vitals (LCP, FID, CLS)

2. Choose a cardinality-safe label set
   - Good: method, status, route (low cardinality, high signal)
   - Bad: user_id, request_id, session_id (high cardinality, explodes storage)
   - Rule: if a label can take >100 values, it belongs in logs/traces, not metrics

3. Instrument with a standard library
   - Prometheus client_lib / OpenTelemetry SDK — emit counters, histograms, gauges
   - Histograms for latency (use bucket boundaries: 5,10,25,50,100,250,500,1000,2500,5000,10000ms)
   - Counters for rate/errors (monotonic, never reset)

4. Build one canonical dashboard per service from the template
   - Top: RED summary (req/s, error %, p95 latency) as 3 stat panels
   - Middle: latency heatmap + error breakdown by route
   - Bottom: upstream/downstream health (their RED) for blast radius

5. Validate before relying on it
   - Generate load (k6/hey), confirm metrics appear and match
   - Kill a dependency, confirm error metric rises
   - If a panel is flat during a known event, your instrumentation is wrong

Example

# Python + prometheus_client
from prometheus_client import Counter, Histogram, generate_latest

REQUESTS = Counter('http_requests_total', 'Total requests', ['method', 'route', 'status'])
LATENCY = Histogram('http_request_duration_seconds', 'Latency', ['route'],
                    buckets=(0.005,0.01,0.025,0.05,0.1,0.25,0.5,1,2.5,5,10))

@app.route('/api/orders')
def orders():
    with LATENCY.time(route='/api/orders'):
        try:
            return handle()
        except Exception:
            REQUESTS.labels('GET','/api/orders','500').inc(); raise
        finally:
            REQUESTS.labels('GET','/api/orders','200').inc()

# Grafana panel query (RED - Rate):
#   rate(http_requests_total[5m])
# RED - Errors:
#   rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])
# RED - Duration p95:
#   histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

Common Problems

⚠Using high-cardinality labels (user_id, request_id) in metrics — explodes storage cost and makes queries slow; put those in logs/traces instead.
⚠Averaging latency instead of using histograms and quantiles — averages hide the tail; p99 is what users actually experience during incidents.
⚠Counting only 5xx as errors while ignoring 4xx spikes — a sudden 401/403 flood often means auth is broken, which is an operational error.
⚠Resetting counters on deploy — breaks rate() calculations; counters must be monotonically increasing across restarts (use a process restart gauge separately).

Best Practices

✓Adopt one method (RED for services, USE for resources) org-wide so every dashboard and alert is composable and on-call engineers don't relearn each service.
✓Use histograms with consistent bucket boundaries across all services so p95/p99 can be compared apples-to-apples in a single panel.
✓Keep label cardinality under 100 per label; if you need to slice by user or request, that's a trace or log query, not a metric.
✓Version your dashboard templates as code (Grafana JSON in Git) so improvements propagate to all services, not just the one an engineer happened to touch.