Skip to content
Back to Operations & SRE

Dashboards

Build dashboards that on-call engineers can read at 3am — fast, obvious, and action-oriented.

Why

A dashboard is a decision tool, not a data dump. At 3am during an outage, an engineer has seconds to locate the signal among the noise, and a 30-panel grid of vanity charts will cost you minutes of MTTR every time. Good dashboards answer one question per panel, put the most important signal top-left (where the eye lands), use color only for state (green/yellow/red), and link each panel to the runbook that explains what to do when it goes red. Teams that treat dashboards as art projects end up with pretty walls of charts that nobody uses under pressure, while the on-call engineer squints at a raw Prometheus query in a terminal instead.

When

After metrics instrumentation is in place. Rebuild if on-call engineers regularly open a terminal instead of the dashboard during incidents, or if a dashboard has >20 panels.

How

1. Define the audience and the question
   - On-call dashboard: "is the service healthy, and if not, where?"
   - Capacity dashboard: "are we near a limit?"
   - Business dashboard: "are users succeeding?" — different audience, different panel

2. Layout follows eye-tracking: top-left = most important
   - Row 1: golden signals (RED summary as 3 big stat panels with thresholds)
   - Row 2: latency heatmap + error rate by route
   - Row 3: dependencies (DB, cache, downstream services) health
   - Row 4: saturation signals (CPU, memory, queue depth, connection pool)

3. One question per panel
   - Bad: "all metrics for the orders service" (a dump)
   - Good: "error rate for POST /api/orders over 5m" (a decision)
   - If you can't phrase the panel as a question, delete it

4. Color = state, not decoration
   - Green/yellow/red thresholds tied to SLO burn rate
   - No rainbow color schemes, no 20-series line charts with a legend
   - A panel that's always green is a candidate for deletion

5. Link every panel to a runbook
   - Panel title or description links to the runbook for "what to do when this is red"
   - Include the query and the alert name so the engineer can investigate deeper

6. Test with a real incident
   - During the next outage, watch the on-call engineer use it
   - If they leave the dashboard for a terminal, the dashboard failed — fix it

Example

# Grafana dashboard as code (JSON in Git)
{
  "title": "Orders Service — On-Call",
  "panels": [
    { "title": "Req/s",  "gridPos": {"h":4,"w":8,"x":0,"y":0},
      "targets": [{"expr": "sum(rate(http_requests_total{service='orders'}[5m]))"}] },
    { "title": "Error %", "gridPos": {"h":4,"w":8,"x":8,"y":0},
      "thresholds": {"steps":[{"color":"green"},{"color":"red","value":1}]},
      "targets": [{"expr": "sum(rate(http_requests_total{service='orders',status=~'5..'}[5m])) / sum(rate(http_requests_total{service='orders'}[5m])) * 100"}] },
    { "title": "p95 latency", "gridPos": {"h":4,"w":8,"x":16,"y":0},
      "targets": [{"expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{service='orders'}[5m])) by (le))"}] }
  ],
  "links": [{"title": "Runbook: orders high error rate", "url": "https://runbooks/operate/orders-high-error"}]
}
# Rule of thumb: if a panel doesn't change behavior when it moves, it's decoration — delete it.

Common Problems

  • ⚠Dashboards with 30+ panels that take 30s to load — on-call engineers abandon them and query Prometheus directly, defeating the purpose.
  • ⚠Using line charts for everything when heatmaps and stat panels communicate state faster — a p99 line that's flat at 50ms tells you nothing about the tail distribution.
  • ⚠No runbook links — a red panel with no "what to do" link forces the engineer to reinvent the investigation at 3am, inflating MTTR.
  • ⚠Color used for decoration (rainbow series) instead of state — engineers can't spot the red signal in a sea of colored lines.

Best Practices

  • ✓Treat dashboards as code: store Grafana JSON in Git, review changes in PRs, and version templates so improvements propagate to all services.
  • ✓Cap on-call dashboards at 12 panels max; if you need more, split into a second dashboard linked from the first, so the primary loads in <3s.
  • ✓Every panel must link to a runbook and have thresholds tied to an SLO — a panel without a threshold is a chart, not a monitoring signal.
  • ✓Do a "dashboard retrospective" after each incident: did the on-call engineer find the cause from the dashboard or a terminal? Fix whichever failed.