Metric Definition

How to define a metric that is measurable, attributable, and resistant to gaming —the foundation of all analytics work.

Why

A metric is the unit of decision. If the unit is wrong, every decision derived from it is wrong, no matter how sophisticated the analysis. Most analytics failures are not math failures but definition failures: a 'signup' that counts bots, an 'active user' that counts a page load as activity, a 'conversion' that double-counts refunds. A good metric definition forces the team to specify the numerator, the denominator, the time window, the exclusion rules, and the known edge cases before any number is reported. This upfront cost is what makes the metric trustworthy six months later when someone asks 'why did revenue drop'. Without a written definition, every team member computes the metric differently and the dashboard shows whatever the last engineer implemented.

When

Before instrumenting any event or building any dashboard. Re-define when the business model changes, when the metric is gamed, or when cross-team numbers disagree by more than 5%.

How

1. Name the metric in plain language: "Daily Active Users" is not a definition; "unique users who performed at least one core action in the last 24h" is
2. Specify the numerator: which events count, deduplicated by user_id, excluding internal/test accounts
3. Specify the denominator: if it is a rate, what is the base population and time window
4. List exclusions: bot traffic, test accounts, refunded orders, employees
5. Write the SQL or event query that computes it, and pin it in a versioned metrics dictionary
6. List 3 known edge cases and how to handle them (e.g., user crosses midnight timezone boundary)
7. Get sign-off from the team that will be measured against it —they will find the gaming vectors you missed
8. Add a unit test that asserts the metric on a fixed dataset so regressions are caught

Example

# Metric: Activation Rate
Owner: @growth-lead  Version: 2.1  Last reviewed: 2026-09-15

Numerator: users who created AND deployed a project within 7 days of signup
Denominator: users who signed up in the same cohort (excluding test/bot accounts)
Time window: 7-day rolling, evaluated at signup_time + 7d
Exclusions: email domain in (test.dev, example.com), user_id < 1000 (internal)

SQL:
  SELECT
    count(DISTINCT a.user_id) AS activated,
    count(DISTINCT s.user_id) AS total,
    count(DISTINCT a.user_id)::float / count(DISTINCT s.user_id) AS rate
  FROM signups s
  LEFT JOIN activations a ON a.user_id = s.user_id
    AND a.activated_at <= s.signup_at + interval '7 days'
  WHERE s.email_domain NOT IN ('test.dev','example.com')
    AND s.user_id >= 1000

Edge cases:
  - User deploys then deletes project: still counts (deployment event fired)
  - User signs up at 23:59 UTC: window is signup_at + 7d in UTC, not local
  - User uses two browsers: deduplicated by user_id, not cookie

Unit test: fixtures/activation_rate.sql asserts rate = 0.421 on 1000-row sample

Common Problems

⚠Defining a metric by its implementation rather than its intent —'DAU is what the query returns' means the metric drifts every time someone edits the query, and nobody notices because there is no definition to compare against.
⚠Counting bots and test accounts in production metrics —a 15% bot inflates DAU and makes every A/B test look more significant than it is, leading to shipping changes that do not actually move the human metric.
⚠Using a vanity metric (total signups, page views) as a north star —it goes up monotonically regardless of product quality, so it cannot tell you whether the product is getting better or worse, only whether time is passing.
⚠No versioning on the metric definition —when the SQL changes, historical numbers change retroactively, and the team loses the ability to compare this month to last month because the definition is a moving target.

Best Practices

✓Maintain a versioned metrics dictionary (a single repo file or Notion page) where every metric has an owner, a definition, a SQL query, and a last-reviewed date —this is the single source of truth that prevents 'which DAU?' arguments.
✓Add a unit test for every metric on a frozen fixture dataset so that a SQL change that alters the number is caught in CI before it reaches the dashboard —the test is what makes the definition binding.
✓Review every metric definition quarterly with the team that is measured against it —they will surface gaming vectors ('if I deploy an empty project does it count as activation?') that the definition owner did not anticipate.
✓Distinguish counter metrics (cumulative, monotonic) from rate metrics (bounded, comparable across time) in the dictionary —mixing them leads to 'total signups went up 10%' headlines that are just time passing.