Skip to content
Back to Analytics & Experimentation

Correlation Analysis

Measuring how two metrics move together, and the critical discipline of not confusing correlation with causation —the most common analytics error.

Why

Correlation analysis is fast, cheap, and seductive: you have two columns of numbers, you run a Pearson coefficient, and you get a number between -1 and 1 that feels like an answer. The danger is that correlation is a mathematical property of two variables, but the question you actually have is a business question: 'does X cause Y, so that if I change X, Y will change?'. Correlation cannot answer that —ice cream sales and shark attacks are correlated, but banning ice cream will not reduce shark attacks. The discipline of correlation analysis is to compute the correlation, report it honestly as 'these move together', and then do the harder work of determining whether the relationship is causal (via a controlled experiment) or confounded (both are driven by a third variable, like summer temperature). Skipping the second step is how teams ship features that 'correlated with retention' but move nothing when launched.

When

When you have two metrics and want to know if they move together, as a precursor to a causal experiment. Never as the basis for a ship decision on its own.

How

1. State the business question first: "does using feature X correlate with 30-day retention?" —the question determines which variables to correlate and how to slice
2. Pull the two series at the same granularity (per-user, per-day) and same time window —correlating daily-aggregate X with per-user Y is a meaningless number
3. Visualize with a scatter plot before computing the coefficient —a Pearson of 0.4 can be a linear cloud, a U-shape, or two clusters, and only the scatter tells you which
4. Compute Pearson (linear) and Spearman (monotonic) —if they differ significantly, the relationship is non-linear and Pearson is misleading
5. Check for confounders: is there a third variable (time, cohort, feature flag) that drives both? Slice by the confounder and re-compute within each slice
6. Report as "X and Y are correlated (r=0.42, n=1200), but we have not established causation" —the caveat is not optional
7. If the correlation is strong and you want to act on it, design an A/B test: randomly assign feature X, measure Y, and compare —only the RCT resolves the correlation/causation gap
8. Document the analysis in a notebook with the scatter, the coefficients, the confounder checks, and the caveat —so the next person does not re-discover that it was confounded

Example

# Question: does tutorial completion correlate with 7-day retention?
n = 4,200 users who signed up 2026-08-01 to 2026-08-31

Scatter: tutorial_steps_completed (0-8) vs days_active_in_first_7d (0-7)
  -> positive trend, but two visible clusters:
     cluster A (n=1800): completed 0-2 steps, retained 0-2 days
     cluster B (n=2400): completed 5-8 steps, retained 4-7 days

Pearson r = 0.61  Spearman rho = 0.58  (close, so roughly monotonic)

Confounder check: signup_source
  organic:  r = 0.18  (weak within this slice)
  referral: r = 0.64  (strong within this slice)
  -> The overall 0.61 is driven by referral users completing more tutorials
     AND retaining more, both caused by pre-existing motivation, not by
     the tutorial itself.

Conclusion: tutorial completion and retention are correlated (r=0.61),
but the correlation is largely confounded by signup source. Users who
were already motivated (referral) both complete the tutorial and retain.
DO NOT ship "force tutorial completion" based on this —run an A/B test
randomizing tutorial prominence and measure retention by source.

Common Problems

  • ⚠Computing correlation on aggregated data (daily totals) instead of per-user —two daily totals that both trend up over time will always correlate highly, but this is a tautology that says 'time passes', not 'X causes Y'.
  • ⚠Ignoring Simpson's paradox —the overall correlation is positive, but within each subgroup it is negative (or vice versa), and acting on the aggregate correlation harms every subgroup.
  • ⚠Treating a high correlation as proof of causation and shipping the feature —'users who use feature X retain better, so we should force feature X' is the classic correlation-to-causation error that A/B testing exists to prevent.
  • ⚠Reporting the coefficient without the scatter plot —a Pearson of 0.5 from a clean linear cloud and a Pearson of 0.5 from two clusters with a gap are completely different relationships, and the number alone hides which one you have.

Best Practices

  • ✓Always visualize the scatter before trusting the coefficient —the coefficient is a summary of the scatter, and a summary of a U-shape or a two-cluster distribution is a number that hides the shape that determines what action to take.
  • ✓Slice by every plausible confounder (cohort, source, plan, geography) and report the within-slice correlations —if the within-slice correlations differ from the aggregate, you have a confounder and the aggregate is misleading.
  • ✓Use correlation to generate hypotheses, then use A/B testing to test them —correlation is fast and cheap and tells you where to look; the RCT is slow and expensive and tells you the truth; you need both, in that order.
  • ✓Report the sample size and confidence interval alongside the coefficient —r=0.6 with n=30 is noise, r=0.6 with n=3000 is a finding, and the number without the n is not interpretable.