Skip to content
Back to Analytics & Experimentation

Sample Size Calculation

Computing the minimum sample size before a test starts —so you know whether the test is feasible and never run an underpowered test that cannot detect the effect you care about.

Why

An underpowered test is worse than no test. No test means you make a decision on intuition; an underpowered test means you make the same decision on intuition but with false confidence that 'the data said so'. If you need 40,000 users per arm to detect a 2% lift and you run the test on 5,000, the result will be non-significant regardless of whether the effect is real —and you will conclude 'no effect' and kill a feature that would have worked. The sample size calculation is the reality check that prevents this: it tells you 'to detect a 2% lift on a 12% baseline with 80% power, you need 38,400 per arm, which at your traffic is 12 days'. If 12 days is too long, you either raise the MDE (accept that you can only detect 5%+ lifts), reduce variants, or do not run the test. Running it anyway is a waste of traffic that produces a non-significant result you will misinterpret.

When

Before every test, as part of the design. Re-calculate if the baseline rate changes significantly (> 20% relative shift).

How

1. Get the baseline rate of the primary metric from historical data (e.g., conversion rate = 12.4% over the last 30 days)
2. Choose the MDE: the minimum effect that would be worth shipping. This is a business decision, not a stats one —"a 2% relative lift on conversion is worth $4k/month, so MDE = 2%"
3. Choose alpha (false-positive rate, typically 0.05) and power (true-positive rate, typically 0.80; use 0.90 for high-stakes tests)
4. For a two-proportion z-test (conversion-rate metrics):
   n_per_arm = 2 * (z_alpha/2 + z_beta)^2 * p*(1-p*) / (p_t - p_c)^2
   where p* = (p_t + p_c)/2, p_t = p_c * (1 + MDE)
   Or use a calculator: https://www.evanmiller.org/ab-testing/sample-size.html
5. For continuous metrics (revenue per user, time on page), use the t-test formula with the historical standard deviation
6. Multiply by the number of variants if testing > 2 arms (each comparison needs its own sample)
7. Compute duration: n_per_arm / daily_traffic_per_arm. If duration > 4 weeks, the test is likely not feasible —raise MDE or reduce variants
8. Sanity-check: if the required sample is > 50% of your monthly traffic, you can run at most 1-2 tests per quarter, and you should consider a multi-armed bandit instead of fixed-horizon A/B

Example

# Sample size: CTA color test
Baseline CTR: 12.4% (last 30 days, 360k impressions)
MDE: 5% relative -> absolute MDE = 12.4% * 0.05 = 0.62pp
  Treatment CTR to detect: 12.4% * 1.05 = 13.02%
Alpha: 0.05 (two-sided)  Power: 0.80
z_alpha/2 = 1.96  z_beta = 0.84

p* = (0.1240 + 0.1302) / 2 = 0.1271
n = 2 * (1.96 + 0.84)^2 * 0.1271 * (1 - 0.1271) / (0.1302 - 0.1240)^2
  = 2 * 7.84 * 0.1271 * 0.8729 / 0.0000384
  = 2 * 7.84 * 0.1109 / 0.0000384
  = 1.739 / 0.0000384
  = 45,289 per arm

Daily impressions: 12,000
Duration: 45289 / 12000 = 3.77 days -> round to 4 days
Total traffic needed: 45289 * 2 * 4/3.77 = ~96,000 impressions

Feasibility: YES (96k < 360k monthly, test takes 4 days)
If MDE were 2% relative: n = 283,000 per arm -> 24 days -> borderline
If MDE were 1% relative: n = 1,132,000 per arm -> 94 days -> infeasible,
  use a bandit or accept that 1% lifts are undetectable at this traffic

Common Problems

  • ⚠Running the test without computing sample size —the team runs a 3-day test on 2,000 users, sees a non-significant result, and concludes 'no effect', when the test was never capable of detecting the 2% effect they cared about; this is a false negative caused by design, not by the feature.
  • ⚠Choosing MDE based on what is detectable rather than what is valuable —'we can detect 10% in 3 days, so MDE = 10%' means you ship only effects that are huge, and you kill every feature that moves the metric 2-5%, which is where most real features live.
  • ⚠Using 80% power by default for high-stakes decisions —80% power means a 20% chance of false negative; for a feature that would generate $500k/year, a 20% chance of killing it is a $100k expected loss, and 90% or 95% power is worth the extra traffic.
  • ⚠Forgetting to account for multiple variants —if you test 4 variants against control, you need 4x the sample per arm (or a correction), and the test takes 4x as long; not accounting for this makes every multi-variant test underpowered.

Best Practices

  • ✓Compute sample size before every test and write it in the registry —this is the gate that prevents underpowered tests; if the required sample is infeasible, the test does not run, and the traffic is saved for a test that can produce a trustworthy answer.
  • ✓Choose MDE from business value first, then check feasibility —'a 2% lift is worth $4k/month' is the right starting point; if 2% is infeasible, decide whether to accept 5% (and miss 2-4% features) or wait for more traffic, do not silently lower the bar.
  • ✓Use 90% power for high-stakes tests (revenue, retention) and 80% for low-stakes (copy, color) —the cost of a false negative on a $500k feature is 10x the cost of a few extra days of traffic, and power should reflect that asymmetry.
  • ✓Re-check the baseline rate 24h into the test —if the baseline has shifted > 20% from the historical estimate, the sample size calculation is now wrong, and the test should be re-designed or stopped.