Hypothesis-Driven Testing

Frame every growth experiment as a falsifiable hypothesis with a metric, a predicted effect, and a stop condition before you ship it.

Why

A hypothesis ('raising the CTA contrast from 30% to 70% will lift activation by 8% ± 2%') forces you to commit to a prediction, which makes learning visible: you either confirmed it, refuted it, or were vague. The pre-registered predicted effect size lets you size the experiment for statistical power, so you don't run a test too small to detect the effect you're looking for and then conclude 'no effect'. The stop condition prevents p-hacking and the 'just one more week' trap that inflates false positives. Without hypotheses, experiments become 'we tried a thing and the number went up' theater — no accumulated learning, no reusable playbook, and the team repeats the same disproven ideas.

When

When you run more than 2 experiments per month and need the learning to compound. Adopt once, enforce on every experiment.

How

1. Write the hypothesis in fixed form: "If we [change], then [metric] will [rise/fall] by [X%] ± [Y%], because [mechanism]"
2. Pick the primary metric (one), the guardrail metrics (2-3 that must not regress), and the minimum detectable effect
3. Calculate the required sample size for 80% power and 5% significance using an online calculator
4. Pre-register the hypothesis, sample size, and stop date in your experiment log BEFORE shipping
5. Ship the variant, split traffic 50/50, and wait until the full sample is collected — do not peek
6. At the stop date, compute the effect and its confidence interval; classify as won, lost, or inconclusive
7. Write one paragraph of learning regardless of outcome — what did this tell you about users?
8. Add confirmed hypotheses to the playbook; archive refuted ones so they are not re-attempted

Example

# Hypothesis (pre-registered before shipping)
H1: "If we add a 30-second product tour video to the empty state,
    activation rate will rise from 25% to 30% ± 2%, because users
    currently don't understand what to do first."
Primary metric: activation_rate (performed core action within 24h)
Guardrails: signup_conv (must not drop >1%), time_to_first_action
MDE: 5% absolute lift
Sample size: 8,400 per arm (calculated for 80% power, 5% sig)
Stop date: 2026-11-15 or when N reached, whichever first

# Result at stop date
Control activation: 25.2% (n=8400)
Variant activation: 28.7% (n=8400)
Effect: +3.5% [95% CI: +1.8% to +5.2%]  → WON
Learning: empty-state confusion is a real activation barrier; video
          works because it shows the first action, not the feature list.

Common Problems

⚠Running experiments without a pre-registered effect size, then concluding 'significant' on a peeked p-value — false positives accumulate.
⚠Stopping the test the day the number looks good — this guarantees inflated effect estimates that vanish in production.
⚠Testing 5 variants against 1 control with no multiple-comparison correction — one will look good by chance.
⚠Never writing down the learning, so the team re-runs the same disproven idea 6 months later.

Best Practices

✓Pre-register the hypothesis, sample size, and stop date before shipping; this is what separates an experiment from a vibe check.
✓Calculate sample size from the minimum detectable effect — a test too small to detect the effect you want is a wasted test.
✓Wait until the full sample is collected before reading the result; peeking inflates false positives to 30%+.
✓Write one paragraph of learning for every experiment, won or lost — the accumulated learning is the real output, not the single win.