Randomization

Why

When

How

1. Use stable assignment: hash the user_id with the experiment_id to a bucket (0-99), and store the assignment in a lookup table —the user is in the same arm for the entire experiment, even if they clear cookies or switch devices
2. Use a deterministic hash (SHA-256 or MurmurHash3), not Math.random() —Math.random() gives a different value on each call, producing unstable assignment; the hash of (user_id, experiment_id) is deterministic and stable
3. Assign to arms using the hash bucket: bucket < 50 -> control, bucket >= 50 -> treatment (for a 50/50 split); for an 80/20 split, bucket < 80 -> control, bucket >= 80 -> treatment
4. Verify balance after assignment: check that the treatment and control groups have similar baselines (pre-experiment conversion rate, ARPU, tenure) —if they differ by more than 1 standard error, the randomization is broken and the experiment is invalid
5. Use block randomization for small samples: if the sample is small (< 1,000 per arm), simple randomization can produce imbalance by chance; block randomization (assign in blocks of 4: 2 control, 2 treatment per block) guarantees balance
6. Use stratified randomization for heterogeneous populations: if the population has distinct segments (free vs paying, US vs international), stratify within each segment so both arms have the same segment mix —this reduces variance and increases power
7. Handle carryover (interference) with cluster randomization: if users interact (B2B, social), randomize by cluster (company, workspace) not by user —all users in a company get the same arm, so there is no cross-arm contamination
8. For network-effect products, use geo-experimentation: randomize by region (US vs EU vs Asia) or by city —this avoids the problem where a user in treatment affects a user in control through the network
9. Exclude users who could be exposed to both arms: employees, QA testers, users who switch accounts —these users contaminate both arms and should be excluded from the analysis
10. Log the assignment: every user's assignment (user_id, experiment_id, arm, assigned_at) is logged, so the analysis can verify stable assignment and reproduce the experiment

Example

# Randomization: EXP-247 (trial length)
Experiment ID: EXP-247
Split: 50/50 (control: 14-day, treatment: 7-day)

Assignment function (TypeScript):
  function assignArm(userId: string, expId: string): 'control' | 'treatment' {
    const hash = murmurhash3(userId + expId);  // deterministic
    const bucket = hash % 100;                  // 0-99
    return bucket < 50 ? 'control' : 'treatment';
  }

  // Stored in experiment_assignments table:
  // (user_id, experiment_id, arm, assigned_at)
  // ON CONFLICT DO NOTHING -- stable assignment

Balance check (pre-experiment, 7-day baseline):
  Metric              | Control  | Treatment | Diff  | SE    | OK?
  -------------------|----------|-----------|-------|-------|----
  prior conversion   | 3.18%    | 3.22%     | 0.04% | 0.12% | yes
  prior ARPU         | $24.40   | $24.60    | $0.20 | $0.80 | yes
  tenure (months)    | 4.2      | 4.3       | 0.1   | 0.3   | yes
  US %               | 62%      | 61%       | 1%    | 3%    | yes
  paying %           | 15%      | 15%       | 0%    | 2%    | yes
  -> All diffs < 1 SE, randomization is balanced

Carryover handling:
  - B2B users (company_id != null): cluster randomization by
    company_id, so all users in a company get the same arm
  - Consumer users (company_id == null): individual randomization
  - Assignment function:
    if (user.companyId) {
      return assignArm(user.companyId, expId);  // cluster
    } else {
      return assignArm(user.id, expId);          // individual
    }

Exclusions:
  - user_id < 1000 (internal/seed accounts)
  - email_domain IN ('test.dev', 'internal.ez4code.com')
  - users who switched accounts in last 30 days (cross-arm risk)

Assignment log (sample):
  user_id | exp_id   | arm       | assigned_at
  --------|----------|-----------|---------------------
  1234    | EXP-247  | control   | 2026-09-22 00:01:00
  1235    | EXP-247  | treatment | 2026-09-22 00:01:00
  1236    | EXP-247  | control   | 2026-09-22 00:01:00
  (38,400 per arm, stable for 18-day experiment)

Common Problems

• ⚠Using Math.random() for assignment —it gives a different value on each call, so a user is in treatment on page load 1 and control on page load 2; the assignment is unstable, the behavior is contaminated, and the experiment measures noise.
• ⚠Not storing the assignment —the hash is recomputed on each request, and if the hash function or the user_id changes (account merge, cookie clear), the user switches arms mid-experiment and their data is unanalyzable.
• ⚠No balance check —the randomization bug puts 60% of paying users in treatment, the treatment 'wins' on revenue, and the team ships a change that does not work; the balance check (pre-experiment baselines) catches this before the experiment starts.
• ⚠Individual randomization in a B2B product —user A in treatment tells user B in control (same company) about the feature, and control's behavior is contaminated; cluster randomization (by company) prevents cross-arm contamination in collaborative products.
• ⚠Including employees and QA testers in the experiment —they see both arms (because they test both), their behavior is unlike real users, and they contaminate both arms; exclude them by user_id or email domain before assignment.

Best Practices

• ✓Use a deterministic hash (MurmurHash3 or SHA-256) of (user_id, experiment_id) for assignment, and store the result in a lookup table —this guarantees stable assignment (the user is in the same arm for the entire experiment) even if cookies are cleared or devices change.
• ✓Verify balance after assignment by comparing pre-experiment baselines (conversion rate, ARPU, tenure) between arms —if any metric differs by more than 1 standard error, the randomization is broken, the experiment is invalid, and the assignment must be debugged before launching.
• ✓Use cluster randomization (by company or workspace) for B2B and collaborative products —individual randomization causes cross-arm contamination when users in different arms interact, and cluster randomization ensures all users in a company see the same arm.
• ✓Use stratified randomization for heterogeneous populations (free vs paying, US vs international) —stratifying within each segment ensures both arms have the same segment mix, which reduces variance and increases the experiment's power.
• ✓Exclude employees, QA testers, and account-switchers from the experiment —they see both arms, their behavior is unlike real users, and they contaminate both arms; the exclusion list (by user_id or email domain) must be applied before assignment.