Variant Creation

Why

When

How

1. Put the change behind a feature flag: if (flagService.get("cta-color-blue", userId)) { render blue } else { render grey }
2. Implement the assignment as a deterministic hash: bucket = hash(userId + experimentId) % 100; treatment = bucket < 50. This is sticky (same user -> same bucket) and reproducible
3. Use a real feature-flag service (LaunchDarkly, Statsig, GrowthBook) or a shared in-house one —do not roll per-experiment assignment logic, because it will be inconsistent across experiments
4. Ensure the variant code is isolated: the treatment branch should not share mutable state with the control branch, and a user in treatment should never see control UI (no "flash of grey then blue")
5. Test the assignment in staging: verify that a known userId maps to the expected bucket, that the bucket is stable across requests, and that disabling the flag returns control for all users
6. Add a fallback: if the flag service is unreachable, default to control (not treatment) —shipping a feature because the flag service is down is the worst kind of incident
7. Instrument the exposure: log an "experiment_exposure" event with experiment_id, variant, user_id on first pageview —this is the denominator for all experiment metrics
8. QA the variant in staging: a human clicks through the treatment flow and verifies the change is present, consistent, and does not break any other page (regression check)

Example

// assignment.ts —shared experiment assignment
import { hash } from "./hash";

export function getVariant(
  experimentId: string,
  userId: string,
  trafficSplit: number = 0.5,
): "control" | "treatment" {
  const bucket = hash(userId + experimentId) % 100;
  return bucket < trafficSplit * 100 ? "treatment" : "control";
}

// cta.tsx —variant behind flag
import { getVariant } from "./assignment";
import { flagService } from "./flags";

export function CheckoutCta({ userId }: { userId: string }) {
  // Fallback to control if flag service is down
  const flagOn = flagService.get("cta-color-blue", userId, false);
  if (!flagOn) return <GreyCta />;
  const variant = getVariant("cta-color-blue-447", userId);
  return variant === "treatment" ? <BlueCta /> : <GreyCta />;
}

// Exposure event (logged once per user per experiment)
useEffect(() => {
  if (!exposureLogged) {
    analytics.track("experiment_exposure", {
      experiment_id: "cta-color-blue-447",
      variant: getVariant("cta-color-blue-447", userId),
      user_id: userId,
    });
    setExposureLogged(true);
  }
}, []);

// Staging QA checklist:
// [x] user_001 -> bucket 23 -> treatment (blue CTA)
// [x] user_002 -> bucket 67 -> control (grey CTA)
// [x] user_001 refreshes 5x -> still treatment (sticky)
// [x] flag disabled -> all users see grey (fallback)
// [x] flag service killed -> all users see grey (fallback)
// [x] no flash of grey-then-blue on any page

Common Problems

• ⚠Re-randomizing on every request instead of sticky assignment —the user sees control on page 1 and treatment on page 2, their behavior is a mix, and the result measures nothing; this is the most common assignment bug and it is invisible in the dashboard.
• ⚠Defaulting to treatment when the flag service is down —a flag-service outage ships the feature to 100% of users with no control bucket, the experiment is destroyed, and you cannot roll back without another deploy.
• ⚠Variant code that leaks into shared components —the treatment change is in a shared Button component, so users in control also see it on pages that use Button, and the control bucket is contaminated.
• ⚠Logging exposure on every pageview instead of once per user —the denominator is inflated, the metric is wrong, and the test appears to have more power than it does, leading to false significance.

Best Practices

• ✓Use a deterministic hash of userId + experimentId for assignment —this is sticky, reproducible, and does not require a database lookup, so it is fast and consistent across services.
• ✓Default to control on any flag-service failure —the cost of showing control to treatment users for a few minutes is a slightly underpowered test; the cost of showing treatment to all users is a destroyed experiment and a potential incident.
• ✓Log exposure exactly once per user per experiment (on first pageview in the experiment scope) —this is the denominator for every metric, and getting it wrong makes every number in the test wrong.
• ✓QA the variant in staging with a checklist: sticky across refreshes, correct bucket for known users, fallback works when flag service is killed, no visual flash —the checklist takes 10 minutes and catches the assignment bugs that would otherwise corrupt a 2-week test.