Experimentation and A/B testing¶

Measuring the effect of a change by randomly splitting users into groups that see different versions, and comparing their outcomes.

Before reading

The data science workflow

Why it matters¶

Observational analysis can show that customers who use a feature buy more, but not whether the feature made them buy more. A randomised experiment can, because the only systematic difference between the groups is the change itself. Experiments depend heavily on data: assignment must be logged, metrics must be defined consistently, and the analysis must avoid a few well-known traps.

How it works¶

An experiment compares a control group, which sees the current version, with one or more treatment groups, which see the change.

The randomisation unit is what gets assigned: usually a user, sometimes a session, device, store or region. Every event must be attributable to the unit, and the unit must stay in its group for the whole experiment, so assignment is usually a deterministic hash of the user ID and the experiment ID. Choosing a session when the outcome is measured per user, or splitting users who share a household, produces groups that influence each other.

Assignment logging is the platform's first job. Each assignment, with the unit, experiment, variant and time, is recorded as an event at the moment the user is exposed to the change, not merely allocated. Analysis then joins assignments to outcome events. Users allocated but never exposed, such as those who never reached the checkout page, dilute the effect and are usually excluded.

Metrics are decided before the experiment starts. A primary metric measures what the change should improve, such as checkout conversion. Guardrail metrics measure what must not get worse, such as revenue per user, page load time or support contacts. Metrics should come from the same governed definitions as dashboards, so that the experiment's revenue is the business's revenue.

Sample size and duration follow from the smallest effect worth detecting, the metric's variation, and the accepted error rates. A power calculation gives the number of users per group needed; the experiment runs for at least that long, and for whole weeks, so that weekday and weekend behaviour are both included.

The analysis compares the groups with a statistical test and reports the effect with a confidence interval, not only a p-value. Two checks come first. A sample ratio mismatch test compares the actual group sizes with the planned split; a 50/50 experiment that ends at 51/49 with large numbers indicates a bug in assignment or logging, and the results cannot be trusted. And the data must be complete, including late events.

flowchart TB
  U[User reaches<br/>checkout] --> A{Hash of user and<br/>experiment ID}
  A -->|50%| C[Control:<br/>current checkout]
  A -->|50%| T[Treatment:<br/>new checkout]
  C --> L[(Exposure log)]
  T --> L
  L --> J[Join with orders<br/>and metrics]
  J --> R[Checks: sample ratio,<br/>completeness. Then<br/>effect and interval]

Common traps are well documented. Peeking: checking results daily and stopping when they look significant inflates false positives unless a sequential method designed for it is used. Many metrics: testing twenty metrics at the 5% level yields about one false positive on average. Novelty effects: users react to anything new for a while. Interference: when treatment users affect control users, as in marketplaces with shared stock.

Key terms¶

Control and treatment: the groups that see the current and the changed version.
Randomisation unit: what is assigned to a group, such as a user.
Exposure: the moment a unit actually experiences its variant.
Primary and guardrail metrics: what should improve, and what must not get worse.
Power: the probability of detecting an effect of a given size if it exists.
Sample ratio mismatch (SRM): a difference between planned and actual group sizes that signals a bug.
Peeking: repeatedly checking results and stopping early.

Example¶

The shop tests a one-page checkout against the current three-step one.

Unit: logged-in user; variant from a hash of user_id and experiment ID; exposure logged when the checkout page loads.
Primary metric: checkout conversion, from the semantic layer. Guardrails: revenue per exposed user, refund rate, page load time.
Power calculation: to detect a 1-point lift on a 30% conversion rate with 80% power at the 5% significance level, about 33,000 users per group; at current traffic, 21 days.
After 21 days: 33,210 control and 33,090 treatment users, consistent with 50/50. Conversion 30.1% vs 31.3%, a lift of 1.2 points with a 95% interval of 0.5 to 1.9 points. Guardrails unchanged.
Decision: roll out to 95% and keep 5% on the old checkout as a holdback for four weeks, to confirm the lift persists once the novelty wears off. After full rollout there is no control group, so a fading lift could not be told apart from seasonality.

Pitfalls¶

Logging allocation instead of exposure. Effects are diluted by users who never saw the change.
Stopping when results look good. Decide the duration in advance or use a sequential method.
Ignoring sample ratio mismatch. A broken split invalidates every number.
Metrics defined differently from the business. The experiment proves a gain the finance report never sees.
Too many metrics without correction. Something will always look significant.

Learn more¶

Ron Kohavi, Diane Tang and Ya Xu, Trustworthy Online Controlled Experiments (Cambridge University Press, 2020).
Fabijan et al., Diagnosing Sample Ratio Mismatch in Online Controlled Experiments: A Taxonomy and Rules of Thumb for Practitioners, KDD 2019: https://www.microsoft.com/en-us/research/?p=651990

Semantic layer · Late and out-of-order data · Data quality dimensions · Model serving and monitoring