Definition
An A/B test, also called a split test, is a randomised controlled experiment with two arms: a control that keeps the existing version and a variant that changes one variable. Each unit, usually a visitor or a user, is assigned to an arm at random, the outcome of interest is measured for both arms over the same period, and a statistical test asks whether the observed difference is larger than random variation would produce. The registry's one-line summary lists the requirements: one variable, adequate sample size, and significance testing. Tests with several variants of the same element are A/B/n tests and follow the same logic; tests that change several elements in combination are multivariate and need far more traffic to read.
How it works
The test begins with a hypothesis: a specific change is expected to move a specific metric for a stated reason. The primary metric is chosen before launch, most often a conversion rate (CVRShare of visits or clicks that convert.), together with guardrail metrics that must not get worse. The minimum effect worth detecting and the baseline rate determine the sample size, and the sample size and traffic determine the duration, which should cover full weekly cycles. Assignment is random and sticky, so a visitor sees the same version on every return. At the end, the result is read once: the difference between arms, its confidence interval and whether it clears the significance threshold set in advance. Evan Miller's "How Not To Run an A/B Test" shows why peeking matters: checking for significance repeatedly and stopping when it appears inflates the false positive rate well beyond the nominal level, because every look is another chance for noise to cross the line.
| Ingredient | What it fixes | What goes wrong without it |
|---|---|---|
| One variable | The explanation for any difference | Several changes, no way to say which one acted |
| Random, sticky assignment | Comparable groups | Groups differ by device, time or source rather than by version |
| Pre-registered metric | The question being asked | The metric that happens to move is reported |
| Sample size set in advance | Power to detect the effect | Underpowered tests that read as "no difference" |
| Fixed duration and single readout | Honest significance | Peeking and early stopping produce false winners |
In practice
The A/B testing guide walks through unit, metric, guardrails, randomisation, QA and reading the result; the sample size and significance guide explains baseline rate, minimum detectable effect, power and confidence intervals in plain language. In paid social the same design is applied to creative, where the creative testing guide describes the platform's split tests and their limits. Whatever the surface, the discipline is the same: write the hypothesis down, compute the sample before launch, do not look until it is reached, and record the result whether it won or lost.
A test that was stopped too early
A checkout page converts at 3.0 percent. A team tests a shorter form, wanting to detect a relative lift of 10 percent, and a calculator says each arm needs about 50,000 visitors. After four days, with 8,000 visitors per arm, the variant shows 3.5 percent against 2.9 percent and a tool reports 95 percent significance. The team ships it. Over the next six weeks the overall rate drifts back to 3.0 percent. A rerun to the planned sample finds the variant at 3.1 percent against 3.0 percent, inside the noise. The early reading was one of the false positives that repeated peeking makes likely; the planned sample would have cost two more weeks and saved a false conclusion.
What it is not
An A/B test is not a before-and-after comparison; changing a page and comparing this month with last month confounds the change with seasonality, campaigns and everything else that moved. It is not a platform's "optimise for the best performer" setting, which reallocates traffic as it goes and does not produce a clean comparison. It is not an incrementality test (IncrementalityConversions caused by marketing.), although both are randomised experiments: the incrementality design withholds marketing from a group to measure what marketing caused, while an A/B test compares two versions of something everyone receives. And a statistically significant result is not a business result until the effect size is weighed against the cost of the change and the risk of being wrong.
Related terms
Conversion rate (CVRShare of visits or clicks that convert.) is the most common primary metric, and a key performance indicator (KPIA metric tied directly to an objective.) framework decides which outcomes qualify as primary and which as guardrails. Incrementality (IncrementalityConversions caused by marketing.) applies the same randomised logic to the question of whether marketing works at all, with exposure to advertising itself as the single variable. Terms such as statistical power, minimum detectable effect, confidence interval and sequential testing are explained in the sample size guide linked above rather than here; the one to remember is power, because an underpowered test can only ever report that it saw nothing, which is not the same as there being nothing to see, and the two are confused in most test reports that reach decision-makers.
Common mistakes
- Stopping when significance first appears, the error Evan Miller's article is about, which turns a nominal error rate into a much larger one without anyone noticing.
- Changing more than one thing and attributing the result to the one that was intended, when the test cannot say which change acted.
- Choosing the metric after the fact from whichever moved, which guarantees a winner and proves nothing.
- Running for less than a full business cycle, so weekday and weekend traffic are unevenly represented across arms and the result describes a few days, not the audience.
- Treating a non-significant result as "no effect" when the test was simply too small to see one, which quietly kills changes that would have worked.
