What a controlled experiment gives you
Most conversion changes are judged by before-and-after comparison: the page changed on a Tuesday, the conversion rate (CVRShare of visits or clicks that convert.) moved the following week, and the change gets the credit. The problem is everything else that moved in the same week — a promotion, a new traffic source, a competitor's stockout, a tracking fix, the weather. Before-and-after cannot separate the change from its context, and a team that relies on it ends up with a list of "wins" that never show in the annual figures.
A controlled experiment solves this by running the old and the new at the same time, on visitors assigned at random. Both groups see the same promotions, the same traffic mix and the same tracking, so the only systematic difference between them is the change under test. Whatever gap remains in the metric is either the effect of the change or chance, and statistics exists to tell those two apart.
That is what the method gives you: a causal answer and an honest statement of its uncertainty. It also gives you a record. A programme of experiments, each with a hypothesis, a plan and a documented result, becomes institutional knowledge about what this audience responds to, which is worth more than any single win. The same logic applied to whole channels rather than page elements is IncrementalityConversions caused by marketing. measurement, covered in the incrementality testing guide; the designs differ, the reasoning is the same.
Test types
| Test type | What is compared | Best for | Watch out for |
|---|---|---|---|
| A/B test | One control against one variant, visitors split at random | Most page and flow changes; the default design | Needs enough traffic to detect the effect worth finding |
| A/B/n test | One control against several variants | Choosing between distinct approaches to one element | Each extra arm lengthens the test and adds a comparison |
| Multivariate test | Combinations of changes to several elements at once | Learning which elements interact, on high-traffic pages | Traffic needed grows with every combination |
| Split URL test | Two separately built pages, traffic redirected between them | Redesigns and structural changes too large for in-page edits | Redirect latency; tracking the redirect correctly |
| Staged rollout | A change exposed to a growing share of users | Engineering releases where a full test is impractical | Not a controlled comparison unless the groups are random and concurrent |
| Holdout | A group kept away from a feature or channel for a period | Measuring the cumulative value of something already live | Long duration; the holdout group forgoes the benefit |
The A/B test is the workhorse because it is the easiest to run correctly and read. MultivariateAn experiment that tests several elements at once by measuring every combination of their versions. designs are tempting and rarely justified: the number of visitors needed grows with every combination, and the question they answer best, which elements interact, is seldom the question a team actually has.
A/B testing, statistics, pitfalls: the core guides
- A/B testing guide — the procedure from hypothesis to documented result: choosing the unit of randomisation, the primary metric and guardrails, allocation, QA before launch, running to the planned sample and reading what comes out. It includes an interactive calculator for comparing two variants.
- Sample size and statistical significance — the statistics in plain language: baseline rate and minimum detectable effect, why sample sizeThe number of users or sessions an experiment needs to detect a given effect reliably. has to be fixed in advance, what significanceA test result unlikely enough under 'no real difference' to pass a threshold chosen in advance. and power actually mean, confidence intervalsA range of values for an effect, such as a lift in conversion rate, that is consistent with the data at a chosen confidence level., and why peeking at a running test ruins it.
- Common testing pitfalls — a diagnosis guide for results that look wrong: sample ratio mismatch, peeking, flicker, novelty effectsA temporary change in behaviour caused by a change being new, which fades as users get used to it., broken tracking, interacting tests and segment fishing, each with a check and a fix.
The hypotheses the tests are built on come from the research section, and the whole discipline is introduced in the conversion rate optimisation guide. Read the research section first if the backlog is empty; read the testing guides first if there are tests running that nobody trusts.
Further guides
Further guides cover the work either side of a split test: deciding when one is the wrong tool, judging a change too large to test as one element, putting a winner live properly, choosing where tests run and buying the work from outside:
- Conversion work without an A/B test — when an A/B test is the wrong tool and what to do instead: changes to just ship, low traffic, template tests, bolder changes and honest before-and-after reads.
- Evaluating a website redesign — evolve or relaunch, recording what already works, testing key templates, rolling out against a comparison group and reading the result fairly.
- Shipping a winning test variant — a tutorial for the work after a test wins: confirming the decision, rebuilding the variant properly, checking tracking, logging the release, watching the metric and retiring the test.
- Choosing between client-side and server-side testing — where each test should run, in the browser, on the server or as separate pages, by what each costs in speed, flicker, build effort and shipping, and who decides.
- Hiring and evaluating a CRO agency — what the business keeps, the questions to ask, the warning signs, reports that can be checked and how winners reach the site.
Building the practice
A testing practice is a set of habits more than a tool, and the habits are few.
A written plan before every launch. Hypothesis, primary metric, guardrail metrics, the effect the test is sized to detect, the sample it needs and the date it will reach that sample. Decisions about stopping are made once, before the data exists, which is the whole point. Evan Miller's essay on how not to run an A/B test remains the clearest account of why stopping when the result looks good produces false wins.
A fixed primary KPIA metric tied directly to an objective.. One metric decides the test. Secondary metrics explain the result; they do not overturn it after the fact.
QA as a gate, not a formality. The variant is checked on real devices, the tracking is confirmed to fire once for each arm, and the allocation is verified before the test counts.
A results log. Every test — won, lost or inconclusive — is recorded with its plan, its data and the decision taken. Inconclusive results are the majority in most programmes and are not failures; a programme that only records wins learns nothing.
A cadence. Tests queued, tests running and tests being read, reviewed on a schedule with the people who own the pages. The research section covers how the backlog is fed and prioritised, and the guides in this section cover what happens to a test once it leaves the backlog.
A meal-kit company's first quarter of testing
A meal-kit company with roughly 180,000 monthly visitors to its plans page starts a programme with three tests a month. The first test, a reordered benefits section, is planned for 60,000 visitors per arm and runs for 19 days before reaching it; the variant converts at 4.1% against a control of 3.9%, a difference the confidence interval cannot separate from zero, and the result is logged as inconclusive. The second, a pricing table that shows the per-serving price first, reaches its planned sample in 16 days and converts at 4.6% against 3.9%, with an interval that excludes zero; it is rolled out. The third is stopped during QA because the variant's conversion tag fired twice on the confirmation page. One win, one null, one caught bug: a normal quarter, and a log that already shows the team what its audience cares about.
Common testing problems
| Symptom | Usual cause | Guide |
|---|---|---|
| The result flips from winning to losing and back as the test runs | Reading an unfinished test; peeking | Sample size and significance |
| A test has run for weeks and is still inconclusive | Effect smaller than the test can detect at this traffic | Sample size and significance |
| The two arms have visibly unequal visitor counts | Sample ratio mismatch from redirects, bots or tag timing | Common testing pitfalls |
| The variant won, was rolled out, and the metric did not move | Novelty effect, segment fishing or a broken conversion tag during the test | Common testing pitfalls |
| Nobody can say what the test was supposed to prove | No written hypothesis or plan | A/B testing guide, research section |
| Two tests on the same journey disagree | Interacting tests sharing traffic | Common testing pitfalls |
| The win is real but small and nobody knows whether to ship it | No minimum detectable effect agreed in advance | A/B testing guide |
| A planned test would take longer than anyone will wait | A small change on a page with too little traffic, or a fix that should simply ship | Conversion work without an A/B test |
| A redesign has launched and nobody can say whether it helped | Nothing recorded about what converted before; no comparison group | Evaluating a website redesign |
| A winning variant is still served by the testing tool long after the test ended | The variant never rebuilt in the site's own code; the test never retired | Shipping a winning test variant |
| Visitors see the old page flash before the variant, or every page slows while a test runs | A client-side script applying the change after the page renders | Choosing between client-side and server-side testing |
| An agency reports many winning tests and sales do not move | Tests read early, wins reported as single figures, or winners never shipped | Hiring and evaluating a CRO agency |
Most of these are procedural, which is the encouraging part: a team that writes the plan, checks the tracking and reads once at the planned sample avoids nearly all of them without learning any new mathematics.
