Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Conversion Rate Optimisation

Experimentation

A controlled experiment is the only method that shows what a change caused rather than what happened around it. This hub introduces the core guides in the section (the procedure for running an A/B test, the statistics that decide how long it runs and what the result means, and the pitfalls that make results look wrong) and further guides on conversion work without a test, evaluating a redesign, shipping a winning variant, choosing between client-side and server-side testing and hiring a CRO agency. It sits between the research that produces hypotheses and the funnel and landing page work that supplies most of the things worth testing.
SectionintermediateUpdated

Learn this as a path

Prefer a guided order? These learning paths read pages from this section in sequence, with exercises and checkpoints.

In this section

What a controlled experiment gives you

Most conversion changes are judged by before-and-after comparison: the page changed on a Tuesday, the conversion rate (CVR) moved the following week, and the change gets the credit. The problem is everything else that moved in the same week — a promotion, a new traffic source, a competitor's stockout, a tracking fix, the weather. Before-and-after cannot separate the change from its context, and a team that relies on it ends up with a list of "wins" that never show in the annual figures.

A controlled experiment solves this by running the old and the new at the same time, on visitors assigned at random. Both groups see the same promotions, the same traffic mix and the same tracking, so the only systematic difference between them is the change under test. Whatever gap remains in the metric is either the effect of the change or chance, and statistics exists to tell those two apart.

That is what the method gives you: a causal answer and an honest statement of its uncertainty. It also gives you a record. A programme of experiments, each with a hypothesis, a plan and a documented result, becomes institutional knowledge about what this audience responds to, which is worth more than any single win. The same logic applied to whole channels rather than page elements is Incrementality measurement, covered in the incrementality testing guide; the designs differ, the reasoning is the same.

Test types

Test typeWhat is comparedBest forWatch out for
A/B testOne control against one variant, visitors split at randomMost page and flow changes; the default designNeeds enough traffic to detect the effect worth finding
A/B/n testOne control against several variantsChoosing between distinct approaches to one elementEach extra arm lengthens the test and adds a comparison
Multivariate testCombinations of changes to several elements at onceLearning which elements interact, on high-traffic pagesTraffic needed grows with every combination
Split URL testTwo separately built pages, traffic redirected between themRedesigns and structural changes too large for in-page editsRedirect latency; tracking the redirect correctly
Staged rolloutA change exposed to a growing share of usersEngineering releases where a full test is impracticalNot a controlled comparison unless the groups are random and concurrent
HoldoutA group kept away from a feature or channel for a periodMeasuring the cumulative value of something already liveLong duration; the holdout group forgoes the benefit

The A/B test is the workhorse because it is the easiest to run correctly and read. Multivariate designs are tempting and rarely justified: the number of visitors needed grows with every combination, and the question they answer best, which elements interact, is seldom the question a team actually has.

A/B testing, statistics, pitfalls: the core guides

  • A/B testing guide — the procedure from hypothesis to documented result: choosing the unit of randomisation, the primary metric and guardrails, allocation, QA before launch, running to the planned sample and reading what comes out. It includes an interactive calculator for comparing two variants.
  • Sample size and statistical significance — the statistics in plain language: baseline rate and minimum detectable effect, why sample size has to be fixed in advance, what significance and power actually mean, confidence intervals, and why peeking at a running test ruins it.
  • Common testing pitfalls — a diagnosis guide for results that look wrong: sample ratio mismatch, peeking, flicker, novelty effects, broken tracking, interacting tests and segment fishing, each with a check and a fix.

The hypotheses the tests are built on come from the research section, and the whole discipline is introduced in the conversion rate optimisation guide. Read the research section first if the backlog is empty; read the testing guides first if there are tests running that nobody trusts.

Further guides

Further guides cover the work either side of a split test: deciding when one is the wrong tool, judging a change too large to test as one element, putting a winner live properly, choosing where tests run and buying the work from outside:

  • Conversion work without an A/B test — when an A/B test is the wrong tool and what to do instead: changes to just ship, low traffic, template tests, bolder changes and honest before-and-after reads.
  • Evaluating a website redesign — evolve or relaunch, recording what already works, testing key templates, rolling out against a comparison group and reading the result fairly.
  • Shipping a winning test variant — a tutorial for the work after a test wins: confirming the decision, rebuilding the variant properly, checking tracking, logging the release, watching the metric and retiring the test.
  • Choosing between client-side and server-side testing — where each test should run, in the browser, on the server or as separate pages, by what each costs in speed, flicker, build effort and shipping, and who decides.
  • Hiring and evaluating a CRO agency — what the business keeps, the questions to ask, the warning signs, reports that can be checked and how winners reach the site.

Building the practice

A testing practice is a set of habits more than a tool, and the habits are few.

A written plan before every launch. Hypothesis, primary metric, guardrail metrics, the effect the test is sized to detect, the sample it needs and the date it will reach that sample. Decisions about stopping are made once, before the data exists, which is the whole point. Evan Miller's essay on how not to run an A/B test remains the clearest account of why stopping when the result looks good produces false wins.

A fixed primary KPI. One metric decides the test. Secondary metrics explain the result; they do not overturn it after the fact.

QA as a gate, not a formality. The variant is checked on real devices, the tracking is confirmed to fire once for each arm, and the allocation is verified before the test counts.

A results log. Every test — won, lost or inconclusive — is recorded with its plan, its data and the decision taken. Inconclusive results are the majority in most programmes and are not failures; a programme that only records wins learns nothing.

A cadence. Tests queued, tests running and tests being read, reviewed on a schedule with the people who own the pages. The research section covers how the backlog is fed and prioritised, and the guides in this section cover what happens to a test once it leaves the backlog.

Illustrative example — not a real result

A meal-kit company's first quarter of testing

A meal-kit company with roughly 180,000 monthly visitors to its plans page starts a programme with three tests a month. The first test, a reordered benefits section, is planned for 60,000 visitors per arm and runs for 19 days before reaching it; the variant converts at 4.1% against a control of 3.9%, a difference the confidence interval cannot separate from zero, and the result is logged as inconclusive. The second, a pricing table that shows the per-serving price first, reaches its planned sample in 16 days and converts at 4.6% against 3.9%, with an interval that excludes zero; it is rolled out. The third is stopped during QA because the variant's conversion tag fired twice on the confirmation page. One win, one null, one caught bug: a normal quarter, and a log that already shows the team what its audience cares about.

Common testing problems

SymptomUsual causeGuide
The result flips from winning to losing and back as the test runsReading an unfinished test; peekingSample size and significance
A test has run for weeks and is still inconclusiveEffect smaller than the test can detect at this trafficSample size and significance
The two arms have visibly unequal visitor countsSample ratio mismatch from redirects, bots or tag timingCommon testing pitfalls
The variant won, was rolled out, and the metric did not moveNovelty effect, segment fishing or a broken conversion tag during the testCommon testing pitfalls
Nobody can say what the test was supposed to proveNo written hypothesis or planA/B testing guide, research section
Two tests on the same journey disagreeInteracting tests sharing trafficCommon testing pitfalls
The win is real but small and nobody knows whether to ship itNo minimum detectable effect agreed in advanceA/B testing guide
A planned test would take longer than anyone will waitA small change on a page with too little traffic, or a fix that should simply shipConversion work without an A/B test
A redesign has launched and nobody can say whether it helpedNothing recorded about what converted before; no comparison groupEvaluating a website redesign
A winning variant is still served by the testing tool long after the test endedThe variant never rebuilt in the site's own code; the test never retiredShipping a winning test variant
Visitors see the old page flash before the variant, or every page slows while a test runsA client-side script applying the change after the page rendersChoosing between client-side and server-side testing
An agency reports many winning tests and sales do not moveTests read early, wins reported as single figures, or winners never shippedHiring and evaluating a CRO agency

Most of these are procedural, which is the encouraging part: a team that writes the plan, checks the tracking and reads once at the planned sample avoids nearly all of them without learning any new mathematics.

Terms used on this page

See it applied

Sources

  1. How Not To Run an A/B Test — Evan Miller

Related