Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Conversion Rate Optimisation › Experimentation

A/B Testing Guide

How to run an A/B test properly: choosing the unit and metric, randomisation and allocation, QA before launch, and running to the planned sample.
Guideintermediate12 min readUpdated

What an A/B test is

An A/B test is a randomised controlled experiment applied to a website, an app or an email. Visitors who qualify for the test are assigned, at random and before they see anything, to the control (the current experience) or the variant (the proposed change). Each group is measured on the same metric over the same period. Because assignment was random, the groups are alike in every respect except the change, so a difference in the metric that is too large to be chance is attributed to the change.

Three things make it a test rather than a comparison. Randomisation: the assignment is made by a process the visitor and the team cannot influence, so the groups do not differ systematically. Concurrency: both experiences run at the same time, so seasonality, campaigns and site incidents hit both groups equally. A plan written in advance: the metric, the sample and the decision rule are fixed before launch, so the result cannot be steered after the fact.

What it is not: a way to find out what visitors want in general. A test answers one narrow question — does this change move this metric for this audience — and answers it well. The question itself has to come from research; the hypothesis-writing tutorial covers how a finding becomes a testable statement, and the experimentation hub places the A/B test among the other designs and the habits that surround it. The same discipline applies to ad copy drafted with AI, where the AI-assisted ad copy testing guide turns a hypothesis into a brief and a small set of purposeful variants. When a change has to ship to everyone and cannot be split, the comparison of control charts and A/B testing explains how a chart of the metric over time tells a real shift from noise, and why it cannot prove cause.

Unit, metric and guardrails

Three decisions shape every test and should be made in this order.

The unit of randomisation is the thing that gets assigned to a group. Usually it is the visitor, identified by a cookie or a login, so that a person sees the same experience on every visit. Sometimes it is the session, which is simpler but lets one person see both versions across visits. For account-based products it may be the account, so that colleagues see the same thing. The unit must match the metric: a visitor-level metric needs visitor-level assignment, or the statistics are computed on the wrong denominator.

The primary metric is the single number that decides the test. It should be as close to the business outcome as the traffic allows: purchases for a shop, qualified leads for a lead-generation site, started trials for software. A metric earlier in the journey — clicks on a button, form starts — is more sensitive but may not reflect the outcome, and a variant can win on clicks while losing on sales.

Guardrails are metrics that must not get worse. Revenue per visitor when testing a discount prompt; average order value when testing a free-delivery threshold; refund rate, support contacts, page speed, error rates. A variant that wins the primary metric and breaches a guardrail is not shipped without a separate decision.

DecisionOptionsChoose byTypical mistake
Unit of randomisationVisitor, session, account, deviceThe level at which the metric is counted and the experience must stay consistentSession-level assignment for a purchase that takes several visits
Primary metricConversion rate (CVR), revenue per visitor, lead rate, trial startsClosest to the business outcome the traffic can detectA proxy metric chosen because it moves faster
GuardrailsOrder value, refund rate, speed, errors, support contacts, downstream conversionWhat the change could plausibly harmNone declared, so harm is discovered after rollout
AudienceAll visitors, new only, one device, one source, one countryWho the hypothesis is aboutIncluding visitors the change cannot affect, which dilutes the effect
Minimum detectable effectThe smallest improvement worth shippingBusiness value of the change against the cost of running longerSized for a large effect that was never realistic
DurationFull weeks until the planned sample is reachedWeekly cycles in behaviour and the sample-size planStopped on a round number of days

The sample-size row depends on the baseline rate and the minimum detectable effect, and that arithmetic belongs to the sample size and significance guide. What matters here is that the number exists before launch and that the plan names the date it should be reached.

For store tests, the comparison of AOV and revenue per visitor shows why a change that lifts order value can still lose revenue.

Randomisation and allocation

Randomisation is done by the testing tool or by the application, and the mechanism should be understood before it is trusted. Client-side tools assign in the browser, usually by hashing a cookie value, and apply the variant by modifying the page after it loads, which is where flicker comes from. Server-side assignment happens before the page is rendered, so there is nothing to flicker, but it needs engineering time. Either way the assignment must be sticky: the same unit gets the same experience for the life of the test.

Allocation is the share of traffic each arm receives. An even split reaches the planned sample fastest and makes the arithmetic simplest. Uneven splits — a small share to a risky variant — are legitimate when the downside is large, but they lengthen the test, because the smaller arm sets the pace. Changing the allocation while a test runs is a mistake: visitors assigned under the old split and the new one differ in time, and time is confounded with everything else.

Two checks confirm the mechanism works. The first is a sample ratio check: the observed split should match the intended one closely, and a persistent gap is a defect in assignment, tracking or exclusion, not a curiosity. The second is an A/A test, where both arms show the identical experience; any apparent difference is noise, and the exercise shows what noise looks like on this site and whether the tool reports it honestly. Teams new to a tool should run one before the first real test.

Exclusions deserve a deliberate decision. Internal traffic, bots, and visitors who cannot be affected by the change (a checkout test should not include people who never reach the checkout) are excluded before assignment, not filtered out afterwards.

QA before launch

QA is where the hypothesis meets reality, and it has three parts.

The experience. Every variant is viewed as a visitor would see it on the devices and browsers that carry most of the traffic, including the slowest. The check is for correctness — the change is present, and only the change — and for flicker, the brief flash of the control before the variant is applied. Flicker is a real effect on behaviour, not a cosmetic issue, and a variant that flickers is testing "flicker plus the change" against "no flicker". Where flicker was common during a run, the result is not read as it stands: the cause is fixed and the test restarted.

The tracking. The primary metric and every guardrail must be recorded identically for both arms. Open the tag debugger, walk the conversion path in each arm, and confirm the event fires once, with the same parameters, and that the test assignment is attached to it so the analysis can split by arm. A conversion tag that fires twice in one arm, or not at all on a variant's redesigned confirmation page, is the most common way a test produces a confident wrong answer. The testing pitfalls guide lists the symptoms this produces and the checks that find it.

The plan. The written document — hypothesis, audience, metric, guardrails, sample, duration, decision rule — is reviewed by someone who did not write it. The question to ask is whether, if the result comes back in each possible direction, the team already knows what it will do.

Only when all three pass does the test start counting. A soft launch to a small share of traffic for a day, to confirm events arrive from real visitors, is cheap insurance. The A/B test launch checklist turns these three parts, and the split check on the first day, into a list to tick before any test is switched on.

Running the test

Once launched, the job is to leave it alone and watch for defects. The two are different: a test should not be read for a decision before it reaches its planned sample, but it should be monitored for problems from the first hour.

Monitor the sample ratio daily. Monitor error rates, page speed and the guardrails, because a variant that breaks something must be stopped immediately regardless of the primary metric. Check that conversions are arriving from both arms in plausible proportions. Note anything that happens on the site or in the market during the test — a campaign launch, a price change, an outage — in the test log, because it may need to be accounted for when reading.

Do not read the primary metric as a result until the plan says so. Evan Miller's essay on how not to run an A/B test shows what happens otherwise: a test that is checked repeatedly and stopped the first time it looks significant will declare false winners far more often than the nominal error rate suggests, because the researcher has given chance many opportunities to cross the line. The sample size and significance guide explains the mechanism and the sequential methods that allow early looks when they are planned for.

Run in whole weeks. Behaviour differs by day of the week, and a test that starts on a Monday and stops on a Thursday has over-represented weekdays. If the planned sample is reached mid-week, run to the end of the week.

Do not change the variant during the test. A fix to a bug in the variant restarts the test; the data before the fix measured something else.

Reading the result

Reading starts with the plan. Was the planned sample reached? Did the sample ratio hold? Were there any incidents? If any of those fails, the result is at best indicative.

Then the primary metric. The question is not whether the variant's rate is higher — it will differ in one direction or the other by chance alone — but whether the difference is large enough, given the sample, that chance is an implausible explanation, and whether the plausible range of the true effect includes the minimum worth shipping. A significant result with a tiny effect may not be worth the engineering; a non-significant result with an interval that spans both harm and a large gain means the test did not have the traffic to answer, and the honest label is inconclusive rather than "no effect".

Then the guardrails. A win on the primary metric with a breach on a guardrail is a decision for the business, not an automatic rollout. To read the primary metric and the guardrails for each arm in Google Analytics as well as in the testing tool, the tutorial on analysing A/B tests in GA4 sends an exposure event with the test and variant and builds the comparison in explorations. The exposure event is what lets analytics count each arm independently of the testing tool.

Then, carefully, the segments. Looking at the result by device, source or new-versus-returning is useful for generating the next hypothesis and dangerous for rescuing a lost test. With enough segments, one will look like a winner by chance. A segment finding is a hypothesis to test next, not a result.

The calculator below performs the comparison most teams need. Enter the visitors and conversions for each arm, and it reports each arm's conversion rate, the relative difference between them, and whether that difference clears the conventional significance threshold under a two-sided test. A reading that does not clear the threshold means keep running to the planned sample or accept that the test is inconclusive; it never means the variant lost. The conversion rate calculator is the plainer tool for turning counts into rates when there is no comparison to make.

A/B test calculator

Illustrative example — not a real result

Reading a checkout button test

An outdoor-equipment retailer tests a redesigned checkout button on its basket page. The plan calls for 24,000 visitors per arm to detect a relative lift of 10% on a baseline basket-to-order rate of 31%, with the test running for at least two full weeks. After 15 days the control has 24,310 visitors and 7,560 orders (31.1%), and the variant 24,180 visitors and 8,010 orders (33.1%). The sample ratio is within a fraction of a percent of even. The two-sided test gives a p-value well under 0.05, and the confidence interval on the relative lift runs from roughly 3% to 10%, entirely above zero and touching the 10% the team hoped for only at its upper edge. Average order value is unchanged at £84 in both arms and the refund guardrail is flat. The decision is to ship: a real improvement, probably smaller than hoped, with no guardrail cost.

Documenting and sharing

A test that is not documented did not happen, as far as the next team is concerned. The record should hold the hypothesis and the research that produced it, screenshots of each arm, the plan with its sample and decision rule, the dates, the data per arm for the primary metric and every guardrail, the analysis with its interval, the segments examined, any incidents, and the decision with its reasoning. The experiment results report template sets those fields out as a one-page readout, in this order, with a filled example. Inconclusive tests are documented with the same care; they are the majority of results in most programmes and they tell the next person which directions have been tried at what sample.

Sharing is a separate act. A short write-up for stakeholders states what was tested, what was found, and what will change, in that order, with the uncertainty stated plainly. A win is "the variant improved the rate by somewhere between the low and high ends of the interval", not a single headline figure. The same discipline applies on paid platforms: the creative testing guide for Meta covers tests where the platform assigns the groups, and the reading rules there are the same as here. Over time the log becomes the programme's most valuable asset: a map of what this audience has responded to, at what size, and what it has ignored.

Before committing traffic, use the conversion rate calculator for landing page tests to turn the planned minimum effect into the extra conversions a winning variant would be worth.

Common mistakes

  • Choosing the metric after the data arrives. The primary KPI is fixed in the plan; anything chosen later is a story about the data.
  • Session-level assignment for multi-visit decisions. The same person sees both versions and the measurement is of neither.
  • Skipping the tracking walk. A tag that fires differently in each arm manufactures a result.
  • Stopping when it looks good. The error rate climbs with every unplanned look.
  • Changing allocation or the variant mid-test. Time becomes a confound; restart.
  • Rescuing a lost test with segments. One segment will always look like a winner; test it next instead.
  • Treating inconclusive as "no effect". A wide interval means the test could not tell, which is a statement about traffic, not about the change.
  • Not writing it down. The next test will repeat this one.

Frequently asked questions

How long should an A/B test run?

Until it reaches the sample planned before launch, rounded up to whole weeks so that every day of the week is represented at least once. Duration is a consequence of traffic, baseline rate and the effect worth detecting, not a fixed number of days, and a test stopped early because it looked good is the most common source of false wins.

Can I run more than one A/B test at the same time?

Yes, if the tests touch different pages or different parts of the journey and the assignment is independent, so that a visitor's arm in one test tells you nothing about their arm in the other. Two tests on the same step, or on consecutive steps of one journey, interact: run them as one test with combined arms, or run them one after the other.

What should I do when the variant wins on clicks but not on purchases?

Trust the metric closest to the business outcome, which is why the primary metric is chosen before launch. A click-level win with a flat purchase rate usually means the change moved attention without moving decisions; record it as a finding about attention and write the next hypothesis about what happens after the click.

Terms used on this page

Tools to try

Where this page is used

Part of these learning paths

Also related

Sources

  1. How Not To Run an A/B Test — Evan Miller

Related