What an A/B test is
An A/B test is a randomised controlled experiment applied to a website, an app or an email. Visitors who qualify for the test are assigned, at random and before they see anything, to the control (the current experience) or the variant (the proposed change). Each group is measured on the same metric over the same period. Because assignment was random, the groups are alike in every respect except the change, so a difference in the metric that is too large to be chance is attributed to the change.
Three things make it a test rather than a comparison. Randomisation: the assignment is made by a process the visitor and the team cannot influence, so the groups do not differ systematically. Concurrency: both experiences run at the same time, so seasonality, campaigns and site incidents hit both groups equally. A plan written in advance: the metric, the sample and the decision rule are fixed before launch, so the result cannot be steered after the fact.
What it is not: a way to find out what visitors want in general. A test answers one narrow question — does this change move this metric for this audience — and answers it well. The question itself has to come from research; the hypothesis-writing tutorial covers how a finding becomes a testable statement, and the experimentation hub places the A/B test among the other designs and the habits that surround it. The same discipline applies to ad copy drafted with AI, where the AI-assisted ad copy testing guide turns a hypothesis into a brief and a small set of purposeful variants. When a change has to ship to everyone and cannot be split, the comparison of control charts and A/B testing explains how a chart of the metric over time tells a real shift from noise, and why it cannot prove cause.
Unit, metric and guardrails
Three decisions shape every test and should be made in this order.
The unit of randomisation is the thing that gets assigned to a group. Usually it is the visitor, identified by a cookie or a login, so that a person sees the same experience on every visit. Sometimes it is the session, which is simpler but lets one person see both versions across visits. For account-based products it may be the account, so that colleagues see the same thing. The unit must match the metric: a visitor-level metric needs visitor-level assignment, or the statistics are computed on the wrong denominator.
The primary metric is the single number that decides the test. It should be as close to the business outcome as the traffic allows: purchases for a shop, qualified leads for a lead-generation site, started trials for software. A metric earlier in the journey — clicks on a button, form starts — is more sensitive but may not reflect the outcome, and a variant can win on clicks while losing on sales.
Guardrails are metrics that must not get worse. Revenue per visitorRevenue divided by the number of visitors in the same period. when testing a discount prompt; average order value when testing a free-delivery threshold; refund rate, support contacts, page speed, error rates. A variant that wins the primary metric and breaches a guardrail is not shipped without a separate decision.
| Decision | Options | Choose by | Typical mistake |
|---|---|---|---|
| Unit of randomisation | Visitor, session, account, device | The level at which the metric is counted and the experience must stay consistent | Session-level assignment for a purchase that takes several visits |
| Primary metric | Conversion rate (CVRShare of visits or clicks that convert.), revenue per visitor, lead rate, trial starts | Closest to the business outcome the traffic can detect | A proxy metric chosen because it moves faster |
| Guardrails | Order value, refund rate, speed, errors, support contacts, downstream conversion | What the change could plausibly harm | None declared, so harm is discovered after rollout |
| Audience | All visitors, new only, one device, one source, one country | Who the hypothesis is about | Including visitors the change cannot affect, which dilutes the effect |
| Minimum detectable effect | The smallest improvement worth shipping | Business value of the change against the cost of running longer | Sized for a large effect that was never realistic |
| Duration | Full weeks until the planned sample is reached | Weekly cycles in behaviour and the sample-size plan | Stopped on a round number of days |
The sample-size row depends on the baseline rate and the minimum detectable effect, and that arithmetic belongs to the sample size and significance guide. What matters here is that the number exists before launch and that the plan names the date it should be reached.
For store tests, the comparison of AOV and revenue per visitor shows why a change that lifts order value can still lose revenue.
Randomisation and allocation
Randomisation is done by the testing tool or by the application, and the mechanism should be understood before it is trusted. Client-side tools assign in the browser, usually by hashing a cookie value, and apply the variant by modifying the page after it loads, which is where flicker comes from. Server-side assignment happens before the page is rendered, so there is nothing to flicker, but it needs engineering time. Either way the assignment must be sticky: the same unit gets the same experience for the life of the test.
Allocation is the share of traffic each arm receives. An even split reaches the planned sample fastest and makes the arithmetic simplest. Uneven splits — a small share to a risky variant — are legitimate when the downside is large, but they lengthen the test, because the smaller arm sets the pace. Changing the allocation while a test runs is a mistake: visitors assigned under the old split and the new one differ in time, and time is confounded with everything else.
Two checks confirm the mechanism works. The first is a sample ratio checkA persistent gap between the share of visitors an A/B test was meant to assign to each variant and the share each actually received.: the observed split should match the intended one closely, and a persistent gap is a defect in assignment, tracking or exclusion, not a curiosity. The second is an A/A test, where both arms show the identical experience; any apparent difference is noise, and the exercise shows what noise looks like on this site and whether the tool reports it honestly. Teams new to a tool should run one before the first real test.
Exclusions deserve a deliberate decision. Internal traffic, bots, and visitors who cannot be affected by the change (a checkout test should not include people who never reach the checkout) are excluded before assignment, not filtered out afterwards.
QA before launch
QA is where the hypothesis meets reality, and it has three parts.
The experience. Every variant is viewed as a visitor would see it on the devices and browsers that carry most of the traffic, including the slowest. The check is for correctness — the change is present, and only the change — and for flicker, the brief flash of the control before the variant is applied. Flicker is a real effect on behaviour, not a cosmetic issue, and a variant that flickers is testing "flicker plus the change" against "no flicker". Where flickerThe brief flash of the original version before a browser-side test script applies the variant, which turns a result about the change into one about the fault. was common during a run, the result is not read as it stands: the cause is fixed and the test restarted.
The tracking. The primary metric and every guardrail must be recorded identically for both arms. Open the tag debugger, walk the conversion path in each arm, and confirm the event fires once, with the same parameters, and that the test assignment is attached to it so the analysis can split by arm. A conversion tag that fires twice in one arm, or not at all on a variant's redesigned confirmation page, is the most common way a test produces a confident wrong answer. The testing pitfalls guide lists the symptoms this produces and the checks that find it.
The plan. The written document — hypothesis, audience, metric, guardrails, sample, duration, decision rule — is reviewed by someone who did not write it. The question to ask is whether, if the result comes back in each possible direction, the team already knows what it will do.
Only when all three pass does the test start counting. A soft launch to a small share of traffic for a day, to confirm events arrive from real visitors, is cheap insurance. The A/B test launch checklist turns these three parts, and the split check on the first day, into a list to tick before any test is switched on.
Running the test
Once launched, the job is to leave it alone and watch for defects. The two are different: a test should not be read for a decision before it reaches its planned sample, but it should be monitored for problems from the first hour.
Monitor the sample ratio daily. Monitor error rates, page speed and the guardrails, because a variant that breaks something must be stopped immediately regardless of the primary metric. Check that conversions are arriving from both arms in plausible proportions. Note anything that happens on the site or in the market during the test — a campaign launch, a price change, an outage — in the test log, because it may need to be accounted for when reading.
Do not read the primary metric as a result until the plan says so. Evan Miller's essay on how not to run an A/B test shows what happens otherwise: a test that is checked repeatedly and stopped the first time it looks significant will declare false winners far more often than the nominal error rate suggests, because the researcher has given chance many opportunities to cross the line. The sample size and significance guide explains the mechanism and the sequential methods that allow early looks when they are planned for.
Run in whole weeks. Behaviour differs by day of the week, and a test that starts on a Monday and stops on a Thursday has over-represented weekdays. If the planned sample is reached mid-week, run to the end of the week.
Do not change the variant during the test. A fix to a bug in the variant restarts the test; the data before the fix measured something else.
Reading the result
Reading starts with the plan. Was the planned sample reached? Did the sample ratio hold? Were there any incidents? If any of those fails, the result is at best indicative.
Then the primary metric. The question is not whether the variant's rate is higher — it will differ in one direction or the other by chance alone — but whether the difference is large enough, given the sample, that chance is an implausible explanation, and whether the plausible range of the true effect includes the minimum worth shipping. A significant result with a tiny effect may not be worth the engineering; a non-significant result with an interval that spans both harm and a large gain means the test did not have the traffic to answer, and the honest label is inconclusive rather than "no effect".
Then the guardrails. A win on the primary metric with a breach on a guardrail is a decision for the business, not an automatic rollout. To read the primary metric and the guardrails for each arm in Google Analytics as well as in the testing tool, the tutorial on analysing A/B tests in GA4 sends an exposure event with the test and variant and builds the comparison in explorations. The exposure eventAn analytics event fired when a visitor sees a test arm, carrying the test and variant, so each arm's behaviour can be counted outside the testing tool. is what lets analytics count each arm independently of the testing tool.
Then, carefully, the segments. Looking at the result by device, source or new-versus-returning is useful for generating the next hypothesis and dangerous for rescuing a lost test. With enough segments, one will look like a winner by chance. A segment finding is a hypothesis to test next, not a result.
The calculator below performs the comparison most teams need. Enter the visitors and conversions for each arm, and it reports each arm's conversion rate, the relative difference between them, and whether that difference clears the conventional significance threshold under a two-sided test. A reading that does not clear the threshold means keep running to the planned sample or accept that the test is inconclusive; it never means the variant lost. The conversion rate calculator is the plainer tool for turning counts into rates when there is no comparison to make.
A/B test calculator
Reading a checkout button test
An outdoor-equipment retailer tests a redesigned checkout button on its basket page. The plan calls for 24,000 visitors per arm to detect a relative lift of 10% on a baseline basket-to-order rate of 31%, with the test running for at least two full weeks. After 15 days the control has 24,310 visitors and 7,560 orders (31.1%), and the variant 24,180 visitors and 8,010 orders (33.1%). The sample ratio is within a fraction of a percent of even. The two-sided test gives a p-value well under 0.05, and the confidence interval on the relative lift runs from roughly 3% to 10%, entirely above zero and touching the 10% the team hoped for only at its upper edge. Average order value is unchanged at £84 in both arms and the refund guardrail is flat. The decision is to ship: a real improvement, probably smaller than hoped, with no guardrail cost.
Documenting and sharing
A test that is not documented did not happen, as far as the next team is concerned. The record should hold the hypothesis and the research that produced it, screenshots of each arm, the plan with its sample and decision rule, the dates, the data per arm for the primary metric and every guardrail, the analysis with its interval, the segments examined, any incidents, and the decision with its reasoning. The experiment results report template sets those fields out as a one-page readoutThe meeting or short document where a study's or test's results are presented against the decision it was run for, ending in a recorded decision., in this order, with a filled example. Inconclusive tests are documented with the same care; they are the majority of results in most programmes and they tell the next person which directions have been tried at what sample.
Sharing is a separate act. A short write-up for stakeholders states what was tested, what was found, and what will change, in that order, with the uncertainty stated plainly. A win is "the variant improved the rate by somewhere between the low and high ends of the interval", not a single headline figure. The same discipline applies on paid platforms: the creative testing guide for Meta covers tests where the platform assigns the groups, and the reading rules there are the same as here. Over time the log becomes the programme's most valuable asset: a map of what this audience has responded to, at what size, and what it has ignored.
Before committing traffic, use the conversion rate calculator for landing page tests to turn the planned minimum effect into the extra conversions a winning variant would be worth.
Common mistakes
- Choosing the metric after the data arrives. The primary KPIA metric tied directly to an objective. is fixed in the plan; anything chosen later is a story about the data.
- Session-level assignment for multi-visit decisions. The same person sees both versions and the measurement is of neither.
- Skipping the tracking walk. A tag that fires differently in each arm manufactures a result.
- Stopping when it looks good. The error rate climbs with every unplanned look.
- Changing allocation or the variant mid-test. Time becomes a confound; restart.
- Rescuing a lost test with segments. One segment will always look like a winner; test it next instead.
- Treating inconclusive as "no effect". A wide interval means the test could not tell, which is a statement about traffic, not about the change.
- Not writing it down. The next test will repeat this one.
