Three reasons not to run an A/B test
A controlled test answers one question well: did this change cause that movement in this metric, for this audience? The experimentation hub explains why nothing else answers it as cleanly. But a test costs traffic, time and attention, and it only earns that cost when three things are true at once: there is a genuine question, the traffic can answer it, and the answer is worth having. When one of them fails, a test is the wrong tool, and running it anyway produces a delay at best and a confident wrong answer at worst. The academy checkpoint on conversion work without a split test tests whether you can sort changes into fix-and-ship, untestable and not worth testing.
| Case | How to recognise it | What to do instead | Where it is covered below |
|---|---|---|---|
| There is no question | The change fixes a fault, a wrong price, an accessibility barrier or a legal requirement | Ship it, verify it, log it | Changes that should simply ship |
| The traffic cannot answer | The sample the test needs would take longer than the business will wait | Make a test possible, or gather evidence another way | When the traffic cannot answer; making a test possible |
| The answer is not worth the cost | The change is cheap, low-risk and easily undone | Ship with a written prediction and a date to check | Changes that should simply ship |
The sample size guide lists the honest responses to a test the traffic cannot support in one paragraph: test a bolder change, test earlier in the journey, measure something else, or decide on judgement and say so. This guide turns that paragraph into a method, so that a business without the traffic for constant testing still learns from the changes it makes. If a test has already run and come back without a verdict, the diagnosis for an A/B test not reaching significance is the place to start instead.
Changes that should simply ship
Bugs, wrong prices and obligations
The CRO pillar and the conversion research hub make the same rule: broken things are fixed, not tested. Errors, dead links, forms that fail on one device, prices that differ between the advert and the page, and steps that some visitors cannot complete are not hypotheses. Nobody needs a controlled experiment to learn that a working form beats a broken one, and a test would keep the fault in place for part of the traffic for as long as it ran.
The same reasoning covers two further groups. Accessibility barriers are bugs for the visitors they stop, and the accessibility guide explains why testing whether to remove one keeps it in place for half the traffic and could not answer the question anyway, because the people affected are usually a small part of any sample. Legal and regulatory fixes, such as a corrected price claim, a consent box that should never have been ticked by default or a missing cost, are not optional, so there is nothing to test.
Shipping without a test does not mean shipping without care. Each fix is checked on real devices, logged with its date, and followed by a look at the step it affects, so that a fix that did not work, or broke something else, is found quickly. The fix was justified by the fault, not by a number, so the step's rate afterwards is a check rather than a verdict. What can sometimes be tested is the choice between two ways of fixing the same thing, once both work.
Cheap, reversible changes
The one-way and two-way doors framework sorts decisions by how hard they are to undo. A two-way door, such as a revised headline, a reordered list of benefits or a new photograph on a minor page, is cheap to reverse, affects a limited audience and leaves nothing signed or spent. For those, the framework's route is to act and watch: write down what should happen, ship the change, check on a set date, and reverse it if the result misses the expectation.
The framework is explicit that a two-way door still needs measurement, because a change is only cheap to reverse if someone notices it should be reversed. The measurement can be light. A one-line HypothesisA testable statement of the change to make, the effect expected on a named metric and the evidence-based reason for expecting it., a metric, a date in the calendar and a note in the change log are enough. What it cannot be is absent, or reversible changes pile up unexamined and nobody can tell which of them helped.
When the traffic cannot answer
Whether a test can answer its question is a matter of arithmetic, done before launch with the site's own figures. Three inputs decide it, and the sample size guide explains each in plain language.
- The baseline rate of the step being tested, measured on the audience the test would include, over full weeks. The conversion rate calculator turns counts into the rate.
- The smallest effect worth having, which is a business decision: how small an improvement would still be worth building, maintaining and explaining? Smaller effects need far more visitors to detect.
- The visitors per arm that a test needs to detect that effect at the conventional significance and power, which follows from the first two.
Divide the visitors per arm by the visitors the page receives each week, multiply by the number of arms, and the answer is the length of the test in weeks. If that is longer than the business will wait, longer than the page will stay unchanged, or long enough that the season will change underneath it, the test cannot answer the question at this traffic. The conversion rate calculator set up for landing page tests translates the planned effect into the extra enquiries or orders a winning variant would be worth, which is a useful check on whether the effect chosen is one the business actually cares about.
Count conversions, not visits. The pillar's answer to how much traffic testing needs makes the point: a test has to detect a change in the rate at which visitors convert, and that needs enough conversions in each arm within a period the team will wait. A page with plenty of visits and few conversions is a low-traffic page for testing purposes.
Two shortcuts make the problem look smaller and are both mistakes. Choosing a large detectable effect because it makes the test short means the test will miss the modest improvements most changes produce and report "no significant difference" for changes that work. Running the test anyway and stopping when the result looks good is worse. Evan Miller's essay on how not to run an A/B test, listed in this page's sources, shows that checking a test repeatedly and stopping the first time it looks significant inflates false positives far beyond the level the significance threshold promises. The testing guides take that point as a rule: the sample is fixed in advance and the result is read once. A test that cannot reach its sample does not become readable by being watched more closely.
Making a test possible: templates, bolder changes, earlier steps
When the arithmetic rules out the test as first imagined, it is often possible to redesign the question so that the traffic can answer it. Each option buys detectability by giving something up, and the trade should be made deliberately.
| Option | What it buys | What it gives up | Read more |
|---|---|---|---|
| Test a template across many pages | The combined traffic of every page that uses it | The answer is about the template, not any single page | Product page optimisation |
| Test a bolder change | A larger effect, which needs fewer visitors to detect | Less certainty about which part of the change worked | Sample size and significance |
| Test earlier in the journey | A more common event, so more conversions per week | A weaker link to revenue; downstream metrics become guardrails | A/B testing guide |
| Decide on a validated proxy, confirm later | A readout within the test period | A provisional decision until the slower outcome arrives | Experimentation loop for lead generation |
Templates. The product page optimisation guide describes the approach in its section on testing product pages: a change to the gallery, the specification table or the delivery message is made to the template and judged across all the products that use it, which gives the volume a single page lacks. The same works for service pages, location pages, article templates and any set of pages built from one design. The question shifts from "does this page convert better" to "does this template convert better", which is usually the question the business has anyway.
Bolder changes. The sample size guide is direct: low-traffic pages cannot test small changes, however well reasoned, and should test bold ones or not at all. A new offer, a different structure for the page or a removed step can produce an effect large enough to detect where a new button colour cannot. The cost is attribution. A variant that changes several things at once tells you the bundle worked, not which part, so record the bundle honestly and, if it wins, decide whether the parts are worth testing separately later.
Earlier steps. An event earlier in the journey, such as starting a form, adding to a basket or reaching the pricing page, happens more often than the final conversion, so a test on it reaches its sample sooner. The A/B testing guide names the risk: a variant can win on clicks while losing on sales. Keep the final conversion as a guardrail, and treat a win on an early step as evidence about attention rather than proof of more revenue.
Proxies for slow outcomes. In lead generation the outcome that matters, a qualified lead or an opportunity, is often too rare to test on and arrives weeks after the test ends. The experimentation loop for lead generation sets out the alternative: decide on a proxy available at submission, but only after checking on past leads that the proxy separates leads that progressed from leads that did not, and schedule a second readout on pipeline once the sales cycle has passed.
Evidence without a split test
Not every useful piece of evidence comes from a controlled comparison. When a test is impossible, these methods still reduce the chance of shipping something that makes things worse.
Usability sessions before shipping. The conversion research methods guide describes usability testingA research session in which a person attempts a realistic task on a site, app or prototype while an observer notes where they struggle. as the right method when there is not enough traffic for behavioural or quantitative methods to say anything, which makes it the backbone of research for low-traffic sites and for new flows that have no visitors yet. Give a few realistic participants a task on a prototype or a staging version of the change, watch where they struggle, fix it, and run a few more. The guide's practical rule is to run a handful of sessions, fix what they found and run another handful, rather than running many at once. Sessions do not say how much a change will lift the rate; they say whether people can use it and whether it answers the questions they have. The usability testing tutorial runs a session round on a conversion journey step by step, from the task to the backlog entry.
Triangulated research. The methods guide's triangulation, in which analytics sizes a problem, recordings show it, polls and support tickets explain it, and a finding is trusted when the sources agree, does not depend on traffic for a split. A change built on evidence from several sources is a safer change to ship without a test than one built on opinion, and the research record is what the team reads if the result surprises them later.
Expert and customer review. Reading the page against the questions customers ask in support tickets, sales calls and reviews costs little and often finds the objection the page never answers. It is weaker evidence than a session, because it relies on the team's judgement, and it is best used to decide what to put in front of participants.
Judgement, stated as such. The sample size guide allows a final option: decide on judgement and say so. The decision record then shows which evidence was used and that no test was possible, so nobody later mistakes the change for a proven win.
Ship, monitor and read honestly
A change shipped without a control group can still be read honestly, but only if the reading is designed before the change goes live. The experimentation hub explains the weakness of before-and-after comparisonA measure read before and after a change with no comparison group, so the season and everything else that moved in those weeks is credited to the change.: everything else that moved in the same weeks, from a promotion to a tracking fix, gets mixed into the result. The routine below does not remove that weakness. It limits it, and it stops the result from being read in whichever way flatters the change.
| Before the change ships, write down | Why it matters |
|---|---|
| The prediction: which metric should move, in which direction, roughly how far | A result can then be checked against something fixed, not against hope |
| The reading date and the length of the reading period | The decision on when to look is made before there is anything to look at |
| The comparison period and how it was chosen | Stops the comparison being picked after the fact to suit the result |
| The minimum count below which the result will not be judged | Keeps small numbers from being read as precise |
| What would make the team reverse the change | Turns a two-way door into one somebody actually walks back through |
| Everything else changing in the same weeks | The list of rival explanations exists before the result does |
Fix the reading date first. The stopping-rule warning the testing guides take from Evan Miller's essay applies here too. Looking at a before-and-after chart every morning and declaring success on the first good day is peeking by another name. Decide when to read before looking, and read once.
Choose the comparison period as the comparing-periods guide sets out. The guide to comparing periods gives the rules: align weekdays and the holidays that move, compare full weeks, use the same weeks last year where SeasonalityPredictable, repeating changes in demand tied to the calendar, such as months, holidays, weekdays or hours. matters, treat any change in tracking or definitions as a break, and compare periods at equal age so that late conversions do not make the most recent weeks look worse. Where a business has several similar units, such as shops, regions or product ranges, and only some receive the change, the others are a comparison group in the same period, which takes the calendar out of the comparison, although the units still differ from each other in other ways.
Read small numbers as small numbers. The guide to reading small numbers explains why rates built on a few dozen conversions swing widely with nothing changed. Show the count beside every rate, put a range around the figure rather than quoting it alone, pool weeks until the count can carry a judgement, and watch for regression to the mean: a page chosen for a fix because it had an unusually bad month will usually look better the next month even if the fix did nothing.
Log the change with its date. A dated entry in the measurement change log, with the expected effect written in its field and the observed effect filled in at the reading date, is what lets the next analyst see why a line moved. The tutorial suggests an optional trading-events type for price changes and similar decisions, marked so it can be filtered apart, which lets the same log hold conversion changes alongside measurement ones.
Keep a group out where one can be kept out. The test hub's table of designs includes two that sit between a full test and a plain before-and-after. A staged rolloutReleasing a change to a growing share of visitors, pages or regions in steps and reading each before the next; a test only if groups are random and concurrent. exposes the change to a growing share of visitors, and it becomes a controlled comparison only if the groups are random and concurrent. A Holdout GroupA randomly chosen part of an audience kept from exposure to measure what exposure caused. keeps a group away from a change for a period to measure its cumulative value, at the cost that the group forgoes the benefit. Where a site can assign even a small random group to the old experience for a fixed period, the reading is far stronger than any calendar comparison. Where it cannot, say so in the write-up.
The experimentation cadence guide records the same balance from the programme side: some stages cannot reach significance in useful time, and the choice is between a longer test, a bolder change, or a before-and-after design read with appropriate caution. Every one of those results goes into the same learning logA single, searchable record of every experiment, losers and inconclusive results included, with the hypothesis, design, result, decision and what it taught. as the tests, labelled with the strength of its evidence. The hierarchy of evidence gives those labels a common order, from expert review through before-and-after readings to randomised tests. For a metric read every week, the comparison of control charts and A/B testing shows how limits drawn from the metric's own history make a shipped change's reading more disciplined, and what that reading still cannot prove.
A furniture maker's year without split tests (illustrative)
Chiselmoth Furniture learns without a control group
Chiselmoth Furniture is an invented maker of made-to-order dining tables that sells through its own site. It receives about 3,400 visits and 9 quote requests a week, roughly 0.26% of visits. In January a customer email reveals that the quote form fails on one mobile browser when a dimension contains a decimal point. The fix ships the same day without a test, and the team logs it. In February the owner wants to test a new size configurator. With a baseline of 0.26% and an improvement worth having of 20% relative, the calculation asks for roughly 150,000 visitors per arm, which at the shop's traffic would take about 88 weeks. The test is dropped. Instead, two rounds of four usability sessions on a staging version find that people do not know whether the length includes the overhang; the label is changed before launch. In April a new product template ships across all 14 table models with a written prediction: quote requests per visit should rise by a fifth or more, read once on 30 June, against the same 13 weeks last year aligned by weekday and with the Easter week matched. On the reading date the period shows 131 quote requests against 104, with traffic up about 6%, a rise the team reports as consistent with the prediction but not proven, because a magazine feature ran in May. For the one change the owner most wants proof of, a delivery and lead-time line under the price, the team pools all 14 product pages and splits visitors at random, accepting that only an effect of about half again would be detectable in the 16 weeks they are prepared to wait. It is inconclusive, and the line stays because customers in the sessions asked for it.
Common mistakes
- Testing a bug fix. A broken step is fixed, not tested; a test keeps the fault in place for part of the traffic.
- Running a test the traffic cannot finish. Do the sample-size arithmetic before launch, with the site's own baseline and the smallest effect worth having.
- Inflating the detectable effect to shorten the test. The test then misses the modest gains most changes produce.
- Watching a before-and-after chart daily. Decide the reading date and the comparison period before the change ships, and read once.
- Comparing a launch month with a different season. Align weekdays and holidays, and use last year's weeks where the business is seasonal.
- Quoting a rate without its count. A rate from a few conversions is a wide range, not a figure.
- Shipping a two-way door and never checking it. A reversible change is only reversible if someone looks on the agreed date.
- Calling a before-and-after result a win. The honest wording is that the result was consistent with the prediction, with the rival explanations listed.
- Treating usability sessions as a measurement of lift. They show whether people can use a change and what confuses them, not how much it will move the rate.
- Leaving judgement calls unlabelled. Record that no test was possible and which evidence the decision rested on.
