Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Conversion Rate Optimisation › Experimentation

Conversion Work Without an A/B Test

When an A/B test is the wrong tool and what to do instead: changes to just ship, low traffic, template tests, bolder changes and honest before-and-after reads.
Guideintermediate16 min readUpdated

Three reasons not to run an A/B test

A controlled test answers one question well: did this change cause that movement in this metric, for this audience? The experimentation hub explains why nothing else answers it as cleanly. But a test costs traffic, time and attention, and it only earns that cost when three things are true at once: there is a genuine question, the traffic can answer it, and the answer is worth having. When one of them fails, a test is the wrong tool, and running it anyway produces a delay at best and a confident wrong answer at worst. The academy checkpoint on conversion work without a split test tests whether you can sort changes into fix-and-ship, untestable and not worth testing.

CaseHow to recognise itWhat to do insteadWhere it is covered below
There is no questionThe change fixes a fault, a wrong price, an accessibility barrier or a legal requirementShip it, verify it, log itChanges that should simply ship
The traffic cannot answerThe sample the test needs would take longer than the business will waitMake a test possible, or gather evidence another wayWhen the traffic cannot answer; making a test possible
The answer is not worth the costThe change is cheap, low-risk and easily undoneShip with a written prediction and a date to checkChanges that should simply ship

The sample size guide lists the honest responses to a test the traffic cannot support in one paragraph: test a bolder change, test earlier in the journey, measure something else, or decide on judgement and say so. This guide turns that paragraph into a method, so that a business without the traffic for constant testing still learns from the changes it makes. If a test has already run and come back without a verdict, the diagnosis for an A/B test not reaching significance is the place to start instead.

Changes that should simply ship

Bugs, wrong prices and obligations

The CRO pillar and the conversion research hub make the same rule: broken things are fixed, not tested. Errors, dead links, forms that fail on one device, prices that differ between the advert and the page, and steps that some visitors cannot complete are not hypotheses. Nobody needs a controlled experiment to learn that a working form beats a broken one, and a test would keep the fault in place for part of the traffic for as long as it ran.

The same reasoning covers two further groups. Accessibility barriers are bugs for the visitors they stop, and the accessibility guide explains why testing whether to remove one keeps it in place for half the traffic and could not answer the question anyway, because the people affected are usually a small part of any sample. Legal and regulatory fixes, such as a corrected price claim, a consent box that should never have been ticked by default or a missing cost, are not optional, so there is nothing to test.

Shipping without a test does not mean shipping without care. Each fix is checked on real devices, logged with its date, and followed by a look at the step it affects, so that a fix that did not work, or broke something else, is found quickly. The fix was justified by the fault, not by a number, so the step's rate afterwards is a check rather than a verdict. What can sometimes be tested is the choice between two ways of fixing the same thing, once both work.

Cheap, reversible changes

The one-way and two-way doors framework sorts decisions by how hard they are to undo. A two-way door, such as a revised headline, a reordered list of benefits or a new photograph on a minor page, is cheap to reverse, affects a limited audience and leaves nothing signed or spent. For those, the framework's route is to act and watch: write down what should happen, ship the change, check on a set date, and reverse it if the result misses the expectation.

The framework is explicit that a two-way door still needs measurement, because a change is only cheap to reverse if someone notices it should be reversed. The measurement can be light. A one-line Hypothesis, a metric, a date in the calendar and a note in the change log are enough. What it cannot be is absent, or reversible changes pile up unexamined and nobody can tell which of them helped.

When the traffic cannot answer

Whether a test can answer its question is a matter of arithmetic, done before launch with the site's own figures. Three inputs decide it, and the sample size guide explains each in plain language.

  • The baseline rate of the step being tested, measured on the audience the test would include, over full weeks. The conversion rate calculator turns counts into the rate.
  • The smallest effect worth having, which is a business decision: how small an improvement would still be worth building, maintaining and explaining? Smaller effects need far more visitors to detect.
  • The visitors per arm that a test needs to detect that effect at the conventional significance and power, which follows from the first two.

Divide the visitors per arm by the visitors the page receives each week, multiply by the number of arms, and the answer is the length of the test in weeks. If that is longer than the business will wait, longer than the page will stay unchanged, or long enough that the season will change underneath it, the test cannot answer the question at this traffic. The conversion rate calculator set up for landing page tests translates the planned effect into the extra enquiries or orders a winning variant would be worth, which is a useful check on whether the effect chosen is one the business actually cares about.

Count conversions, not visits. The pillar's answer to how much traffic testing needs makes the point: a test has to detect a change in the rate at which visitors convert, and that needs enough conversions in each arm within a period the team will wait. A page with plenty of visits and few conversions is a low-traffic page for testing purposes.

Two shortcuts make the problem look smaller and are both mistakes. Choosing a large detectable effect because it makes the test short means the test will miss the modest improvements most changes produce and report "no significant difference" for changes that work. Running the test anyway and stopping when the result looks good is worse. Evan Miller's essay on how not to run an A/B test, listed in this page's sources, shows that checking a test repeatedly and stopping the first time it looks significant inflates false positives far beyond the level the significance threshold promises. The testing guides take that point as a rule: the sample is fixed in advance and the result is read once. A test that cannot reach its sample does not become readable by being watched more closely.

Making a test possible: templates, bolder changes, earlier steps

When the arithmetic rules out the test as first imagined, it is often possible to redesign the question so that the traffic can answer it. Each option buys detectability by giving something up, and the trade should be made deliberately.

OptionWhat it buysWhat it gives upRead more
Test a template across many pagesThe combined traffic of every page that uses itThe answer is about the template, not any single pageProduct page optimisation
Test a bolder changeA larger effect, which needs fewer visitors to detectLess certainty about which part of the change workedSample size and significance
Test earlier in the journeyA more common event, so more conversions per weekA weaker link to revenue; downstream metrics become guardrailsA/B testing guide
Decide on a validated proxy, confirm laterA readout within the test periodA provisional decision until the slower outcome arrivesExperimentation loop for lead generation

Templates. The product page optimisation guide describes the approach in its section on testing product pages: a change to the gallery, the specification table or the delivery message is made to the template and judged across all the products that use it, which gives the volume a single page lacks. The same works for service pages, location pages, article templates and any set of pages built from one design. The question shifts from "does this page convert better" to "does this template convert better", which is usually the question the business has anyway.

Bolder changes. The sample size guide is direct: low-traffic pages cannot test small changes, however well reasoned, and should test bold ones or not at all. A new offer, a different structure for the page or a removed step can produce an effect large enough to detect where a new button colour cannot. The cost is attribution. A variant that changes several things at once tells you the bundle worked, not which part, so record the bundle honestly and, if it wins, decide whether the parts are worth testing separately later.

Earlier steps. An event earlier in the journey, such as starting a form, adding to a basket or reaching the pricing page, happens more often than the final conversion, so a test on it reaches its sample sooner. The A/B testing guide names the risk: a variant can win on clicks while losing on sales. Keep the final conversion as a guardrail, and treat a win on an early step as evidence about attention rather than proof of more revenue.

Proxies for slow outcomes. In lead generation the outcome that matters, a qualified lead or an opportunity, is often too rare to test on and arrives weeks after the test ends. The experimentation loop for lead generation sets out the alternative: decide on a proxy available at submission, but only after checking on past leads that the proxy separates leads that progressed from leads that did not, and schedule a second readout on pipeline once the sales cycle has passed.

Evidence without a split test

Not every useful piece of evidence comes from a controlled comparison. When a test is impossible, these methods still reduce the chance of shipping something that makes things worse.

Usability sessions before shipping. The conversion research methods guide describes usability testing as the right method when there is not enough traffic for behavioural or quantitative methods to say anything, which makes it the backbone of research for low-traffic sites and for new flows that have no visitors yet. Give a few realistic participants a task on a prototype or a staging version of the change, watch where they struggle, fix it, and run a few more. The guide's practical rule is to run a handful of sessions, fix what they found and run another handful, rather than running many at once. Sessions do not say how much a change will lift the rate; they say whether people can use it and whether it answers the questions they have. The usability testing tutorial runs a session round on a conversion journey step by step, from the task to the backlog entry.

Triangulated research. The methods guide's triangulation, in which analytics sizes a problem, recordings show it, polls and support tickets explain it, and a finding is trusted when the sources agree, does not depend on traffic for a split. A change built on evidence from several sources is a safer change to ship without a test than one built on opinion, and the research record is what the team reads if the result surprises them later.

Expert and customer review. Reading the page against the questions customers ask in support tickets, sales calls and reviews costs little and often finds the objection the page never answers. It is weaker evidence than a session, because it relies on the team's judgement, and it is best used to decide what to put in front of participants.

Judgement, stated as such. The sample size guide allows a final option: decide on judgement and say so. The decision record then shows which evidence was used and that no test was possible, so nobody later mistakes the change for a proven win.

Ship, monitor and read honestly

A change shipped without a control group can still be read honestly, but only if the reading is designed before the change goes live. The experimentation hub explains the weakness of before-and-after comparison: everything else that moved in the same weeks, from a promotion to a tracking fix, gets mixed into the result. The routine below does not remove that weakness. It limits it, and it stops the result from being read in whichever way flatters the change.

Before the change ships, write downWhy it matters
The prediction: which metric should move, in which direction, roughly how farA result can then be checked against something fixed, not against hope
The reading date and the length of the reading periodThe decision on when to look is made before there is anything to look at
The comparison period and how it was chosenStops the comparison being picked after the fact to suit the result
The minimum count below which the result will not be judgedKeeps small numbers from being read as precise
What would make the team reverse the changeTurns a two-way door into one somebody actually walks back through
Everything else changing in the same weeksThe list of rival explanations exists before the result does

Fix the reading date first. The stopping-rule warning the testing guides take from Evan Miller's essay applies here too. Looking at a before-and-after chart every morning and declaring success on the first good day is peeking by another name. Decide when to read before looking, and read once.

Choose the comparison period as the comparing-periods guide sets out. The guide to comparing periods gives the rules: align weekdays and the holidays that move, compare full weeks, use the same weeks last year where Seasonality matters, treat any change in tracking or definitions as a break, and compare periods at equal age so that late conversions do not make the most recent weeks look worse. Where a business has several similar units, such as shops, regions or product ranges, and only some receive the change, the others are a comparison group in the same period, which takes the calendar out of the comparison, although the units still differ from each other in other ways.

Read small numbers as small numbers. The guide to reading small numbers explains why rates built on a few dozen conversions swing widely with nothing changed. Show the count beside every rate, put a range around the figure rather than quoting it alone, pool weeks until the count can carry a judgement, and watch for regression to the mean: a page chosen for a fix because it had an unusually bad month will usually look better the next month even if the fix did nothing.

Log the change with its date. A dated entry in the measurement change log, with the expected effect written in its field and the observed effect filled in at the reading date, is what lets the next analyst see why a line moved. The tutorial suggests an optional trading-events type for price changes and similar decisions, marked so it can be filtered apart, which lets the same log hold conversion changes alongside measurement ones.

Keep a group out where one can be kept out. The test hub's table of designs includes two that sit between a full test and a plain before-and-after. A staged rollout exposes the change to a growing share of visitors, and it becomes a controlled comparison only if the groups are random and concurrent. A Holdout Group keeps a group away from a change for a period to measure its cumulative value, at the cost that the group forgoes the benefit. Where a site can assign even a small random group to the old experience for a fixed period, the reading is far stronger than any calendar comparison. Where it cannot, say so in the write-up.

The experimentation cadence guide records the same balance from the programme side: some stages cannot reach significance in useful time, and the choice is between a longer test, a bolder change, or a before-and-after design read with appropriate caution. Every one of those results goes into the same learning log as the tests, labelled with the strength of its evidence. The hierarchy of evidence gives those labels a common order, from expert review through before-and-after readings to randomised tests. For a metric read every week, the comparison of control charts and A/B testing shows how limits drawn from the metric's own history make a shipped change's reading more disciplined, and what that reading still cannot prove.

A furniture maker's year without split tests (illustrative)

Illustrative example — not a real result

Chiselmoth Furniture learns without a control group

Chiselmoth Furniture is an invented maker of made-to-order dining tables that sells through its own site. It receives about 3,400 visits and 9 quote requests a week, roughly 0.26% of visits. In January a customer email reveals that the quote form fails on one mobile browser when a dimension contains a decimal point. The fix ships the same day without a test, and the team logs it. In February the owner wants to test a new size configurator. With a baseline of 0.26% and an improvement worth having of 20% relative, the calculation asks for roughly 150,000 visitors per arm, which at the shop's traffic would take about 88 weeks. The test is dropped. Instead, two rounds of four usability sessions on a staging version find that people do not know whether the length includes the overhang; the label is changed before launch. In April a new product template ships across all 14 table models with a written prediction: quote requests per visit should rise by a fifth or more, read once on 30 June, against the same 13 weeks last year aligned by weekday and with the Easter week matched. On the reading date the period shows 131 quote requests against 104, with traffic up about 6%, a rise the team reports as consistent with the prediction but not proven, because a magazine feature ran in May. For the one change the owner most wants proof of, a delivery and lead-time line under the price, the team pools all 14 product pages and splits visitors at random, accepting that only an effect of about half again would be detectable in the 16 weeks they are prepared to wait. It is inconclusive, and the line stays because customers in the sessions asked for it.

Common mistakes

  • Testing a bug fix. A broken step is fixed, not tested; a test keeps the fault in place for part of the traffic.
  • Running a test the traffic cannot finish. Do the sample-size arithmetic before launch, with the site's own baseline and the smallest effect worth having.
  • Inflating the detectable effect to shorten the test. The test then misses the modest gains most changes produce.
  • Watching a before-and-after chart daily. Decide the reading date and the comparison period before the change ships, and read once.
  • Comparing a launch month with a different season. Align weekdays and holidays, and use last year's weeks where the business is seasonal.
  • Quoting a rate without its count. A rate from a few conversions is a wide range, not a figure.
  • Shipping a two-way door and never checking it. A reversible change is only reversible if someone looks on the agreed date.
  • Calling a before-and-after result a win. The honest wording is that the result was consistent with the prediction, with the rival explanations listed.
  • Treating usability sessions as a measurement of lift. They show whether people can use a change and what confuses them, not how much it will move the rate.
  • Leaving judgement calls unlabelled. Record that no test was possible and which evidence the decision rested on.

Frequently asked questions

How do I know whether my site has enough traffic for A/B testing?

There is no universal traffic level, and any figure quoted without your baseline is a guess. Work it out for the specific test: take the page's current conversion rate from your own analytics, decide the smallest improvement that would be worth shipping, and let a sample-size calculation say how many visitors each arm needs. Divide that by the visitors the page receives each week. If the answer is longer than the business will wait, or longer than the page will stay unchanged, that test cannot answer its question at your traffic, whatever the site's total visits.

Is a before-and-after comparison ever good enough?

It is good enough for decisions that are cheap to reverse, provided it is set up honestly: a written prediction, a reading date and a comparison period chosen before the change ships, counts shown beside every rate, and a dated note of everything else that changed in the same weeks. It cannot separate the change from its context the way a controlled test can, so the claim it supports is modest: the result was consistent with the prediction, or it was not. For changes that are costly to undo, look for a way to test or pilot first.

Should bug fixes be A/B tested?

No. A broken form, a wrong price or a step that fails on one browser is a bug, not a hypothesis, and conversion programmes fix bugs rather than test them. A test would keep the fault in place for part of the traffic while it ran. What can sometimes be tested is the choice between two ways of fixing something, once both versions work. Log the fix with its date, check it on real devices and watch the step afterwards.

Terms used on this page

Tools to try

Frameworks

Where this page is used

Part of these learning paths

Also related

Sources

  1. How Not To Run an A/B Test — Evan Miller

Related