Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Conversion Rate Optimisation › Troubleshooting

A/B Test Not Reaching Significance

Why an A/B test never reaches significance: too little traffic, a change too small to notice, a distant metric, too many variants or no real effect.
Troubleshootingintermediate8 min readUpdated

First, check the test against its planned sample and minimum effect

Find the test plan and read three numbers from it: the baseline conversion rate (CVR) of the step being tested, the minimum detectable effect the test was designed to find, and the visitors per arm that combination required. If there is no plan, reconstruct those numbers now using the method in the sample size and significance guide; most inconclusive tests turn out never to have had a realistic plan. Then compare the plan with what happened: how many visitors each arm has received, how long that took, and whether the baseline during the test matches the baseline the plan assumed.

The comparison sorts the causes. If the arms have not yet reached the planned sample, the test is unfinished rather than inconclusive. If they have, and the confidence interval for the difference spans both a worthwhile gain and a loss, the test lacked the power to answer. If the interval is narrow and sits close to zero, the change most likely does little. Before reading anything, confirm the test is healthy: if the arms are unevenly split, the result swung back and forth, or a winner failed to hold after rollout, the problem is a pitfall rather than power, and common testing pitfalls is the page to use. The conversion rate troubleshooting hub lists the other conversion symptoms.

Diagnosis table

Diagnosis

SymptomCheckLikely causeFix
The planned sample was reached, and the interval spans gain and lossThe minimum detectable effect in the plan against effects seen in past testsSample too small for the effectRe-plan for a realistic effect, or test something bolder
Both arms convert almost identically from the first weekWhether visitors could notice the change; what evidence the hypothesis rested onChange too small to noticeTest a change large enough to alter behaviour
Clicks on the changed element moved, but the deciding metric did notHow many steps lie between the change and the primary metricPrimary metric too far from the changeUse the nearest reliable metric; keep outcomes as guardrails
Several variants each have a fraction of the visitors and none separatesNumber of arms and visitors per arm against the planTraffic split across too many variantsCut to the strongest variant, or sequence the tests
The test needs many months to reach its sampleEligible visitors per week on the page against the sample requiredPage has too little trafficMove the test to a busier page or step, or decide without one
A well-powered test ends with a narrow interval around zeroThe interval's width and position; the learning log for similar testsThe change makes no real differenceRecord the null result and stop investing in the idea

Fix, verify and prevent each cause

A sample too small for the effect being sought

The smaller the effect a test must detect, the more visitors it needs, and the relationship is steep. A plan that assumed a large lift, because a large lift makes the test short, will miss the smaller effects most real changes produce.

Fix: re-plan with a minimum detectable effect drawn from the effects past tests on similar changes actually produced, calculate the Sample Size it requires as the sample size and significance guide explains, and run the new plan to its end; Evan Miller's essay on how not to run an A/B test explains why the sample must be fixed in advance rather than extended until a result appears.

Verify: the re-run reaches its planned sample within the time agreed, and its interval is narrow enough to support a decision either way.

Prevent: require a sample-size calculation, based on realistic effects, in every test plan before launch.

A change too slight for visitors to notice

A new shade of button, a reworded subhead or a moved icon may be invisible to most visitors, so behaviour does not change and no amount of traffic will show a difference.

Fix: go back to the evidence behind the hypothesis, as the writing hypotheses tutorial describes, and propose a change that addresses the visitor's problem directly and visibly, with a predicted effect sized from that evidence; one clear change, not a bundle.

Verify: the new variant produces a measurable shift in behaviour at the changed element, such as clicks or progress past it, within the first full weeks.

Prevent: reject hypotheses that cannot state the visitor problem they solve and the size of the effect they expect.

A deciding metric far from the change

A change to a product page judged on completed orders must move a rate several steps away, and the noise of every step in between swamps it.

Fix: choose the primary metric nearest to the change that still reflects the outcome, such as add-to-basket for a product page change, and keep orders and revenue as guardrails, following the unit, metric and guardrail decisions in the A/B testing guide.

Verify: the re-planned test needs a smaller sample for the same relative effect and reaches a verdict on the nearer metric, with guardrails unharmed.

Prevent: write the primary metric and the reason it was chosen into the plan, and have an analyst check it against the change before launch.

Visitors divided among too many variants

Every extra variant takes a share of the traffic and adds a comparison, so a test with several arms needs far longer than a simple A against B, and the smallest arm sets the pace.

Fix: cut the test to the control and the variant with the strongest evidence behind it, and run the others later in sequence, as the allocation section of the A/B testing guide and the test-type comparison in the experimentation section explain.

Verify: the reduced test reaches its planned sample per arm in the time the plan allowed.

Prevent: allow extra variants only when the sample-size calculation shows the traffic can support every arm within the time ceiling.

A page without enough visitors

Some pages, however important, see too few eligible visitors for any realistic effect to be detected in a period the business will wait.

Fix: calculate the run length the test would need on that page, as the traffic constraint in prioritising tests sets out; if it exceeds the ceiling, move the test to a busier page or an earlier step where the event is more common, test a much bolder change, or decide on judgement backed by usability research, and say so.

Verify: the moved or redesigned test has a run length within the ceiling before it launches.

Prevent: attach a run length to every hypothesis in the backlog and park those that exceed the ceiling.

A change that genuinely does nothing

A well-planned test that reaches its sample and returns a narrow interval around zero has answered the question: the change does not matter much to visitors. Treating that as failure and re-running it wastes traffic.

Fix: record the null result, the interval and the reasoning in the learning log described in the experimentation programme playbook, and move to the next hypothesis in the backlog.

Verify: the log shows the decision taken, and the idea is not retested without new evidence.

Prevent: read the learning log before approving new hypotheses so that ideas already shown to make no difference are not proposed again under a new name.

A prevention routine for decisive tests

Most inconclusive tests are designed that way before they start, so the routine sits at planning. No test launches without a written plan that states the baseline, the minimum effect worth shipping drawn from past results, the sample per arm, the expected run length, the primary metric and its distance from the change, the guardrails and the number of arms. An analyst checks that plan against the page's eligible traffic and rejects any test whose run length exceeds the agreed ceiling. During the run, the team checks only health, such as the split between arms and incidents, and does not read the result until the planned sample is reached. After the run, every result, including nulls and inconclusive ones, goes into the learning log with its interval. Quarterly, review the log for the share of tests that ended without a decision; if it is high, the minimum effects in plans are too optimistic or the backlog contains too many small changes on quiet pages, and the prioritisation rules should change before the next quarter's tests.

Illustrative example — not a real result

A travel company's stalled headline test

A travel company ran a test of 4 headline variants against its control on a tour page with about 6,000 visitors a month, judged on completed bookings at a 1.2% baseline. After 10 weeks no variant had separated from the control. The plan had assumed a 30% relative lift; past headline tests in the company's log had produced lifts between 3% and 8%. At a 5% effect, each arm would have needed roughly 500,000 visitors. The team stopped the test, recorded it as inconclusive, and replaced it with one bolder variant that put prices and dates in the first screen, judged on clicks to the booking form, which converted at 9%. Planned for a 15% relative lift with 2 arms, it needed about 8,000 visitors per arm, reached them in under 3 months on the combined tour pages, and showed a lift of 17%.

Common mistakes

Launching without a sample-size plan. Assuming large effects because they make tests short. Extending a test until it crosses the line. Testing small changes on quiet pages. Splitting traffic across many variants. Re-running a clear null result in the hope of a different answer.

Frequently asked questions

Should we keep running a test until it becomes significant?

No. Running on until the result crosses the significance line, and stopping there, makes false winners far more likely, which is the core warning in Evan Miller's essay on A/B testing. Run to the sample size planned before launch, read the result once, and if it is inconclusive, say so. If the plan was unrealistic, redesign the test rather than extending it indefinitely.

Does an inconclusive test mean the variant does not work?

Not necessarily. It means the test could not distinguish the variant's effect from noise at the sample it had. The confidence interval says how large or small the true effect could plausibly be: if it spans both a meaningful gain and a meaningful loss, the test lacked the traffic to answer. If it is narrow and centred near zero, the change most likely makes little difference.

Can we lower the significance threshold to get a result?

Changing the threshold after the test has run turns the plan into whatever produces a winner, which defeats the purpose of testing. Decide the threshold, the power and the minimum effect worth detecting before launch. If the business is willing to accept more risk of a false winner, that is a decision for the next test's plan, made in advance and written down.

Terms used on this page

Where this page is used

Also related

Sources

  1. How Not To Run an A/B Test — Evan Miller

Related