First, check the test against its planned sample and minimum effect
Find the test plan and read three numbers from it: the baseline conversion rate (CVRShare of visits or clicks that convert.) of the step being tested, the minimum detectable effect the test was designed to find, and the visitors per arm that combination required. If there is no plan, reconstruct those numbers now using the method in the sample size and significance guide; most inconclusive tests turn out never to have had a realistic plan. Then compare the plan with what happened: how many visitors each arm has received, how long that took, and whether the baseline during the test matches the baseline the plan assumed.
The comparison sorts the causes. If the arms have not yet reached the planned sample, the test is unfinished rather than inconclusive. If they have, and the confidence interval for the difference spans both a worthwhile gain and a loss, the test lacked the power to answer. If the interval is narrow and sits close to zero, the change most likely does little. Before reading anything, confirm the test is healthy: if the arms are unevenly split, the result swung back and forth, or a winner failed to hold after rollout, the problem is a pitfall rather than power, and common testing pitfalls is the page to use. The conversion rate troubleshooting hub lists the other conversion symptoms.
Diagnosis table
Diagnosis
| Symptom | Check | Likely cause | Fix |
|---|---|---|---|
| The planned sample was reached, and the interval spans gain and loss | The minimum detectable effect in the plan against effects seen in past tests | Sample too small for the effect | Re-plan for a realistic effect, or test something bolder |
| Both arms convert almost identically from the first week | Whether visitors could notice the change; what evidence the hypothesis rested on | Change too small to notice | Test a change large enough to alter behaviour |
| Clicks on the changed element moved, but the deciding metric did not | How many steps lie between the change and the primary metric | Primary metric too far from the change | Use the nearest reliable metric; keep outcomes as guardrails |
| Several variants each have a fraction of the visitors and none separates | Number of arms and visitors per arm against the plan | Traffic split across too many variants | Cut to the strongest variant, or sequence the tests |
| The test needs many months to reach its sample | Eligible visitors per week on the page against the sample required | Page has too little traffic | Move the test to a busier page or step, or decide without one |
| A well-powered test ends with a narrow interval around zero | The interval's width and position; the learning log for similar tests | The change makes no real difference | Record the null result and stop investing in the idea |
Fix, verify and prevent each cause
A sample too small for the effect being sought
The smaller the effect a test must detect, the more visitors it needs, and the relationship is steep. A plan that assumed a large lift, because a large lift makes the test short, will miss the smaller effects most real changes produce.
Fix: re-plan with a minimum detectable effect drawn from the effects past tests on similar changes actually produced, calculate the Sample SizeThe number of users or sessions an experiment needs to detect a given effect reliably. it requires as the sample size and significance guide explains, and run the new plan to its end; Evan Miller's essay on how not to run an A/B test explains why the sample must be fixed in advance rather than extended until a result appears.
Verify: the re-run reaches its planned sample within the time agreed, and its interval is narrow enough to support a decision either way.
Prevent: require a sample-size calculation, based on realistic effects, in every test plan before launch.
A change too slight for visitors to notice
A new shade of button, a reworded subhead or a moved icon may be invisible to most visitors, so behaviour does not change and no amount of traffic will show a difference.
Fix: go back to the evidence behind the hypothesis, as the writing hypotheses tutorial describes, and propose a change that addresses the visitor's problem directly and visibly, with a predicted effect sized from that evidence; one clear change, not a bundle.
Verify: the new variant produces a measurable shift in behaviour at the changed element, such as clicks or progress past it, within the first full weeks.
Prevent: reject hypotheses that cannot state the visitor problem they solve and the size of the effect they expect.
A deciding metric far from the change
A change to a product page judged on completed orders must move a rate several steps away, and the noise of every step in between swamps it.
Fix: choose the primary metric nearest to the change that still reflects the outcome, such as add-to-basket for a product page change, and keep orders and revenue as guardrails, following the unit, metric and guardrail decisions in the A/B testing guide.
Verify: the re-planned test needs a smaller sample for the same relative effect and reaches a verdict on the nearer metric, with guardrails unharmed.
Prevent: write the primary metric and the reason it was chosen into the plan, and have an analyst check it against the change before launch.
Visitors divided among too many variants
Every extra variant takes a share of the traffic and adds a comparison, so a test with several arms needs far longer than a simple A against B, and the smallest arm sets the pace.
Fix: cut the test to the control and the variant with the strongest evidence behind it, and run the others later in sequence, as the allocation section of the A/B testing guide and the test-type comparison in the experimentation section explain.
Verify: the reduced test reaches its planned sample per arm in the time the plan allowed.
Prevent: allow extra variants only when the sample-size calculation shows the traffic can support every arm within the time ceiling.
A page without enough visitors
Some pages, however important, see too few eligible visitors for any realistic effect to be detected in a period the business will wait.
Fix: calculate the run length the test would need on that page, as the traffic constraint in prioritising tests sets out; if it exceeds the ceiling, move the test to a busier page or an earlier step where the event is more common, test a much bolder change, or decide on judgement backed by usability research, and say so.
Verify: the moved or redesigned test has a run length within the ceiling before it launches.
Prevent: attach a run length to every hypothesis in the backlog and park those that exceed the ceiling.
A change that genuinely does nothing
A well-planned test that reaches its sample and returns a narrow interval around zero has answered the question: the change does not matter much to visitors. Treating that as failure and re-running it wastes traffic.
Fix: record the null result, the interval and the reasoning in the learning log described in the experimentation programme playbook, and move to the next hypothesis in the backlog.
Verify: the log shows the decision taken, and the idea is not retested without new evidence.
Prevent: read the learning log before approving new hypotheses so that ideas already shown to make no difference are not proposed again under a new name.
A prevention routine for decisive tests
Most inconclusive tests are designed that way before they start, so the routine sits at planning. No test launches without a written plan that states the baseline, the minimum effect worth shipping drawn from past results, the sample per arm, the expected run length, the primary metric and its distance from the change, the guardrails and the number of arms. An analyst checks that plan against the page's eligible traffic and rejects any test whose run length exceeds the agreed ceiling. During the run, the team checks only health, such as the split between arms and incidents, and does not read the result until the planned sample is reached. After the run, every result, including nulls and inconclusive ones, goes into the learning log with its interval. Quarterly, review the log for the share of tests that ended without a decision; if it is high, the minimum effects in plans are too optimistic or the backlog contains too many small changes on quiet pages, and the prioritisation rules should change before the next quarter's tests.
A travel company's stalled headline test
A travel company ran a test of 4 headline variants against its control on a tour page with about 6,000 visitors a month, judged on completed bookings at a 1.2% baseline. After 10 weeks no variant had separated from the control. The plan had assumed a 30% relative lift; past headline tests in the company's log had produced lifts between 3% and 8%. At a 5% effect, each arm would have needed roughly 500,000 visitors. The team stopped the test, recorded it as inconclusive, and replaced it with one bolder variant that put prices and dates in the first screen, judged on clicks to the booking form, which converted at 9%. Planned for a 15% relative lift with 2 arms, it needed about 8,000 visitors per arm, reached them in under 3 months on the combined tour pages, and showed a lift of 17%.
Common mistakes
Launching without a sample-size plan. Assuming large effects because they make tests short. Extending a test until it crosses the line. Testing small changes on quiet pages. Splitting traffic across many variants. Re-running a clear null result in the hope of a different answer.
