Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Conversion Rate Optimisation › Experimentation

Common Testing Pitfalls

A diagnosis guide for A/B testing problems: sample ratio mismatch, peeking, flicker, novelty effects, broken tracking, interacting tests, and segment fishing.
Troubleshootingintermediate12 min readUpdated

How to use this page

Start by establishing what kind of wrong the result is. A result can be wrong because the test did not measure what it claimed to (assignment or tracking faults), because it was read before it could be trusted (peeking, novelty, too little traffic), or because the thing it measured was not the change (flicker, interacting tests, incidents during the run). The checks differ, and the first group should be ruled out before the second, because no amount of statistics repairs a broken denominator.

Then confirm the basics: the planned sample from the test plan, the dates, the sample ratio per day, and the test log for anything that happened on the site during the run. Open the tag debugger and walk the conversion path in each arm, as the preview-mode debugging tutorial sets out, because tracking faults are both the most common cause and the fastest to confirm. Only when assignment and tracking are clean does the reading of the result become the question, and for that the sample size and significance guide is the reference.

Diagnosis table

Diagnosis

SymptomCheckLikely causeFix
The arms have noticeably unequal visitor counts and the gap persists day after dayObserved split against the planned split, daily, and by device and browserSample ratio mismatch: redirects failing on some browsers, bots landing in one arm, the variant erroring before the assignment event fires, or exclusions applied to one arm onlyFind and fix the mechanism; discard the data and restart; a test with a persistent mismatch cannot be read
The variant leads, then trails, then leads again as days passVisitors per arm against the planned sample; how many times the result has been viewedPeeking at an unfinished test; the swings are noise at small samplesStop reading until the planned sample is reached, or switch to a sequential method designed for repeated looks
The variant wins strongly in the first days and the lead shrinks steadilyDaily lift plotted over the run; new against returning visitorsNovelty effect: returning visitors react to the change itself, not to its meritRun longer and read the trend; weight the judgement towards new visitors
The variant won, was rolled out, and the metric did not movePost-rollout rate against the pre-test baseline; the test's confidence interval; how the test was stoppedA false positive from peeking or multiple comparisons, a novelty effect, or a tracking fault that inflated one armAudit the test record; re-test with a fixed sample if the change matters
The control's conversion rate during the test differs sharply from its usual rateControl rate against the weeks before the test and the same period last year; the exclusion rulesAudience filtered differently from the baseline, bots, or a tracking change at launchRebuild the baseline on the test's audience; fix tracking; restart
Conversions in one arm are far higher than the orders or leads the business recordedTag debugger walk of the conversion path in each arm; conversions against the back office for one closed dayThe conversion tag fires twice, or on reload, in one arm only; a redesigned confirmation page fires differentlyFix the tag; discard the data; restart
Conversions in one arm are zero or near zero while traffic is normalTag debugger on the variant's conversion page; the tag's trigger conditionsThe variant's page or element broke the trigger, or the assignment parameter is missing from the eventFix the trigger; restart
Visitors report seeing the old page flash before the new oneLoad the variant on a slow connection and an older device; measure time until the variant is appliedFlicker from client-side variant injectionMove the change server-side, apply it before first paint, or reduce the tool's load delay; restart if flicker was common
Two tests running on the same journey give results that contradict each otherWhich pages each test touches; whether assignment is independent between the testsInteracting tests sharing traffic or stepsRun them as one test with combined arms, or sequence them
The overall result is flat but one segment shows a large winNumber of segments examined; the segment's sample against the planned sample; whether the segment was named in the planSegment fishing: with enough slices, one will look like a winner by chanceTreat the segment as a hypothesis; re-test it with its own plan
The result changed when the test was re-analysed in analytics rather than in the toolUnit of counting in each system; date boundaries; filters and exclusionsDifferent units (users against sessions), different windows or different exclusionsAgree one source and one unit before the test; reconcile by cause
A campaign, outage or price change happened during the runThe test log and the site's change log against the daily liftAn incident that affected the arms unequally, or a changed traffic mixAnnotate; if the arms were affected differently, discard the affected days or restart

Sample ratio mismatch

The planned split is the first thing to check and the last thing teams look at. If the assignment is random, the observed split should sit very close to the intended one, and a gap that persists across days is not chance; it is a defect that is removing visitors from one arm selectively, which means the arms are no longer comparable. The usual mechanisms: a redirect that fails on some browsers so those visitors fall back to the control, bots that only ever see one arm, a variant that throws an error before the assignment event fires, a cookie that is set in one arm and not the other, or an exclusion rule applied after assignment to one arm only. Check the split daily and by device and browser, because the mechanism is often confined to one platform. A test with a persistent mismatch cannot be rescued by analysis; find the cause, fix it, discard the data and restart.

Peeking

A result that swings between winning and losing as the days pass is not a volatile audience; it is a sample too small to have settled, viewed too often. Each look is a chance for noise to cross the significance line, and a team that stops at the first crossing will ship changes that do nothing. Evan Miller's essay on how not to run an A/B test shows how large the inflation is. The check is simple: compare visitors per arm with the planned sample, and ask how many times the result has been read. The fix is procedural — run to the planned sample and read once — or methodological, by switching to a sequential method designed for repeated looks, with its own stricter thresholds. Monitoring for defects is not peeking; deciding from the primary metric is.

Flicker

Client-side testing tools apply the variant after the page has started to render, and on slow devices the visitor sees the control for a moment before it changes. That flash is a real experience — it looks broken — and it affects behaviour in the variant arm only, so the test is comparing "flicker plus the change" against "no flicker". Check by loading the variant on a slow connection and an older phone, and by measuring how long the tool takes to apply the change. The fixes are to render the variant server-side, to apply the change before first paint through the tool's anti-flicker mechanism, or to simplify the change so it applies faster. If flicker was common during the run, the data measures the wrong thing and the test should restart once it is fixed. Which tests can stay in the browser at all, and which belong on the server or as separate pages, is settled change by change in the guide to choosing between client-side and server-side testing.

Novelty effects

Returning visitors notice change. A new button, a moved element or a different layout draws attention simply by being different, and attention converts for a while. The signature is a strong early lead that shrinks steadily as the test runs. Check by plotting the daily lift across the run and by splitting new from returning visitors: a novelty effect lives in the returning segment and fades. The opposite also happens — a primacy effect, where returning visitors resist a change they have to relearn and the variant starts behind — with the same shape reversed. Both argue for running long enough for the trend to flatten and for reading new-visitor results as the better guide to the long-run effect. A test stopped during the novelty spike will report a win that rollout does not deliver. The same check applies to upstream metrics such as add-to-cart rate, which can show a novelty lift that never reaches orders.

Broken tracking

The most common cause of a confident wrong answer is a conversion tag that behaves differently in the two arms. A redesigned confirmation page whose trigger no longer matches, a tag that fires on reload in the variant, an assignment parameter missing from the event, or a variant that loads a second copy of the tag: each manufactures a difference that has nothing to do with the change. Check by walking the conversion path in each arm with the tag debugger, confirming the event fires once with the same parameters and the arm identifier attached, and by reconciling conversions against the back office for one closed day per arm. A conversion rate (CVR) in one arm that disagrees with the business's own count is the tell. The fix is in the container, and the test restarts afterwards; the data quality guide covers the weekly checks that catch this before a test is launched on top of it. The tutorial on analysing A/B tests in GA4 builds the exposure event that attaches the arm identifier and walks the conversion path in each arm before launch.

Interacting tests

Two tests that share traffic on the same journey — one on the product page, one on the basket — can each be well designed and still produce results that contradict each other, because a visitor's experience in one depends on their arm in the other. If the assignments are independent and the tests touch unrelated pages, the interaction is usually negligible. If they touch the same step or consecutive steps, the arms of one test are unevenly exposed to the arms of the other and neither result is clean. Check which pages each running test modifies and how assignment is done. The fixes are to run them as a single test with combined arms, which needs more traffic, or to sequence them, which takes longer. A programme calendar that shows every running test on a map of the journey prevents the collision.

Segment fishing

A test that is flat overall but shows a large win on one device, one source or one country is the most seductive pitfall, because the segment result looks like insight. It is usually noise: slice the audience enough ways and one slice will land far from the mean. Check how many segments were examined, whether the segment was named in the plan, and whether its sample alone approaches the planned size — it almost never does. The honest reading is that the segment is a hypothesis, to be tested with its own plan and its own sample. Segment analysis is valuable for generating the next test and dangerous for rescuing this one; the A/B testing guide puts it last in the reading order for that reason.

Incidents during the run

Tests run on a live site, and live sites have campaigns, outages, price changes and releases. If an event affects both arms equally it mostly adds noise; if it affects them unequally — an outage on the variant's template, a campaign that lands on a page one arm changed — it adds bias. Check the test log and the site's change log against the daily lift, looking for days where the lift jumps. Annotate everything; discard affected days when the effect was confined, and restart when it was not. Knowing about the incident at all depends on the log existing, which is why the experimentation hub lists the test log among the habits that make a practice.

Illustrative example — not a real result

A fashion retailer's vanishing win

A fashion retailer's test of a new size guide reports a 14% relative lift on add-to-basket after nine days and is rolled out. Three weeks later the add-to-basket rate is back at its pre-test level of 8.2%. The audit finds two faults. The split had drifted to 54/46 in the variant's favour, traced to a redirect that failed on an older in-app browser used by a share of mobile visitors, who all fell into the control; those visitors converted at 5.1%, dragging the control down. And the result had been checked every morning and stopped on the first day it crossed the significance line, with 11,000 visitors per arm against a planned 38,000. The retailer fixes the redirect, re-runs the test to the full sample over four weeks, and finds a lift of about 3% with an interval that only just clears zero: real, modest, and nothing like the headline.

Prevention

  • Run an A/A test on any new tool or site. Two identical arms show what noise looks like here and whether the tool's assignment and reporting can be trusted.
  • Check the sample ratio daily, by device and browser. The check takes a minute and catches the fault that invalidates everything else.
  • Walk the tracking in every arm before launch. Once with the debugger, once against the back office on the first full day.
  • Fix the sample and the reading date in the plan. Then monitor for defects and nothing else until the date.
  • Test for flicker on the slowest common device. If it flickers, fix it before counting.
  • Keep a programme calendar. Every running test on a map of the journey, so collisions are seen before launch. The A/B test launch checklist puts the collision, flicker, assignment and split checks into one gate before a test starts.
  • Name the segments in the plan. Anything not named is exploratory and goes into the next hypothesis, not this result.
  • Log incidents as they happen. Campaigns, releases, outages and price changes, with dates, next to the daily lift.
  • Read the primary KPI after rollout. The post-rollout rate is the only check on whether the test was right, and a programme that never looks never learns its false-positive rate.

The release itself, from rebuilding the variant in the site's own code to a holdback group and a dated reading for new and returning visitors, is set out in the tutorial on shipping a winning test variant.

Common mistakes

  • Analysing before checking assignment and tracking. No statistics repairs a broken denominator.
  • Rescuing a mismatched test with a filter. The mismatch means the arms differ; filtering hides it.
  • Trusting the tool's dashboard over the tag debugger. The dashboard shows consequences; the debugger shows causes.
  • Calling an early swing "volatile traffic". It is a small sample being read too often.
  • Reading the segment that won. It will not win next time.
  • Restarting without fixing. A restarted test with the same fault produces the same wrong answer, more slowly.
  • Not auditing rollouts. A win that did not deliver is the cheapest lesson a programme can get, if someone looks.
  • Blaming the audience or the tool. The data reflects how the test was assigned, tracked and read; the cause is in one of those three places, and the CVR only reports what reached it.

Frequently asked questions

How big does a sample ratio mismatch have to be before I worry?

Judge it by persistence and pattern rather than by size: a gap that holds day after day, or that appears only on one browser or device, is a defect even if it looks small, because something is removing visitors selectively. A tiny random wobble around the planned split is normal. Testing tools and statisticians use a formal test for this, and any persistent gap it flags means the arms are no longer comparable.

My test shows a winner on mobile but not on desktop. Can I ship the change for mobile only?

Only if the mobile segment was named in the plan and its sample alone reaches the planned size, which is rare. Otherwise treat it as a hypothesis: write a mobile-only test with its own plan and sample, and run it. Shipping on an unplanned segment result ships noise about as often as it ships a real effect.

Should I pause a test during a sale or a big campaign?

If both arms will see the campaign traffic equally, let it run and annotate the dates, because the result then includes the audience you will actually have during campaigns. If the campaign lands on a page that one arm changed, or brings an audience the hypothesis is not about, pause or exclude it, and extend the run so the planned sample is still reached in whole weeks.

Terms used on this page

Where this page is used

Part of these learning paths

Also related

Sources

  1. How Not To Run an A/B Test — Evan Miller

Related