Definition
A sample ratio mismatch is a difference between the planned and the observed allocation of visitors in an experiment that is too large or too persistent to be explained by random assignment. The planned ratio is written into the test plan before launch; the observed ratio is the count of visitors each arm actually received, taken from the testing tool and, where the test is also measured in analytics, from the exposure data there.
The idea rests on the logic of the A/B TestControlled comparison of two variants. itself. Randomisation is what makes the arms alike in everything except the change being tested, so a difference in outcomes can be attributed to the change. If the arms do not even contain the expected number of visitors, randomisation has been interfered with somewhere, and the assumption that the arms are alike no longer holds. The check is therefore a validity check, not a result: it says whether the experiment is working, before anyone asks whether the variant won.
In practice: the daily check by arm, device and browser
The A/B testing guide treats the sample ratio check as one of two checks that confirm the testing mechanism works (the other is an A/A test) and asks for it to be monitored daily during the run. The check is simple arithmetic: each arm's share of assigned visitors against the share the plan set. It is read three ways.
Over time. A random split wanders a little from day to day and settles. A mismatch shows up as a gap that keeps pointing the same way.
By device and browser. The mechanism behind a mismatch is often confined to one platform, so the blended split can look acceptable while one browser is badly skewed. The tutorial on analysing A/B tests in GA4 builds this view in an exploration, with date as a second dimension, and compares the exposure counts with the testing tool's own assignment counts.
Formally. Testing tools and statisticians apply a formal test to the observed counts, and the guide to common testing pitfalls advises judging a gap by persistence and pattern rather than by its size alone.
The check belongs in the paperwork too. The experiment results report template records it under validity checks beside the tracking checks, and the experimentation programme playbook has the developer and analyst look for it in the first days of every test, as a health check that can justify a restart.
Why a mismatch invalidates the result
The visitors who go missing from one arm, or arrive in it uninvited, are rarely a random slice. The pitfalls guide lists the usual mechanisms: a redirect that fails on some browsers so those visitors fall back to the control, bots that only ever see one arm, a variant that errors before the assignment event fires, a cookie set in one arm only, or an exclusion rule applied after assignment to one arm. Each one removes or adds a particular kind of visitor, such as people on an older browser or automated traffic, and those visitors convert differently from everyone else. The comparison then mixes the effect of the change with the effect of a different audience, and nothing in the analysis can separate the two. That is why a test with a persistent mismatch is not reweighted or trimmed: the fault is found and fixed, the data is discarded and the test runs again from the start.
A tea shop restarts a checkout test
Steepvantle Tea Merchants, an online loose-leaf tea shop, planned a 50/50 split for a test of a shorter checkout. After four days the testing tool showed 6,120 visitors in the control and 5,410 in the variant, a 53/47 split that had leaned the same way every day. Split by browser, the desktop and Android figures sat close to even, but on one mobile browser the variant held only 38% of visitors. The developer found that the variant's redirect failed on that browser, sending those shoppers back to the control. The team fixed the redirect, discarded the four days of data and restarted the test; the new split held between 49% and 51% each day.
Related terms
A sample ratio mismatch is a failure of the random assignment at the heart of an A/B TestControlled comparison of two variants., and it must be ruled out before Statistical SignificanceA test result unlikely enough under 'no real difference' to pass a threshold chosen in advance. is calculated, because a significance test assumes the arms are comparable. It is judged against the planned Sample SizeThe number of users or sessions an experiment needs to detect a given effect reliably. per arm, which fixes the counts each arm should reach. Like a Guardrail MetricA metric an experiment must not worsen beyond an agreed bound, watched beside the primary metric that decides the test., it is monitored during the run rather than at the end, but a guardrail watches for harm caused by the variant, while the sample ratio watches for a fault in the experiment itself.
Common confusions
- A small gap is harmless. Size is not the test; a gap that persists, or that appears on one browser or device only, is a defect even when it looks small.
- The result can be corrected afterwards. Reweighting the arms does not bring back the visitors that were lost, because they were lost for a reason related to who they are.
- Checking the split is peeking. Monitoring the split looks at the health of the experiment, not at the primary metric, so it does not bias the result.
- Any unequal split is a mismatch. A test planned with an uneven allocation is judged against that plan; the mismatch is the gap between plan and outcome, not between the arms.
- Analytics and the testing tool must agree exactly. Counts can differ between the two, for example through consent, without a mismatch, provided the difference is the same proportion in each arm.
