Why this matters
Nobody audits good news, which makes an apparent win the riskiest result a programme produces. The guide to sample size and significance explains why a test read before its planned sample is mostly noise, and the guide to test pitfalls gives the order to check faults in: assignment and tracking first, then the reading.
The test readout
Gigstub Tickets is an invented ticketing site for small venues and festivals. The team tested a new event page that shows the seat map and the full price, fees included, before the buy button. The product manager wants it shipped.
The plan, written before launch
| Field | Plan |
|---|---|
| Primary metric | Ticket purchase rate per visitor to event pages |
| Baseline | 4.3% |
| Smallest lift worth shipping | 10% relative |
| Visitors per arm | 36,600 |
| Split | Even, by visitor |
| Duration | Four full weeks, read once at the end |
At the planned split, 36,600 visitors per arm means 73,200 event-page visitors in total, which the site's usual traffic delivers in about 25 days, hence four full weeks.
What happened
The test started on a Thursday. The dashboard was checked every morning, and the test was stopped after nine days, on a Friday, when the tool first showed significance.
| Row label | Control | Variant |
|---|---|---|
| Visitors | 14,200 | 12,100 |
| Purchases counted by the test tool | 610 | 593 |
| Purchase rate in the tool | 4.30% | 4.90% |
| Tool's verdict | +14% relative, p = 0.02 | |
| Orders in the back office, by arm | 606 | 541 |
Visitors by browser type:
| Row label | Control | Variant |
|---|---|---|
| Desktop browsers | 6,100 | 6,000 |
| Mobile browsers | 6,200 | 6,100 |
| In-app browsers (links opened inside email and social apps) | 1,900 | 0 |
Daily relative lift in the tool: +22% across days 1 to 3, +15% across days 4 to 6, +8% across days 7 to 9.
On day 6 marketing sent a festival presale email linking to event pages. The product manager's note adds that "on mobile the variant is up 31%".
Checking validity
Work through the checks in the order the pitfalls guide gives, and do not read the lift until the first two pass.
- Assignment. Does the split match the plan, overall and by browser?
- Tracking. Do the tool's counts agree with the back office in each arm?
- Sample and timing. Was the planned sample reached, in whole weeks, read once?
- Incidents. Did anything during the run reach one arm more than the other?
- Reading. Only then: what does the primary metric show?
Task
- Calculate the observed split and say whether it is a sample ratio mismatch, with its likely cause.
- Reconcile the tool's purchases with the back office for each arm and explain the gap.
- Compare the sample with the plan and list every way the timing departed from it.
- Explain how the presale email interacts with the faults you have found.
- Recompute the purchase rates from the back office and state what they show.
- Decide whether to ship, rerun or stop, and write the next steps.
Worked answer
The split is broken. The variant arm received no visitors from in-app browsers, which points to a redirect that fails in those browsers and drops those visitors into the control. A persistent mismatch means the arms are no longer comparable, and no filter afterwards repairs it.
The split and the reconciliation
| Check | Control | Variant | Reading |
|---|---|---|---|
| Share of visitors | 14,200 ÷ 26,300 = 54.0% | 12,100 ÷ 26,300 = 46.0% | Against a planned 50/50 |
| Tool purchases minus back-office orders | 610 − 606 = 4 | 593 − 541 = 52 | The variant over-counts |
| Back-office purchase rate | 606 ÷ 14,200 = 4.27% | 541 ÷ 12,100 = 4.47% | +4.8% relative |
| Two-sided test on back-office orders | p ≈ 0.42 | ||
| Share of planned sample | 14,200 ÷ 36,600 = 39% | 12,100 ÷ 36,600 = 33% |
The tracking over-counts the variant. A small gap in the control is normal timing noise; a large one in the variant only means its new confirmation page fires the purchase event more than once, usually on reload. The tool was counting events, not tickets, so the headline lift was partly manufactured. The question the test exists to answer is the IncrementalityConversions caused by marketing. of the new page, the ticket sales it adds, and only the back office measures sales.
The timing broke the plan. Neither arm reached half its planned sample, the test ran nine days rather than four full weeks, so some weekdays appear twice and others once, and the result was read every morning and stopped at the first crossing. The shrinking daily lift is the shape of noise settling, possibly with novelty on top.
The email made it worse. Presale clicks opened in in-app browsers, and every one of those visitors landed in the control. The incident reached the arms unequally, which the pitfalls guide treats as bias rather than noise.
The honest reading. On back-office orders the variant's purchase CVRShare of visits or clicks that convert. is a little higher, with a difference well within what chance produces at this sample, and that is before the mismatch is accounted for. The test cannot say whether the page works. The mobile figure is a segment found after the fact; it becomes a hypothesis for a later test, not a result.
Decision: rerun, do not ship. The idea came from research and may well be right, so stopping it would waste the evidence. Fix the redirect and confirm the in-app split in a short A/A check, fix the confirmation tag and reconcile one closed day per arm with the back office, then restart the A/B TestControlled comparison of two variants. from zero for four full weeks and read it once. Log the email schedule in advance so any campaign during the run is annotated.
Common mistakes
- Reading the lift first. Assignment and tracking come before any statistics.
- Filtering out in-app visitors to "fix" the split. The mismatch means the arms differ; a filter hides it.
- Trusting the tool over the back office. The tool reports events; the business records tickets.
- Running the broken test on to its planned sample. Faulty data does not improve with volume.
- Shipping the mobile segment. Test it with its own plan; the pitfalls guide treats unplanned slices as hypotheses.
- Forgetting the basics. The A/B test glossary entry lists them: one variable, an adequate sample, a result read once against a threshold set in advance.
Next, take the sample size, significance and pitfalls checkpoint or return to the intermediate CRO level.
