Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Conversion Rate Optimisation › Intermediate

Judging an A/B Test Result for a Ticketing Site

An exercise in auditing a test that looked like a win before anyone ships it. You compare the readout with the plan, check the split, the tracking and the timing, and decide whether to ship, rerun or stop, with the reasoning written down for a sceptical product manager.
ExerciseIntermediate levelModule 2: Statistics and test pitfalls5 minUpdated

Why this matters

Nobody audits good news, which makes an apparent win the riskiest result a programme produces. The guide to sample size and significance explains why a test read before its planned sample is mostly noise, and the guide to test pitfalls gives the order to check faults in: assignment and tracking first, then the reading.

The test readout

Gigstub Tickets is an invented ticketing site for small venues and festivals. The team tested a new event page that shows the seat map and the full price, fees included, before the buy button. The product manager wants it shipped.

Illustrative example — not a real result

The plan, written before launch

FieldPlan
Primary metricTicket purchase rate per visitor to event pages
Baseline4.3%
Smallest lift worth shipping10% relative
Visitors per arm36,600
SplitEven, by visitor
DurationFour full weeks, read once at the end

At the planned split, 36,600 visitors per arm means 73,200 event-page visitors in total, which the site's usual traffic delivers in about 25 days, hence four full weeks.

Illustrative example — not a real result

What happened

The test started on a Thursday. The dashboard was checked every morning, and the test was stopped after nine days, on a Friday, when the tool first showed significance.

Row labelControlVariant
Visitors14,20012,100
Purchases counted by the test tool610593
Purchase rate in the tool4.30%4.90%
Tool's verdict+14% relative, p = 0.02
Orders in the back office, by arm606541

Visitors by browser type:

Row labelControlVariant
Desktop browsers6,1006,000
Mobile browsers6,2006,100
In-app browsers (links opened inside email and social apps)1,9000

Daily relative lift in the tool: +22% across days 1 to 3, +15% across days 4 to 6, +8% across days 7 to 9.

On day 6 marketing sent a festival presale email linking to event pages. The product manager's note adds that "on mobile the variant is up 31%".

Checking validity

Work through the checks in the order the pitfalls guide gives, and do not read the lift until the first two pass.

  • Assignment. Does the split match the plan, overall and by browser?
  • Tracking. Do the tool's counts agree with the back office in each arm?
  • Sample and timing. Was the planned sample reached, in whole weeks, read once?
  • Incidents. Did anything during the run reach one arm more than the other?
  • Reading. Only then: what does the primary metric show?

Task

  1. Calculate the observed split and say whether it is a sample ratio mismatch, with its likely cause.
  2. Reconcile the tool's purchases with the back office for each arm and explain the gap.
  3. Compare the sample with the plan and list every way the timing departed from it.
  4. Explain how the presale email interacts with the faults you have found.
  5. Recompute the purchase rates from the back office and state what they show.
  6. Decide whether to ship, rerun or stop, and write the next steps.

Worked answer

The split is broken. The variant arm received no visitors from in-app browsers, which points to a redirect that fails in those browsers and drops those visitors into the control. A persistent mismatch means the arms are no longer comparable, and no filter afterwards repairs it.

Illustrative example — not a real result

The split and the reconciliation

CheckControlVariantReading
Share of visitors14,200 ÷ 26,300 = 54.0%12,100 ÷ 26,300 = 46.0%Against a planned 50/50
Tool purchases minus back-office orders610 − 606 = 4593 − 541 = 52The variant over-counts
Back-office purchase rate606 ÷ 14,200 = 4.27%541 ÷ 12,100 = 4.47%+4.8% relative
Two-sided test on back-office ordersp ≈ 0.42
Share of planned sample14,200 ÷ 36,600 = 39%12,100 ÷ 36,600 = 33%

The tracking over-counts the variant. A small gap in the control is normal timing noise; a large one in the variant only means its new confirmation page fires the purchase event more than once, usually on reload. The tool was counting events, not tickets, so the headline lift was partly manufactured. The question the test exists to answer is the Incrementality of the new page, the ticket sales it adds, and only the back office measures sales.

The timing broke the plan. Neither arm reached half its planned sample, the test ran nine days rather than four full weeks, so some weekdays appear twice and others once, and the result was read every morning and stopped at the first crossing. The shrinking daily lift is the shape of noise settling, possibly with novelty on top.

The email made it worse. Presale clicks opened in in-app browsers, and every one of those visitors landed in the control. The incident reached the arms unequally, which the pitfalls guide treats as bias rather than noise.

The honest reading. On back-office orders the variant's purchase CVR is a little higher, with a difference well within what chance produces at this sample, and that is before the mismatch is accounted for. The test cannot say whether the page works. The mobile figure is a segment found after the fact; it becomes a hypothesis for a later test, not a result.

Decision: rerun, do not ship. The idea came from research and may well be right, so stopping it would waste the evidence. Fix the redirect and confirm the in-app split in a short A/A check, fix the confirmation tag and reconcile one closed day per arm with the back office, then restart the A/B Test from zero for four full weeks and read it once. Log the email schedule in advance so any campaign during the run is annotated.

Common mistakes

  • Reading the lift first. Assignment and tracking come before any statistics.
  • Filtering out in-app visitors to "fix" the split. The mismatch means the arms differ; a filter hides it.
  • Trusting the tool over the back office. The tool reports events; the business records tickets.
  • Running the broken test on to its planned sample. Faulty data does not improve with volume.
  • Shipping the mobile segment. Test it with its own plan; the pitfalls guide treats unplanned slices as hypotheses.
  • Forgetting the basics. The A/B test glossary entry lists them: one variable, an adequate sample, a result read once against a threshold set in advance.

Next, take the sample size, significance and pitfalls checkpoint or return to the intermediate CRO level.

Pages this lesson applies

Terms used on this page

Also related

Related