Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Growth Marketing › Advanced

Auditing an Experiment Backlog for a Tutoring Marketplace

An invented tutoring marketplace has shipped a run of winning tests while sign-ups stayed flat. You audit its test log for power, peeking, sample ratio mismatch, interacting tests, segment claims and an unaffordable multivariate design, sort every result into trustworthy, inconclusive or invalid, and rewrite the rules the team tests by.
ExerciseAdvanced levelModule 1: Experiment rigour at scale5 minUpdated

Why this matters

A test log is a growth team's evidence base, and false winners in it corrupt every decision built on them. The intermediate level ran a programme; this level checks whether its answers deserve belief. The sample size and significance guide and the common testing pitfalls page are the references for every judgement below.

The marketplace and its test log

Wychmoor Tutors is an invented marketplace matching parents with private tutors. Parents arrive on the homepage or a subject landing page, search tutors, view profiles, create an account and book a trial lesson. Each step of that Funnel has been tested at least once.

Illustrative example — not a real result

Wychmoor Tutors: traffic and the test log

Monthly visitors: homepage 40,000; search results 50,000; tutor profiles 60,000; parents' landing page 18,000; sign-up form 12,000. Visit-to-sign-up rate across the site: 4.0% before the tests, 4.1% after every winner shipped.

TestPage and primary metricPlanned per armReached per armHow it endedReported
A: New hero headlineHomepage, sign-up17,0006,100Stopped on day five, the first day it was significant+18%, shipped
B: Booking button colourProfiles, trial bookingsNo plan9,000Four weeks, p = 0.21+4%, shipped as "directional"
C: Subject and price filtersSearch, profile views per search15,00015,200 and 15,050Read once at plan+6% (range +1% to +11%), shipped
D: Shorter sign-up formForm, completion8,0008,900 and 7,600Read at plan; split near 54/46 every day+12%, shipped
E: Parent testimonialsLanding page, sign-up10,00010,100Read at plan, alongside F+9%, shipped
F: Price per lesson, not per monthLanding page, sign-up10,00010,050Read at plan, alongside E−7%, dropped
G: Booking page redesignBooking, bookings12,00012,300Read once; range −3% to +4%Flat; "mobile +15%" noted
H: Homepage: 3 headlines × 2 images × 2 buttonsHomepage, sign-upNo plan28,000 in totalEnded after three weeks, nothing significant"Rerun for longer"

Sorting trustworthy results from noise

Check in the pitfalls page's order. Did the test measure what it claimed (assignment, tracking, sample ratio)? Was it read fairly (planned sample reached, read once, segments named in advance)? Did something else change the result (another test on the page, an incident, cells that could never fill)? A result is trustworthy when it passes all three, inconclusive when it was clean but could not separate the effect from noise, and invalid when the set-up or the reading broke the comparison.

Task

  1. Work out what the reported winners would have done to the sign-up rate if they were all real, and compare it with what happened.
  2. Classify each test as trustworthy, inconclusive or invalid, with the reason.
  3. For each invalid or inconclusive test, decide whether to rerun, redesign or drop it.
  4. Say what can and cannot be concluded from test G's mobile result.
  5. Rewrite the team's testing rules so the same faults cannot recur.

Worked answer

Illustrative example — not a real result

Wychmoor's audit arithmetic

  • Shipped "wins" on the sign-up path, compounded: 1.18 × 1.04 × 1.12 × 1.09 ≈ 1.50. A 4.0% rate would have become about 6.0%; it became 4.1%.
  • Test A stopped at 6,100 ÷ 17,000 ≈ 36% of its planned sample.
  • Test D: 8,900 and 7,600 against an expected 8,250 each. A gap that size, every day, is far beyond chance.
  • Test H: 12 combinations sharing about 28,000 visitors ≈ 2,300 per combination, against the 17,000 per arm the team's own calculator gave a single homepage headline test.
TestVerdictReasonNext step
AInvalidPeeked and stopped at the first significant dayRerun to the full sample; a headline is worth knowing
BInconclusiveNo plan, and a p-value nowhere near the lineDrop; a small change on a busy page is not a priority
CTrustworthyPlanned, balanced, read once, range clear of zeroKeep, but its metric is a proxy: confirm sign-ups did not fall
DInvalidSample ratio mismatch: one arm is losing visitorsFind the cause, fix it, discard the data, restart
E and FInvalidTwo tests on the same page and step interactedRerun as one test with four arms, or in sequence
GTrustworthy nullNarrow range around zero; no meaningful effectKeep the old page or ship on design grounds; no win claimed
HInvalid designCombinations could never reach a readable sampleReplace with an A/B Test of the two strongest headlines

Why sign-ups stayed flat. The three biggest shipped wins were a peeked result, a broken split and half of an interacting pair. None was evidence. The one clean win, test C, moved an early metric, and a better search-to-profile rate need not raise the sign-up CVR.

Test G's mobile result. A hypothesis, not a finding: unplanned, a fraction of the planned sample, and with enough slices one always looks like a winner. It becomes a new test with its own plan.

The new rules.

  • Every test is planned in the experiment hypothesis template before launch: primary metric, minimum detectable effect, sample, read date and named segments.
  • Tests that cannot reach their sample within a reasonable window are not run; the idea is made bolder, moved earlier in the journey or answered by research.
  • Read once at the planned sample, or use a sequential method as designed.
  • Check the sample ratio daily by device and browser, and walk the tracking in every arm before launch.
  • One test per page and step at a time, on a shared calendar, unless designed as one combined test.
  • A/B tests by default; multivariate only when every combination can fill, as the comparison of A/B and multivariate testing sets out.
  • Only the analyst declares a result, and every shipped win gets a post-rollout read, following the experimentation programme playbook.

Common mistakes

  • Rescuing a mismatched split with a filter. The arms already differ; restart.
  • Reading interacting tests separately. Each result depends on the other's arms.
  • Shipping "directional" results. A result that did not clear the bar is not a smaller win.
  • Treating an unplanned segment as the finding. It is the next test.

To revisit how a programme is set up in the first place, go back to the intermediate growth level; to test your judgement now, take the experiment rigour checkpoint.

Pages this lesson applies

Terms used on this page

Also related

Related