Why this matters
A test log is a growth team's evidence base, and false winners in it corrupt every decision built on them. The intermediate level ran a programme; this level checks whether its answers deserve belief. The sample size and significance guide and the common testing pitfalls page are the references for every judgement below.
The marketplace and its test log
Wychmoor Tutors is an invented marketplace matching parents with private tutors. Parents arrive on the homepage or a subject landing page, search tutors, view profiles, create an account and book a trial lesson. Each step of that FunnelModel of the customer journey stages. has been tested at least once.
Wychmoor Tutors: traffic and the test log
Monthly visitors: homepage 40,000; search results 50,000; tutor profiles 60,000; parents' landing page 18,000; sign-up form 12,000. Visit-to-sign-up rate across the site: 4.0% before the tests, 4.1% after every winner shipped.
| Test | Page and primary metric | Planned per arm | Reached per arm | How it ended | Reported |
|---|---|---|---|---|---|
| A: New hero headline | Homepage, sign-up | 17,000 | 6,100 | Stopped on day five, the first day it was significant | +18%, shipped |
| B: Booking button colour | Profiles, trial bookings | No plan | 9,000 | Four weeks, p = 0.21 | +4%, shipped as "directional" |
| C: Subject and price filters | Search, profile views per search | 15,000 | 15,200 and 15,050 | Read once at plan | +6% (range +1% to +11%), shipped |
| D: Shorter sign-up form | Form, completion | 8,000 | 8,900 and 7,600 | Read at plan; split near 54/46 every day | +12%, shipped |
| E: Parent testimonials | Landing page, sign-up | 10,000 | 10,100 | Read at plan, alongside F | +9%, shipped |
| F: Price per lesson, not per month | Landing page, sign-up | 10,000 | 10,050 | Read at plan, alongside E | −7%, dropped |
| G: Booking page redesign | Booking, bookings | 12,000 | 12,300 | Read once; range −3% to +4% | Flat; "mobile +15%" noted |
| H: Homepage: 3 headlines × 2 images × 2 buttons | Homepage, sign-up | No plan | 28,000 in total | Ended after three weeks, nothing significant | "Rerun for longer" |
Sorting trustworthy results from noise
Check in the pitfalls page's order. Did the test measure what it claimed (assignment, tracking, sample ratio)? Was it read fairly (planned sample reached, read once, segments named in advance)? Did something else change the result (another test on the page, an incident, cells that could never fill)? A result is trustworthy when it passes all three, inconclusive when it was clean but could not separate the effect from noise, and invalid when the set-up or the reading broke the comparison.
Task
- Work out what the reported winners would have done to the sign-up rate if they were all real, and compare it with what happened.
- Classify each test as trustworthy, inconclusive or invalid, with the reason.
- For each invalid or inconclusive test, decide whether to rerun, redesign or drop it.
- Say what can and cannot be concluded from test G's mobile result.
- Rewrite the team's testing rules so the same faults cannot recur.
Worked answer
Wychmoor's audit arithmetic
- Shipped "wins" on the sign-up path, compounded: 1.18 × 1.04 × 1.12 × 1.09 ≈ 1.50. A 4.0% rate would have become about 6.0%; it became 4.1%.
- Test A stopped at 6,100 ÷ 17,000 ≈ 36% of its planned sample.
- Test D: 8,900 and 7,600 against an expected 8,250 each. A gap that size, every day, is far beyond chance.
- Test H: 12 combinations sharing about 28,000 visitors ≈ 2,300 per combination, against the 17,000 per arm the team's own calculator gave a single homepage headline test.
| Test | Verdict | Reason | Next step |
|---|---|---|---|
| A | Invalid | Peeked and stopped at the first significant day | Rerun to the full sample; a headline is worth knowing |
| B | Inconclusive | No plan, and a p-value nowhere near the line | Drop; a small change on a busy page is not a priority |
| C | Trustworthy | Planned, balanced, read once, range clear of zero | Keep, but its metric is a proxy: confirm sign-ups did not fall |
| D | Invalid | Sample ratio mismatch: one arm is losing visitors | Find the cause, fix it, discard the data, restart |
| E and F | Invalid | Two tests on the same page and step interacted | Rerun as one test with four arms, or in sequence |
| G | Trustworthy null | Narrow range around zero; no meaningful effect | Keep the old page or ship on design grounds; no win claimed |
| H | Invalid design | Combinations could never reach a readable sample | Replace with an A/B TestControlled comparison of two variants. of the two strongest headlines |
Why sign-ups stayed flat. The three biggest shipped wins were a peeked result, a broken split and half of an interacting pair. None was evidence. The one clean win, test C, moved an early metric, and a better search-to-profile rate need not raise the sign-up CVRShare of visits or clicks that convert..
Test G's mobile result. A hypothesis, not a finding: unplanned, a fraction of the planned sample, and with enough slices one always looks like a winner. It becomes a new test with its own plan.
The new rules.
- Every test is planned in the experiment hypothesis template before launch: primary metric, minimum detectable effect, sample, read date and named segments.
- Tests that cannot reach their sample within a reasonable window are not run; the idea is made bolder, moved earlier in the journey or answered by research.
- Read once at the planned sample, or use a sequential method as designed.
- Check the sample ratio daily by device and browser, and walk the tracking in every arm before launch.
- One test per page and step at a time, on a shared calendar, unless designed as one combined test.
- A/B tests by default; multivariate only when every combination can fill, as the comparison of A/B and multivariate testing sets out.
- Only the analyst declares a result, and every shipped win gets a post-rollout read, following the experimentation programme playbook.
Common mistakes
- Rescuing a mismatched split with a filter. The arms already differ; restart.
- Reading interacting tests separately. Each result depends on the other's arms.
- Shipping "directional" results. A result that did not clear the bar is not a smaller win.
- Treating an unplanned segment as the finding. It is the next test.
To revisit how a programme is set up in the first place, go back to the intermediate growth level; to test your judgement now, take the experiment rigour checkpoint.
