Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Conversion Rate Optimisation › Intermediate

Checkpoint: Sample Size, Significance and Pitfalls

Scenario questions on sizing a test before launch, reading p-values and confidence intervals honestly, and diagnosing the faults that make results wrong, from peeking to broken tracking. Every answer is explained in plain language with a link to the page that teaches it, and the task ends with naming the pitfall you are most likely to fall into.
CheckpointIntermediate levelModule 2: Statistics and test pitfalls4 minUpdated

What this checkpoint covers

The second module of the intermediate CRO level: baseline, minimum detectable effect, sample size, significance, power and confidence intervals from the guide to sample size and significance, and the diagnosis order from the guide to test pitfalls. What an A/B Test is and needs, as the A/B test glossary entry defines it, is the shared starting point. It follows the ticketing test exercise.

Task

  • Answer every question in a sentence or two, with your reason, before reading the explanations.
  • Compare your reasoning with each explanation and note anything you missed.
  • Write down the pitfall you are most likely to fall into in your own work, and one habit from the pitfalls guide's prevention list that would stop it. Keep the note with your next test plan.

Questions

  1. A team wants its test to finish in a fortnight, so it raises the minimum detectable effect until the calculator agrees. What has it traded away?
  2. A result comes back with a p-value just under the agreed level. The manager says the variant is almost certainly better. How do you correct this?
  3. A test reaches its planned sample. The confidence interval for the lift runs from a small loss to a large gain. The report says "no effect". Is that right?
  4. The arms of a planned even split hold noticeably different numbers of visitors, and the gap is there every day. Someone proposes reading the result anyway, since the rates are what matter. What do you say?
  5. A variant leads strongly in the first days, and its lead shrinks steadily as the test runs. Returning visitors make up much of the traffic. What is the likely explanation, and how would you check?
  6. A test is flat overall, but one country shows a large win. The country lead wants the variant shipped there. What do you advise?
  7. A page sees modest traffic and converts rarely. The calculator says the improvement the business cares about would take most of a year to detect. What are the honest options?
  8. The tool reports a strong win, but the back office shows almost no difference between the arms in completed orders. Where do you look first?

Answers explained

  1. Its ability to detect the modest effects most changes produce. Smaller effects need many more visitors, so a large detectable effect shortens the test by making it blind to realistic gains. The guide to sample size and significance calls the minimum detectable effect a business decision: the smallest gain worth shipping, not the one that fits the calendar.
  2. The p-value is how surprising the data would be if nothing had changed, not the probability that the variant is better. A result just under the line is not meaningfully different from one just over it. Report the confidence interval as the result, since it shows how large the true effect plausibly is.
  3. No: it is "could not tell". A wide interval straddling zero means the test lacked the data to separate the effect from noise. Only a narrow interval around zero supports "no meaningful effect", as the significance guide explains.
  4. Stop and find the cause. A persistent sample ratio mismatch means something is removing visitors from one arm selectively, so the arms are no longer comparable and their rates cannot be compared. The guide to test pitfalls puts it first in the diagnosis table: check the split by device and browser, fix the mechanism, discard the data and restart.
  5. A novelty effect. Returning visitors react to change itself, and the reaction fades. Plot the daily lift across the run and split new from returning visitors; run long enough for the trend to flatten and weight the judgement towards new visitors.
  6. Treat it as a hypothesis, not a result. Slice an audience enough ways and one slice will land far from the mean by chance. Check whether the country was named in the plan and whether its sample approaches the planned size, then test it with its own plan.
  7. Test a bolder change, test earlier in the journey where events are commoner, use research instead, or decide on judgement and say so. What does not work is running the test anyway and reading noise. The significance guide sets out each option.
  8. The tracking in each arm. A conversion event that fires twice, often on a redesigned confirmation page, inflates one arm. Walk the conversion path with the tag debugger and reconcile one closed day per arm with the back office, as the pitfalls guide recommends, before any reading of the A/B Test is trusted.

Where to go next

If you missed the statistics questions, reread the section on confidence intervals and rework one of your own past results as a range rather than a single figure. If you missed the pitfall questions, return to the ticketing test exercise and check its readout again in the pitfalls guide's order. Then start the third module, on checkout, product pages and running tests as a programme, with the checkout friction exercise.

Pages this lesson applies

Terms used on this page

Also related

Related