Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Growth Marketing › Advanced

Checkpoint: Experiment Rigour at Scale

Scenario questions set in an invented note-taking app test the judgements of the first advanced growth module: whether a page can support a test at all, why early stopping manufactures wins, what a broken split reveals, how to run several tests without contamination, when a multivariate design earns its traffic, and how to treat a result found in one segment.
CheckpointAdvanced levelModule 1: Experiment rigour at scale4 minUpdated

What this checkpoint covers

Scribbet Notes is an invented note-taking app with a free plan and a paid upgrade. Its growth team tests the marketing site, the first-run experience inside the app and the upgrade page. The questions draw on the sample size and significance guide, the common testing pitfalls page and the comparison of A/B and multivariate testing, and they assume the tutoring marketplace audit.

Illustrative example — not a real result

Scribbet Notes: figures used in the questions

  • Upgrade page: about 3,000 visitors a week, 2% of whom upgrade. The team's calculator says a 10% relative lift needs about 78,000 visitors per arm.
  • First-run test planned at 50/50: after a week, 21,000 users in control and 19,000 in the variant, a gap that has held every day and sits mostly on Android.
  • Marketing home page: about 120,000 visitors a month. A test of two headlines, two screenshots and two buttons would need about 25,000 visitors in each of its eight combinations.

Task

Write an answer and its reason for each question before reading the explanations; it counts only if the reasoning matches. Then take the last test you ran and check it against every question.

Questions

  1. The team wants to test a new upgrade page layout and would ship it for a modest relative lift. Can the page support that test?
  2. A product manager checks a first-run test every morning. On the fourth day the result crosses the significance line and she wants to ship. What do you say?
  3. Look at the first-run split in the figures. What does it tell you, and what happens to the data collected so far?
  4. Three tests are proposed for the same fortnight: an onboarding checklist on the first-run screens, a template gallery on the same screens, and an email to existing users about a new feature. Which can run together?
  5. Marketing wants the headline, screenshot and button test on the home page; product wants to test a fully redesigned upgrade page. Which design suits each?
  6. A first-run test is flat overall, but tablet users show a large lift. The designer wants to ship it to tablets only. How do you treat that?
  7. A new editor toolbar leads strongly in its first days, and the lead shrinks steadily as the test runs. Most of the early lift comes from returning users. What is happening?

Answers explained

  1. No, not for a modest lift. At that traffic the required sample would take about a year to collect. The honest options are a bolder change with a larger expected effect, testing earlier in the Funnel where events are more frequent, or deciding on research and judgement and saying so. Running it anyway produces noise read as a result (when a test cannot answer the question).
  2. Keep running to the planned sample. Every look is another chance for noise to cross the line, so stopping at the first crossing ships changes that do nothing. Read once at the planned sample, or use a sequential method designed for repeated looks (peeking).
  3. Assignment is broken. A persistent gap that size is not chance: something is removing users from one arm, probably a variant that fails or redirects on some Android devices. The arms are no longer comparable, so find the cause, fix it, discard the data and restart; no filter rescues it (sample ratio mismatch).
  4. The email can run alongside; the two first-run tests cannot. Tests on the same screens interact, because each user's experience in one depends on their arm in the other. Run them as one test with combined arms if traffic allows, or in sequence. The email reaches a different audience at a different step (interacting tests).
  5. A/B for the redesign; multivariate only if the interaction matters. A redesign asks whether the whole new page is better, which an A/B Test answers with the least traffic. The home page can fill eight combinations in under two months, so a multivariate test is affordable there, but only worth it if the team suspects the best headline depends on the screenshot; otherwise two sequential A/B tests are faster and clearer (decision table).
  6. As a hypothesis. It was not named in the plan, its sample is a fraction of the planned one, and among enough slices one will always stand out. Write it up as a new test on tablets with its own sample, and ship nothing on this evidence (segment fishing).
  7. A novelty effect. Returning users react to anything new, and the effect fades as it wears off. Run long enough for the trend to flatten, give more weight to new users' results, and judge the change on its settled CVR, not the early spike (novelty effects).

Where to go next

If you missed the power or peeking questions, rework the sample arithmetic in the sample size and significance guide. If you missed the design questions, redo the classification step of the tutoring marketplace audit. Then continue to the incrementality module with the budgeting app exercise, or revisit the experimentation programme on the intermediate growth level.

Pages this lesson applies

Terms used on this page

Also related

Related