Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Checklists

A/B Test Launch Checklist

A launch checklist for an A/B test: hypothesis and stopping rule, sample size, variant rendering and flicker, assignment, tracking and the experiment log.
Checklistintermediate6 min readUpdated

When to run this checklist

Run it once the test is designed and built and before any real visitor sees it, with the person who designed the test, the developer who built it and the analyst who will read it all available. Each group needs a different one of them, and a launch where any of the three is missing tends to skip that person's group. Run the launch group again on the day after the test starts, since its first line can only be ticked once real traffic has arrived.

This list is the switch-on gate for a single test. How a programme of tests is prioritised, resourced and reviewed is the experimentation programme playbook, which treats design and QA as one phase among several; this page expands that phase into lines to tick. The individual test designs in the experiment library each carry their own hypothesis and metrics, and this list applies to any of them without restating what they test.

Before you start

  • The test brief, with the hypothesis, the primary metric, the guardrail metrics and the audience. The writing test hypotheses guide explains what a usable one contains.
  • The sample size calculation, with the inputs used, so the expected duration can be checked against the traffic the page actually receives.
  • The test in preview or QA mode in the testing tool, on the real page, not a mock-up.
  • A list of the main browsers and devices the page's visitors use, from analytics.
  • The schedule of other tests and site releases, so overlaps can be spotted.
  • The experiment log, ready to receive the new entry.

The checklist

The groups follow the order a test is made: designed, built, assigned, measured and launched. The design group is ticked before anyone looks at a result, because every line in it is a decision that becomes biased once numbers exist. A Sample Size chosen after the test starts is not a sample size; it is a reason to stop.

0/13

Design

Build

Assignment

Tracking

Launch

Where each check is explained

The widget keeps each line short enough to tick; the reasoning sits in the guides. Each check links to the page that explains it.

CheckExplained in
Write the hypothesis, primary metric and guardrails before launchWriting test hypotheses
Calculate the sample size and the expected durationSample size and statistical significance
Write down the stopping rule and the decision ruleA/B testing guide
Check no other test runs on the same page and audienceCommon testing pitfalls
Check every variant renders correctly on the main browsers and devicesA/B testing guide
Check the original does not flash before the variant loadsCommon testing pitfalls
Confirm only the intended element differs between variantsA/B testing guide
Confirm visitors are assigned at random and stay in their variantCommon testing pitfalls
Exclude internal traffic and bots from the testSudden traffic spike in GA4
Confirm the primary metric records in every variantA/B testing guide
Pass the variant name to analytics for later segmentationEvents and parameters
Check the traffic split matches the plan after the first dayCommon testing pitfalls
Log the test and its start date in the experiment logExperimentation programme playbook

If an item fails

Design failures hold the launch until they are written down. A missing stopping rule is the most common and the most damaging: without one, the test is checked daily and called the first time it crosses Statistical Significance, which is the pattern the sources list describes as the quickest way to a false winner. If the calculated duration turns out to be longer than the business will wait, the answer is a bolder change, a higher-traffic page or a different primary metric, decided now, not a shorter test decided later. The sample size and significance guide explains the trade-offs.

Build and tracking failures hold the launch until fixed. A variant that flickers, breaks on one browser or does not record the primary metric produces a result about the fault rather than the idea; if the CVR in one variant depends on a tag that only fires in the other, no amount of traffic will rescue the comparison. Fix the build, then rerun the build and tracking groups together.

The launch group catches what only real traffic reveals. If the split on the first day is clearly off the plan, pause the test and investigate rather than letting it run: an uneven split usually means assignment or tracking is broken for some visitors, and a result from a broken test cannot be repaired afterwards. The testing pitfalls guide covers the usual causes.

Illustrative example — not a real result

A luggage brand's product page test

Vantorre Luggage, an online suitcase brand, plans a test of a new size guide on its product pages. The sample size calculation, based on about four thousand product page visitors a day, gives a duration of four weeks, and the stopping rule is written as "four full weeks, then decide". The build group fails on one line: on older phones the original page shows for a fraction of a second before the size guide appears. The developer moves the change server-side, and the flicker goes. The day after launch, the split reads close to even in the testing tool but analytics shows noticeably fewer sessions tagged with the variant, because the variant name was only passed on the first page view. The tracking is fixed, the test restarted, and the restart date recorded in the experiment log.

Common mistakes

  • Deciding the rules afterwards. The primary metric, the duration and the decision rule are chosen once the numbers are in, and they are chosen to fit the numbers.
  • Testing in the builder's preview only. The variant looks right in the editor and flashes the original on a real phone.
  • Two tests, one audience. A pricing test and a navigation test share visitors, and neither result can be read on its own.
  • Counting the team. Internal visits and bots land in both variants and dilute a real difference.
  • Not checking the split. Assignment is broken for one browser, and the test runs for weeks before anyone looks.
  • No log entry. The test ends, the result lives in a slide, and next year someone runs it again.

The experimentation programme playbook places this gate in the wider cycle of tests, and the checklists library holds the rest.

Terms used on this page

Where this page is used

Sources

  1. How Not To Run an A/B Test — Evan Miller

Related