When to run this checklist
Run it once the test is designed and built and before any real visitor sees it, with the person who designed the test, the developer who built it and the analyst who will read it all available. Each group needs a different one of them, and a launch where any of the three is missing tends to skip that person's group. Run the launch group again on the day after the test starts, since its first line can only be ticked once real traffic has arrived.
This list is the switch-on gate for a single test. How a programme of tests is prioritised, resourced and reviewed is the experimentation programme playbook, which treats design and QA as one phase among several; this page expands that phase into lines to tick. The individual test designs in the experiment library each carry their own hypothesis and metrics, and this list applies to any of them without restating what they test.
Before you start
- The test brief, with the hypothesis, the primary metric, the guardrail metricsA metric an experiment must not worsen beyond an agreed bound, watched beside the primary metric that decides the test. and the audience. The writing test hypotheses guide explains what a usable one contains.
- The sample size calculation, with the inputs used, so the expected duration can be checked against the traffic the page actually receives.
- The test in preview or QA mode in the testing tool, on the real page, not a mock-up.
- A list of the main browsers and devices the page's visitors use, from analytics.
- The schedule of other tests and site releases, so overlaps can be spotted.
- The experiment log, ready to receive the new entry.
The checklist
The groups follow the order a test is made: designed, built, assigned, measured and launched. The design group is ticked before anyone looks at a result, because every line in it is a decision that becomes biased once numbers exist. A Sample SizeThe number of users or sessions an experiment needs to detect a given effect reliably. chosen after the test starts is not a sample size; it is a reason to stop.
Design
Build
Assignment
Tracking
Launch
Where each check is explained
The widget keeps each line short enough to tick; the reasoning sits in the guides. Each check links to the page that explains it.
| Check | Explained in |
|---|---|
| Write the hypothesis, primary metric and guardrails before launch | Writing test hypotheses |
| Calculate the sample size and the expected duration | Sample size and statistical significance |
| Write down the stopping rule and the decision rule | A/B testing guide |
| Check no other test runs on the same page and audience | Common testing pitfalls |
| Check every variant renders correctly on the main browsers and devices | A/B testing guide |
| Check the original does not flash before the variant loads | Common testing pitfalls |
| Confirm only the intended element differs between variants | A/B testing guide |
| Confirm visitors are assigned at random and stay in their variant | Common testing pitfalls |
| Exclude internal traffic and bots from the test | Sudden traffic spike in GA4 |
| Confirm the primary metric records in every variant | A/B testing guide |
| Pass the variant name to analytics for later segmentation | Events and parameters |
| Check the traffic split matches the plan after the first day | Common testing pitfalls |
| Log the test and its start date in the experiment log | Experimentation programme playbook |
If an item fails
Design failures hold the launch until they are written down. A missing stopping rule is the most common and the most damaging: without one, the test is checked daily and called the first time it crosses Statistical SignificanceA test result unlikely enough under 'no real difference' to pass a threshold chosen in advance., which is the pattern the sources list describes as the quickest way to a false winner. If the calculated duration turns out to be longer than the business will wait, the answer is a bolder change, a higher-traffic page or a different primary metric, decided now, not a shorter test decided later. The sample size and significance guide explains the trade-offs.
Build and tracking failures hold the launch until fixed. A variant that flickers, breaks on one browser or does not record the primary metric produces a result about the fault rather than the idea; if the CVRShare of visits or clicks that convert. in one variant depends on a tag that only fires in the other, no amount of traffic will rescue the comparison. Fix the build, then rerun the build and tracking groups together.
The launch group catches what only real traffic reveals. If the split on the first day is clearly off the plan, pause the test and investigate rather than letting it run: an uneven split usually means assignment or tracking is broken for some visitors, and a result from a broken test cannot be repaired afterwards. The testing pitfalls guide covers the usual causes.
A luggage brand's product page test
Vantorre Luggage, an online suitcase brand, plans a test of a new size guide on its product pages. The sample size calculation, based on about four thousand product page visitors a day, gives a duration of four weeks, and the stopping rule is written as "four full weeks, then decide". The build group fails on one line: on older phones the original page shows for a fraction of a second before the size guide appears. The developer moves the change server-side, and the flicker goes. The day after launch, the split reads close to even in the testing tool but analytics shows noticeably fewer sessions tagged with the variant, because the variant name was only passed on the first page view. The tracking is fixed, the test restarted, and the restart date recorded in the experiment log.
Common mistakes
- Deciding the rules afterwards. The primary metric, the duration and the decision rule are chosen once the numbers are in, and they are chosen to fit the numbers.
- Testing in the builder's preview only. The variant looks right in the editor and flashes the original on a real phone.
- Two tests, one audience. A pricing test and a navigation test share visitors, and neither result can be read on its own.
- Counting the team. Internal visits and bots land in both variants and dilute a real difference.
- Not checking the split. Assignment is broken for one browser, and the test runs for weeks before anyone looks.
- No log entry. The test ends, the result lives in a slide, and next year someone runs it again.
The experimentation programme playbook places this gate in the wider cycle of tests, and the checklists library holds the rest.
