When to run this checklist
Run it after a holdout, geo or lift test of a channel, a mailing or an automated flow has been designed and before the first day of the test, while the cells can still be redrawn. Whether the Holdout GroupA randomly chosen part of an audience kept from exposure to measure what exposure caused. is a random share of a list or a set of matched regions in a Geo ExperimentA test that runs a campaign in some regions and holds it back in comparable ones, then compares total outcomes, where people cannot be randomised one by one., the aim is the same: an estimate of IncrementalityConversions caused by marketing., what the activity caused rather than what it touched. Almost every line is free to settle at that point and impossible to settle afterwards: assignment, matching, sizing, exclusions and the reading date all describe the state of the test on its first day. Run it again for each new test, including a repeat of an earlier one, because the season, the spend and the other activity have changed.
The person who ticks it is whoever owns the test, usually an analyst or the marketing lead, with the channel owner for the account settings, the CRM or email owner for list holdouts, and whoever will act on the result for the first line. The incrementality testing guide owns the designs, cells, size, duration and reading; the guide to measuring what TikTok adds beyond last click applies them to a paid social channel and its lift studies; and the guide to measuring direct mail applies them to a holdout drawn from a mailing list. This list compresses what those three agree on.
Neighbouring lists take the cases this one does not. Tests that split website visitors between page variants are gated by the A/B test launch checklist. Holding back regions as one line of a television, radio, print or outdoor flight belongs to the offline media flight launch checklist, with the guide to measuring TV, radio, podcasts and print behind it. A holdout from an automated email flow follows the design notes in the email metrics and testing guide. The experiments library holds ready-made designs, such as the budget increase incrementality test, which runs the geo logic in reverse to measure what extra spend buys, and the direct mail format test, which draws one held-back group beside two mailed pieces so that each piece is judged on what it adds after its cost.
Before you start
- The test design as written: the design chosen, the cells, the outcome and the period.
- The decision the result will feed, and the name of the person who will make it.
- The business's own outcome history by person or by region, for matching and sizing.
- The final list or region set, after every selection and suppression rule has been applied.
- Every campaign, flow and promotion planned for the channel and the site during the test period.
- The platform's location settings and any lift study requirements from its help centre, if a paid channel is being tested.
- The team's test log and a copy of the experiment results report template.
The checklist
The groups follow the order in which a test can go wrong: the question and the design, then the cells, then the separation of the groups, then the reading plan and the records. A line in the first four groups that cannot be ticked moves the start date; the records group is finished before day one, not after the result.
The question and the design
Cells
Keeping the groups apart
The reading plan
Records
Where each check is explained
The widget keeps the lines short so they can be ticked; the reasoning lives in the guides. Each check links to the page that explains it.
| Check | Explained in |
|---|---|
| Decision named | Incrementality testing |
| Design chosen deliberately | Incrementality testing |
| Withhold or reduce decided | Incrementality testing |
| Geo holdout or platform lift study chosen | Measuring what TikTok adds beyond last click |
| Lift study booked before the flight | Measuring what TikTok adds beyond last click |
| Holdout drawn only from those who could be treated | Measuring direct mail |
| Random assignment | Measuring direct mail |
| Geo cells matched and checked in a pre-period | Incrementality testing |
| Holdout sized from own volumes | Measuring direct mail |
| Holdout flagged in the CRM | Measuring direct mail |
| Channel withheld in location settings | Measuring what TikTok adds beyond last click |
| Everything else equal for both groups | Measuring direct mail |
| Other changes frozen | Incrementality testing |
| Promotions and peaks avoided | Incrementality testing |
| Outcome from the business's own records | Measuring direct mail |
| Test period and cool-down set | Incrementality testing |
| Reading date fixed, no early stop | Measuring direct mail |
| Range planned, not one number | Incrementality testing |
| Comparison with the attributed figure planned | Incrementality testing |
| Cells, outcome and smallest lift written down | Incrementality testing |
| Test logged with design and owner | Experimentation programme playbook |
| Write-up planned in the report format | Experiment results report template |
If an item fails
The cells group is the hard stop. A holdout drawn before the suppression rules have run mixes people who may not be mailed with people who were deliberately not mailed, and the comparison no longer means anything; the direct mail guide is explicit that people removed before the draw are a different group from the holdout, and that who may be mailed at all is a privacy question it leaves to the ICO's direct marketing guidance. Redraw the holdout from the final list. A holdout chosen by the last digit of an account number, by region convenience or by whoever the printer could not fit in is not random: redraw it with a random number per person, within each segment. Geo cells that did not move together in the pre-period are not yet a control; rematch them, or accept that the design has dropped a rung on the hierarchy of evidence and say so in the write-up. And if nobody can say how large the holdout must be for the smallest lift that matters, the test start waits for that calculation, because a result of no significant lift cannot be read without it.
For a mailing, the break-even calculator set up for direct mail gives the lift in order rate a segment needs to pay for itself, a natural floor for the smallest lift that matters.
In the separation group, a failure usually means a campaign that would reach the holdout. Check every paid campaign on the channel, not only the one being tested: TikTok's location targeting, documented in its help centre as the TikTok guide records, is what makes withheld regions possible, and a separate retargeting or always-on campaign targeting the whole country undoes it. A promotion or a site change planned inside the test period either moves the test or moves the change. These are fixable before day one and fatal afterwards.
In the reading plan, the reading date is the line that cannot slip. The direct mail guide uses Evan Miller's article on A/B testing to explain why a test read repeatedly and stopped when it looks good reports noise, and the incrementality guide applies the same rule to every holdout. Set the date, the cool-downA stretch after a holdout's treatment stops and before the final reading, so orders still on their way land in both groups and a lasting effect can be watched. for late conversionsThe time between an ad interaction and the conversion it leads to, which leaves the most recent reporting days incomplete. and the outcome before the start. The outcome comes from the business's own orders or leads, across every channel; a platform's conversion count can be compared with it, not substituted for it. Plan to report the lift with its Confidence IntervalA range of values for an effect, such as a lift in conversion rate, that is consistent with the data at a chosen confidence level., as the guide to reading small numbers recommends for any figure built on modest counts.
A lift study that was not booked before the flight is not a stop for the test, only for that design. The TikTok guide notes that a platform study has to be booked before launch and has conditions of its own, documented in the help centre; if it is too late, the comparison of geo holdouts and conversion lift studies sets out what a geo design measures instead. The records group can be finished the day before the start, but it must be finished: a test whose cells, outcome and smallest lift that matters were not written down can no longer fail.
A cheese seller plans two holdouts in one quarter (illustrative)
Two holdouts, three fixes, two fixed dates
Zelbrumi Cheesemongers, an invented mail-order cheese seller, plans two tests for its winter quarter. The first withholds paid social from 6 regions matched to 6 treated regions on two years of weekly orders. The second holds back a random tenth of its seasonal catalogue list, around 2,600 of 26,000 customers, to see what the catalogue adds.
Running the list for the catalogue test, the analyst finds the holdout had been drawn from the full customer file, before the 1,400 people who had asked not to receive post were removed. The holdout is redrawn from the final list, within each segment, and every held-back customer is flagged in the CRM. For the geo test, an eight-week pre-period confirms the matched regions' orders moved together, but a check of every live campaign finds a national retargeting campaign that would have reached the withheld regions; it is set to exclude them in the platform's location settings.
Both reading dates are fixed before the first day: the geo test runs eight weeks with a two-week cool-down and is read on the Monday after, and the catalogue test is read six weeks after the drop. The smallest lift that would change each decision is written down: £1.50 of gross margin per catalogue recipient, and an incremental cost per order at or below £28 for paid social. Both tests go into the team's log, and the write-ups are planned in the results report format, with each lift reported as a range beside what the platform and the matchback had credited.
Common mistakes
- Starting without a named decision. A test with no decision attached produces a number nobody acts on.
- Drawing the holdout before the list is final. Suppressed people end up in the holdout and the comparison breaks.
- Holding back whoever is convenient. Convenience correlates with something; only random assignment does not.
- Forgetting the other campaigns. A retargeting or always-on campaign that reaches the withheld regions quietly ends the test.
- Changing the channel or the site mid-test. Two changes, one result, no way to tell them apart.
- Reading the platform's own count as the outcome. A geo test is worth running because it reads every order, online and offline, against the BaselineAn estimate of what a metric would have done without a given change, campaign or promotion, against which the effect is measured..
- Stopping when the result looks good. Early results are the noisiest, and the reading date exists to stop this.
- Reporting one number. A lift without its range invites a decision the test cannot support; the ROAS calculator turns each end of the range into an incremental return to show how far the decision can move.
The checklists library holds the lists that sit beside this one, from the A/B test launch to the offline media flight launch.
