Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Checklists

Holdout Test Readiness Checklist

A pre-launch checklist for holdout, geo and lift tests of a channel, a mailing or a flow: the decision, the cells, keeping groups apart and the reading plan.
Checklistadvanced10 min readUpdated

When to run this checklist

Run it after a holdout, geo or lift test of a channel, a mailing or an automated flow has been designed and before the first day of the test, while the cells can still be redrawn. Whether the Holdout Group is a random share of a list or a set of matched regions in a Geo Experiment, the aim is the same: an estimate of Incrementality, what the activity caused rather than what it touched. Almost every line is free to settle at that point and impossible to settle afterwards: assignment, matching, sizing, exclusions and the reading date all describe the state of the test on its first day. Run it again for each new test, including a repeat of an earlier one, because the season, the spend and the other activity have changed.

The person who ticks it is whoever owns the test, usually an analyst or the marketing lead, with the channel owner for the account settings, the CRM or email owner for list holdouts, and whoever will act on the result for the first line. The incrementality testing guide owns the designs, cells, size, duration and reading; the guide to measuring what TikTok adds beyond last click applies them to a paid social channel and its lift studies; and the guide to measuring direct mail applies them to a holdout drawn from a mailing list. This list compresses what those three agree on.

Neighbouring lists take the cases this one does not. Tests that split website visitors between page variants are gated by the A/B test launch checklist. Holding back regions as one line of a television, radio, print or outdoor flight belongs to the offline media flight launch checklist, with the guide to measuring TV, radio, podcasts and print behind it. A holdout from an automated email flow follows the design notes in the email metrics and testing guide. The experiments library holds ready-made designs, such as the budget increase incrementality test, which runs the geo logic in reverse to measure what extra spend buys, and the direct mail format test, which draws one held-back group beside two mailed pieces so that each piece is judged on what it adds after its cost.

Before you start

  • The test design as written: the design chosen, the cells, the outcome and the period.
  • The decision the result will feed, and the name of the person who will make it.
  • The business's own outcome history by person or by region, for matching and sizing.
  • The final list or region set, after every selection and suppression rule has been applied.
  • Every campaign, flow and promotion planned for the channel and the site during the test period.
  • The platform's location settings and any lift study requirements from its help centre, if a paid channel is being tested.
  • The team's test log and a copy of the experiment results report template.

The checklist

The groups follow the order in which a test can go wrong: the question and the design, then the cells, then the separation of the groups, then the reading plan and the records. A line in the first four groups that cannot be ticked moves the start date; the records group is finished before day one, not after the result.

0/22

The question and the design

Cells

Keeping the groups apart

The reading plan

Records

Where each check is explained

The widget keeps the lines short so they can be ticked; the reasoning lives in the guides. Each check links to the page that explains it.

CheckExplained in
Decision namedIncrementality testing
Design chosen deliberatelyIncrementality testing
Withhold or reduce decidedIncrementality testing
Geo holdout or platform lift study chosenMeasuring what TikTok adds beyond last click
Lift study booked before the flightMeasuring what TikTok adds beyond last click
Holdout drawn only from those who could be treatedMeasuring direct mail
Random assignmentMeasuring direct mail
Geo cells matched and checked in a pre-periodIncrementality testing
Holdout sized from own volumesMeasuring direct mail
Holdout flagged in the CRMMeasuring direct mail
Channel withheld in location settingsMeasuring what TikTok adds beyond last click
Everything else equal for both groupsMeasuring direct mail
Other changes frozenIncrementality testing
Promotions and peaks avoidedIncrementality testing
Outcome from the business's own recordsMeasuring direct mail
Test period and cool-down setIncrementality testing
Reading date fixed, no early stopMeasuring direct mail
Range planned, not one numberIncrementality testing
Comparison with the attributed figure plannedIncrementality testing
Cells, outcome and smallest lift written downIncrementality testing
Test logged with design and ownerExperimentation programme playbook
Write-up planned in the report formatExperiment results report template

If an item fails

The cells group is the hard stop. A holdout drawn before the suppression rules have run mixes people who may not be mailed with people who were deliberately not mailed, and the comparison no longer means anything; the direct mail guide is explicit that people removed before the draw are a different group from the holdout, and that who may be mailed at all is a privacy question it leaves to the ICO's direct marketing guidance. Redraw the holdout from the final list. A holdout chosen by the last digit of an account number, by region convenience or by whoever the printer could not fit in is not random: redraw it with a random number per person, within each segment. Geo cells that did not move together in the pre-period are not yet a control; rematch them, or accept that the design has dropped a rung on the hierarchy of evidence and say so in the write-up. And if nobody can say how large the holdout must be for the smallest lift that matters, the test start waits for that calculation, because a result of no significant lift cannot be read without it.

For a mailing, the break-even calculator set up for direct mail gives the lift in order rate a segment needs to pay for itself, a natural floor for the smallest lift that matters.

In the separation group, a failure usually means a campaign that would reach the holdout. Check every paid campaign on the channel, not only the one being tested: TikTok's location targeting, documented in its help centre as the TikTok guide records, is what makes withheld regions possible, and a separate retargeting or always-on campaign targeting the whole country undoes it. A promotion or a site change planned inside the test period either moves the test or moves the change. These are fixable before day one and fatal afterwards.

In the reading plan, the reading date is the line that cannot slip. The direct mail guide uses Evan Miller's article on A/B testing to explain why a test read repeatedly and stopped when it looks good reports noise, and the incrementality guide applies the same rule to every holdout. Set the date, the cool-down for late conversions and the outcome before the start. The outcome comes from the business's own orders or leads, across every channel; a platform's conversion count can be compared with it, not substituted for it. Plan to report the lift with its Confidence Interval, as the guide to reading small numbers recommends for any figure built on modest counts.

A lift study that was not booked before the flight is not a stop for the test, only for that design. The TikTok guide notes that a platform study has to be booked before launch and has conditions of its own, documented in the help centre; if it is too late, the comparison of geo holdouts and conversion lift studies sets out what a geo design measures instead. The records group can be finished the day before the start, but it must be finished: a test whose cells, outcome and smallest lift that matters were not written down can no longer fail.

A cheese seller plans two holdouts in one quarter (illustrative)

Illustrative example — not a real result

Two holdouts, three fixes, two fixed dates

Zelbrumi Cheesemongers, an invented mail-order cheese seller, plans two tests for its winter quarter. The first withholds paid social from 6 regions matched to 6 treated regions on two years of weekly orders. The second holds back a random tenth of its seasonal catalogue list, around 2,600 of 26,000 customers, to see what the catalogue adds.

Running the list for the catalogue test, the analyst finds the holdout had been drawn from the full customer file, before the 1,400 people who had asked not to receive post were removed. The holdout is redrawn from the final list, within each segment, and every held-back customer is flagged in the CRM. For the geo test, an eight-week pre-period confirms the matched regions' orders moved together, but a check of every live campaign finds a national retargeting campaign that would have reached the withheld regions; it is set to exclude them in the platform's location settings.

Both reading dates are fixed before the first day: the geo test runs eight weeks with a two-week cool-down and is read on the Monday after, and the catalogue test is read six weeks after the drop. The smallest lift that would change each decision is written down: £1.50 of gross margin per catalogue recipient, and an incremental cost per order at or below £28 for paid social. Both tests go into the team's log, and the write-ups are planned in the results report format, with each lift reported as a range beside what the platform and the matchback had credited.

Common mistakes

  • Starting without a named decision. A test with no decision attached produces a number nobody acts on.
  • Drawing the holdout before the list is final. Suppressed people end up in the holdout and the comparison breaks.
  • Holding back whoever is convenient. Convenience correlates with something; only random assignment does not.
  • Forgetting the other campaigns. A retargeting or always-on campaign that reaches the withheld regions quietly ends the test.
  • Changing the channel or the site mid-test. Two changes, one result, no way to tell them apart.
  • Reading the platform's own count as the outcome. A geo test is worth running because it reads every order, online and offline, against the Baseline.
  • Stopping when the result looks good. Early results are the noisiest, and the reading date exists to stop this.
  • Reporting one number. A lift without its range invites a decision the test cannot support; the ROAS calculator turns each end of the range into an incremental return to show how far the decision can move.

The checklists library holds the lists that sit beside this one, from the A/B test launch to the offline media flight launch.

Frequently asked questions

Which tests does this checklist cover?

Any test that withholds a channel, a mailing or an automated flow from a comparable group of people or places and compares outcomes: a random holdout drawn from a mailing or email list, a holdout from an automated flow, a geo test that stops or reduces a paid channel in matched regions, and a platform's own lift study. Tests that split website visitors between page variants have their own launch checklist, and the regional holdback inside an offline media flight is a line of that flight's list.

Should we use the platform's lift study or a geo holdout?

They answer slightly different questions. A platform lift study is easier to run, because the platform splits its own audience, but it reads the outcomes the platform can see and is run by the party whose spend is being judged. A geo holdout is harder to run but reads the business's own outcomes by region, including sales that no tracking sees. The site's guides suggest choosing from what the decision needs, and note that a platform study has to be booked before the flight starts and has conditions of its own.

What if the business's volumes are too small for a readable test?

Find out before the start, not after. Work out from the business's own volumes how large the holdout or how many regions the smallest lift worth acting on would need. If the answer is out of reach, the options the guides describe are a larger holdout, a more frequent outcome measured as a proxy, a longer test period, or a weaker design accepted for what it is. Running an underpowered test and reading no significant lift as no effect is the outcome to avoid.

Terms used on this page

Tools to try

Frameworks

Where this page is used

Also related

Sources

  1. How Not To Run an A/B Test — Evan Miller
  2. TikTok Ads Help Center — TikTok for Business
  3. Direct marketing guidance — Information Commissioner's Office (UK)

Related