Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Glossary term

Holdout Group

A randomly chosen part of an audience kept from exposure to measure what exposure caused.
AnalyticsconceptUpdated

Definition

A holdout group is the untreated arm of a controlled comparison. The audience that qualifies for a piece of marketing, such as everyone entering a welcome flow, every recent site visitor eligible for remarketing or every member of a loyalty programme, is split at random into two groups. One receives the marketing as usual; the other is excluded from it but otherwise treated normally. Because chance alone decided who went where, the two groups are alike in everything except exposure, so any difference in outcome can be credited to the marketing.

Randomness is the whole point. A comparison between people who saw an ad and people who did not, where the platform decided who saw it, compares two different populations: the people the platform chose are those it predicted were likeliest to convert. That is why attribution, which credits conversions that followed exposure, cannot answer the causal question, and why the incrementality testing guide treats the user-level holdout as the cleanest design wherever it can be run.

In practice

Holdouts are easiest where the business controls delivery:

  • Email and lifecycle flows. A random share of people entering a flow are held out at entry. The email metrics and testing guide recommends a small permanent holdout on the flows that carry the most volume or incentive, and a larger one-off holdout when a flow is redesigned.
  • Remarketing. Everyone who qualifies for the audience is assigned once to exposed or held out. The remarketing holdout test sets out that design, with conversions measured in the business's own data because platform reports cannot see the held-out group.
  • Paid social and other bought media. Platforms offer lift tools that create a randomised holdout inside their own audience, and geographic holdouts withhold ads from matched regions instead of people. The paid social sales playbook uses holdouts to decide how much of the platform-reported figure to believe before scaling spend.
  • Retention programmes, where effects build slowly, need holdouts kept for months. The experimentation loop for retention explains why a persistent holdout read at fixed cohort ages works better there than a short split test.

Size, duration and contamination

Three conditions decide whether the result means anything:

  • Size. The holdout must be large enough that the expected difference is bigger than random noise. A small flow may need months to accumulate a readable holdout, and a test sized after the fact is a guess.
  • Duration. The window must cover a full buying cycle so that held-out people have time to buy on their own.
  • Contamination. Held-out people must not receive the treatment by another route: shared devices, a second audience that includes them, or a colleague's campaign that targets the same list. Equally, they should not be excluded from everything else, or the test measures the whole programme instead of one part.

Reading the result

The output is two rates over the same window: conversions per person in the exposed group and in the holdout. Their difference, multiplied by the number of people exposed, is the incremental conversions; divided by the holdout rate, it is the relative lift. Incremental revenue then replaces reported revenue when return on spend is calculated, which is how the ROAS calculator guide for retargeting turns a holdout result into an incremental return that can be set against break-even.

Illustrative example — not a real result

Remarketing that mostly collected credit

Featherwend Mattresses splits 60,000 recent site visitors at random: 54,000 are eligible for remarketing and 6,000 are held out. Over six weeks the platform reports 1,080 purchases worth £648,000 from the exposed group, about £16 for every £1 of the £40,000 spent. In the business's own order data, 2.0% of the exposed group bought, against 1.85% of the holdout. The difference of 0.15 percentage points across 54,000 people is about 81 incremental purchases, not 1,080. At £150 of margin per order that is roughly £12,150, well below the cost. Featherwend cuts remarketing to a short window after a basket is abandoned and repeats the test there.

A holdout is how Incrementality is measured: the incremental effect is the gap between the groups. An A/B Test compares two versions of a treatment; a holdout compares a treatment with no treatment, though both rest on random assignment. Remarketing is the classic case for a holdout, because its audience was already likely to buy. Sample Size decides how large the holdout must be to detect the effect worth acting on.

Common confusions

  • A holdout is not any comparison group. Without random assignment it measures who was chosen, not what exposure did.
  • Platform-reported conversions are not the exposed group's result. Measure both groups in the same data source.
  • A null result is not proof of no effect. It may mean the test was too small to detect one.
  • Holding people out of everything tests the programme, not the channel. Exclude only the treatment.
  • A holdout and an A/B test are not the same design. One withholds; the other varies.

More terms to look up

Calculators

Where this term is used

Related