Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Experiment Library

Remarketing Holdout Test

A holdout design for remarketing: the hypothesis, incremental conversions as the metric, cost and frequency guardrails, audience split and reading the lift.
Experimentadvanced6 min readUpdated

Hypothesis

Remarketing ads cause purchases among recent site visitors beyond those who would have returned anyway, so the exposed group will convert at a higher rate than a randomly held-out group. The hypothesis is deliberately sceptical, because Incrementality rather than credit is what decides the budget. Remarketing audiences are made of people who have already visited, browsed products or abandoned a basket, and a large share of them were always going to come back. The question is how many more return because of the ads.

That question cannot be answered by the platform's own reporting. Standard Attribution credits the ad for a conversion that follows exposure, whether or not the ad made the difference. The prospecting versus remarketing guide explains why remarketing often looks like the best-performing activity in an account and why that can be misleading. Write the hypothesis, the audience definition and the holdout share in the experiment hypothesis template before setting anything up.

Metrics and guardrails

The primary metric is incremental conversions: conversions in the exposed group minus the conversions the exposed group would have had without ads, estimated from the held-out group's conversion rate. Express it as a count and as a lift in conversion rate. Count conversions from your own systems where you can, matched to audience members, because the held-out group by definition generates no platform-reported conversions for the campaign.

Cost per incremental conversion is the first guardrail: remarketing spend divided by incremental conversions. It is almost always much higher than the platform-reported cost per conversion, and the useful question is whether it is still affordable. Frequency is the second guardrail. Remarketing audiences are small and often see ads many times; high frequency can annoy people who were going to buy anyway and, in some cases, make the exposed group convert less. Report it for the exposed group throughout.

Design

Randomise at user level within the remarketing audience. Every person who qualifies for the audience is assigned, at random and once, to exposed or held out. The exposed group receives remarketing as usual; the held-out group is excluded from remarketing campaigns but otherwise treated normally. Platform lift study tools, documented in the Meta Business Help Centre and Google Ads Help, do this assignment inside the platform. Where they are unavailable, split the audience yourself before uploading it, using the methods in the custom audiences guide.

Design elementChoiceReason
Unit of randomisationUser in the remarketing audienceHoldout must be random among people who qualify
VariantsExposed to remarketing versus held outMeasures the effect of the ads themselves
Holdout shareA minority of the audience, sized for the testLimits lost exposure while staying readable
Exclusion scopeHeld-out group excluded from all remarketing campaignsPrevents leaks
Conversion windowMatches the buying cycle from audience entryCaptures delayed purchases
MeasurementYour own conversion data joined to group membershipPlatform reports cannot see the holdout

The conversion window starts when each person enters the audience and lasts as long as your typical buying cycle. A short window misses slow buyers; an overly long one adds noise from unrelated purchases.

Sample size reasoning

Two forces make this test demanding. The remarketing audience already converts at a relatively high rate, so the baseline is high and the incremental difference may be a small share of it. And the held-out group is usually the smaller arm, which limits precision. The inputs are the held-out group's expected conversion rate, the smallest lift that would justify the spend, the holdout share and the confidence you require. The sample size and significance guide shows how a lopsided split affects the sample needed, and the conversion rate calculator gives the baseline rate.

If the audience is too small, run the test for longer to accumulate more people, or pool several remarketing campaigns into one test. The incrementality testing guide covers alternatives such as geo holdouts when user-level holdouts are not feasible.

Reading the results

After every audience member has completed the conversion window, compare conversion rates between exposed and held-out groups. The difference, multiplied by the size of the exposed group, gives incremental conversions. Divide remarketing spend by that figure to get the cost per incremental conversion, and compare it with the most the business can afford to pay.

Then set the result beside the platform's reported conversions for the same period. The ratio between incremental and reported conversions is one of the most useful numbers a team can have, because it lets you discount future platform reports for this activity. If the lift is positive and affordable, keep remarketing and consider whether lower frequency would achieve the same lift for less. If the lift is close to zero, remarketing is mostly collecting credit; reduce it and move budget to activity that creates demand. A negative lift is rare but possible with very high frequency.

Illustrative example — not a real result

An outdoor clothing shop measures its retargeting

Ridgeback Outfitters, an invented outdoor clothing retailer, holds out a random 20% of its basket abandoner remarketing audience for six weeks. About 48,000 people enter the audience. With a fourteen-day window, the held-out group converts at 6.0% and the exposed group at 7.1%. That is around 420 incremental purchases from £12,600 of spend, or £30 per incremental purchase, while the platform reported about 1,900 conversions at under £7 each. Frequency averaged high in the exposed group. Ridgeback keeps remarketing but caps frequency and cuts spend.

Pitfalls

Leaky holdouts are the first pitfall. If held-out people are excluded from one remarketing campaign but included in another, or are reached by a dynamic product ad set up separately, they are not truly held out, and the lift is understated. Audit every campaign that could reach the audience before launch.

Other channels retargeting the same people are the second. Email abandonment flows, retargeting on other platforms and affiliate networks may all reach the held-out group. That is acceptable if it is the normal state of the business, because the test then measures this channel's added effect on top of the others, but it must be described that way. The third pitfall is comparing the test result to platform reports and concluding the test is wrong. The gap between incremental and reported conversions is the finding, not an error; explain it to stakeholders before the result lands.

Once the holdout has produced an incremental revenue figure, the walkthrough on entering incremental revenue into the ROAS calculator shows how to read the resulting incremental return against break-even.

Common mistakes

Holdout tests often deliver unwelcome news, which makes them easy to dismiss on a technicality. Avoid giving anyone that technicality. The budget increase incrementality test asks the related question for prospecting spend, and the experiment library lists every design.

  • Trusting reported conversions. Attribution credits exposure, not cause.
  • A holdout that leaks. Every campaign that can reach the audience must exclude the held-out group.
  • A window shorter than the buying cycle. Slow buyers are missed in both groups.
  • Too small a holdout. The test cannot detect a modest lift and ends inconclusive.
  • Ignoring frequency. Over-exposure costs money and can suppress conversions.
  • Not recording the ratio. The incremental-to-reported ratio is what makes future reports usable.

Frequently asked questions

Why do platform reports overstate remarketing?

Remarketing reaches people who have already shown interest, many of whom would buy anyway. Attribution gives the ad credit when a conversion follows exposure, whether or not the ad changed anything. A holdout measures what would have happened without the ads.

How large should the holdout group be?

Large enough to measure the difference you care about, and no larger, because every held-out person is someone not receiving ads. The size depends on audience volume and the expected lift; the sample size guide explains the trade-off.

Can we use the platform's own lift study tools?

Yes, where available. Meta and Google document conversion lift and similar features that create a randomised holdout inside the platform. They are often the cleanest way to run this test, provided you also check the result against your own sales data.

Terms used on this page

Tools to try

Frameworks

Where this page is used

Part of these learning paths

Also related

Sources

  1. Meta Business Help Centre — Meta
  2. Google Ads Help — Google Ads Help

Related