Hypothesis
Remarketing ads cause purchases among recent site visitors beyond those who would have returned anyway, so the exposed group will convert at a higher rate than a randomly held-out group. The hypothesis is deliberately sceptical, because IncrementalityConversions caused by marketing. rather than credit is what decides the budget. RemarketingAdvertising to previous visitors or engagers. audiences are made of people who have already visited, browsed products or abandoned a basket, and a large share of them were always going to come back. The question is how many more return because of the ads.
That question cannot be answered by the platform's own reporting. Standard AttributionAssigning conversion credit to touchpoints. credits the ad for a conversion that follows exposure, whether or not the ad made the difference. The prospecting versus remarketing guide explains why remarketing often looks like the best-performing activity in an account and why that can be misleading. Write the hypothesis, the audience definition and the holdout share in the experiment hypothesis template before setting anything up.
Metrics and guardrails
The primary metric is incremental conversions: conversions in the exposed group minus the conversions the exposed group would have had without ads, estimated from the held-out group's conversion rate. Express it as a count and as a lift in conversion rate. Count conversions from your own systems where you can, matched to audience members, because the held-out group by definition generates no platform-reported conversions for the campaign.
Cost per incremental conversion is the first guardrail: remarketing spend divided by incremental conversions. It is almost always much higher than the platform-reported cost per conversion, and the useful question is whether it is still affordable. Frequency is the second guardrail. Remarketing audiences are small and often see ads many times; high frequency can annoy people who were going to buy anyway and, in some cases, make the exposed group convert less. Report it for the exposed group throughout.
Design
Randomise at user level within the remarketing audience. Every person who qualifies for the audience is assigned, at random and once, to exposed or held out. The exposed group receives remarketing as usual; the held-out group is excluded from remarketing campaigns but otherwise treated normally. Platform lift study tools, documented in the Meta Business Help Centre and Google Ads Help, do this assignment inside the platform. Where they are unavailable, split the audience yourself before uploading it, using the methods in the custom audiences guide.
| Design element | Choice | Reason |
|---|---|---|
| Unit of randomisation | User in the remarketing audience | Holdout must be random among people who qualify |
| Variants | Exposed to remarketing versus held out | Measures the effect of the ads themselves |
| Holdout share | A minority of the audience, sized for the test | Limits lost exposure while staying readable |
| Exclusion scope | Held-out group excluded from all remarketing campaigns | Prevents leaks |
| Conversion window | Matches the buying cycle from audience entry | Captures delayed purchases |
| Measurement | Your own conversion data joined to group membership | Platform reports cannot see the holdout |
The conversion window starts when each person enters the audience and lasts as long as your typical buying cycle. A short window misses slow buyers; an overly long one adds noise from unrelated purchases.
Sample size reasoning
Two forces make this test demanding. The remarketing audience already converts at a relatively high rate, so the baseline is high and the incremental difference may be a small share of it. And the held-out group is usually the smaller arm, which limits precision. The inputs are the held-out group's expected conversion rate, the smallest lift that would justify the spend, the holdout share and the confidence you require. The sample size and significance guide shows how a lopsided split affects the sample needed, and the conversion rate calculator gives the baseline rate.
If the audience is too small, run the test for longer to accumulate more people, or pool several remarketing campaigns into one test. The incrementality testing guide covers alternatives such as geo holdouts when user-level holdouts are not feasible.
Reading the results
After every audience member has completed the conversion window, compare conversion rates between exposed and held-out groups. The difference, multiplied by the size of the exposed group, gives incremental conversions. Divide remarketing spend by that figure to get the cost per incremental conversion, and compare it with the most the business can afford to pay.
Then set the result beside the platform's reported conversions for the same period. The ratio between incremental and reported conversions is one of the most useful numbers a team can have, because it lets you discount future platform reports for this activity. If the lift is positive and affordable, keep remarketing and consider whether lower frequency would achieve the same lift for less. If the lift is close to zero, remarketing is mostly collecting credit; reduce it and move budget to activity that creates demand. A negative lift is rare but possible with very high frequency.
An outdoor clothing shop measures its retargeting
Ridgeback Outfitters, an invented outdoor clothing retailer, holds out a random 20% of its basket abandoner remarketing audience for six weeks. About 48,000 people enter the audience. With a fourteen-day window, the held-out group converts at 6.0% and the exposed group at 7.1%. That is around 420 incremental purchases from £12,600 of spend, or £30 per incremental purchase, while the platform reported about 1,900 conversions at under £7 each. Frequency averaged high in the exposed group. Ridgeback keeps remarketing but caps frequency and cuts spend.
Pitfalls
Leaky holdouts are the first pitfall. If held-out people are excluded from one remarketing campaign but included in another, or are reached by a dynamic product ad set up separately, they are not truly held out, and the lift is understated. Audit every campaign that could reach the audience before launch.
Other channels retargeting the same people are the second. Email abandonment flows, retargeting on other platforms and affiliate networks may all reach the held-out group. That is acceptable if it is the normal state of the business, because the test then measures this channel's added effect on top of the others, but it must be described that way. The third pitfall is comparing the test result to platform reports and concluding the test is wrong. The gap between incremental and reported conversions is the finding, not an error; explain it to stakeholders before the result lands.
Once the holdout has produced an incremental revenue figure, the walkthrough on entering incremental revenue into the ROAS calculator shows how to read the resulting incremental return against break-even.
Common mistakes
Holdout tests often deliver unwelcome news, which makes them easy to dismiss on a technicality. Avoid giving anyone that technicality. The budget increase incrementality test asks the related question for prospecting spend, and the experiment library lists every design.
- Trusting reported conversions. Attribution credits exposure, not cause.
- A holdout that leaks. Every campaign that can reach the audience must exclude the held-out group.
- A window shorter than the buying cycle. Slow buyers are missed in both groups.
- Too small a holdout. The test cannot detect a modest lift and ends inconclusive.
- Ignoring frequency. Over-exposure costs money and can suppress conversions.
- Not recording the ratio. The incremental-to-reported ratio is what makes future reports usable.
