Why this matters
RemarketingAdvertising to previous visitors or engagers. reaches people who have already shown interest, and many of them were coming back anyway. Platform reports credit the ads for every purchase that follows exposure, so remarketing almost always looks cheap. The remarketing holdout test is the ready-made design for checking that credit, and the testing and scaling guide explains why a fair test needs randomisation, a fixed duration and a prediction written first. Here you turn both into a specific plan.
The retailer and its remarketing
Bolsterwren Furniture is an invented online retailer of sofas, tables and storage. Its marketing lead suspects the remarketing line takes credit for sales it did not cause, but the finance team likes its reported return and will not cut it on suspicion alone.
Bolsterwren Furniture: remarketing and order data
| Item | Value |
|---|---|
| Remarketing spend | £4,000 a month across social and display |
| Platform-reported purchases | 160 a month, about £25 each |
| Average order value | £600 |
| Contribution margin | 35%, so £210 per order |
| Owner's target cost per incremental order | £150, leaving profit after ad cost |
| People entering the audience (viewed a product, no purchase) | About 10,000 a month |
| Audience purchase rate within 28 days, from the order system | About 3% |
| Buying cycle | Nine in ten orders arrive within 28 days of the first product view |
Other activity reaching the same people:
- A basket-abandonment email flow, which runs for everyone.
- A dynamic product ads campaign on social, set up separately from the main remarketing campaign.
- A Performance Max campaign using site visitors as an audience signal.
Designing the holdout
A holdout is a form of A/B TestControlled comparison of two variants. in which one arm sees nothing. Four choices define it. The split assigns each person who qualifies for the audience, once and at random, to exposed or held out. The metric is incremental orders counted in Bolsterwren's own order system, because the held-out group generates no platform conversions to compare. The run time covers enough audience entries for the smallest worthwhile lift to be detectable, plus a full buying cycle for the last people to enter. The decision rule says in advance what cost per incremental order keeps, trims or cuts the line.
The hardest number is the smallest worthwhile lift: the lift at which the spend pays for itself, which comes from margin, not hope.
Task
- Decide the holdout share and how to keep the held-out group out of every campaign that could reach it.
- Name the primary metric and the guardrails, and say where each is measured.
- Calculate the smallest lift that would cover the spend.
- Estimate how long the test must run, using the sample size and significance guide.
- Write the decision rule: what result keeps, reduces or cuts remarketing, and what happens to the money.
- List what you would tell the finance team before the result lands.
Worked answer
Bolsterwren's holdout design and arithmetic
Smallest worthwhile lift:
- Incremental orders needed to cover spend: £4,000 ÷ £210 ≈ 19 a month.
- With a 20% holdout, about 8,000 people a month are exposed, so the lift must be at least 19 ÷ 8,000 ≈ 0.24 percentage points on a 3% baseline, roughly 8% in relative terms.
Sample and run time, at conventional confidence and power:
- A 20% holdout needs roughly 5,000 held-out people, so about 25,000 audience entries: around ten weeks of entries, then 28 days for the last entrants to complete the window. About fourteen weeks in total.
- A 30% holdout needs roughly 5,700 held-out people and about 19,000 entries: around eight weeks of entries plus the window, about twelve weeks in total, at the cost of more lost exposure.
Decision rule, on cost per incremental order (exposed-arm spend ÷ incremental orders):
- £150 or less: keep remarketing and test a lower frequency cap next.
- Between £150 and £210: keep it smaller, with a tighter cap and a shorter membership window.
- Above £210, or a lift indistinguishable from zero: cut to a small basket-abandoner line and move the money to prospecting, checked with a budget increase test.
Split. Choose the larger holdout. It shortens the test by weeks, and the exposure it gives up is worth less than the reported figures suggest, which is the point of the test. Build the held-out list once and exclude it from the main remarketing campaign and the separate dynamic product ads campaign, or the holdout leaks and the lift is understated. Performance Max uses visitors only as a signal, so it cannot be cleanly excluded; record that it continues for both groups. The email flow also continues for both, so the test measures remarketing's effect on top of email, and the write-up must say so.
Metric and guardrails. Incremental orders, joined to group membership in the order system, are primary. Cost per incremental order and frequency in the exposed group are guardrails, reported weekly but judged only at the end.
Run time. Fix the end date before launch and do not stop early because one arm leads. The window starts when each person enters the audience, matching the buying cycle.
Telling finance. The incremental figure will almost certainly be far lower than the platform's. That gap is the finding, not an error. Agree now that the ratio of incremental to reported orders will be recorded and used to discount future remarketing reports, which is what makes IncrementalityConversions caused by marketing. evidence useful beyond one decision. Write the plan into the experiment hypothesis template and record the outcome where the next budget review will see it.
Common mistakes
- A leaky holdout. Excluding the group from one campaign while another still reaches it understates the lift.
- A window shorter than the buying cycle. Slow furniture buyers are missed in both arms.
- Too small a holdout. The test ends inconclusive and the reported figures win by default.
- No decision rule. Without one, an unwelcome result is dismissed on a technicality.
- Reading platform conversions. The held-out group has none; only the order system can compare the arms.
The budget increase incrementality test asks the matching question for prospecting spend. Next, bring the whole level together in the performance marketing programme capstone, or return to the intermediate performance marketing level.
