Why this matters
Reported return answers which ads were present when people bought, not which purchases the ads caused. The incrementality testing guide explains why only a withheld, comparable group can answer the budget question, and the remarketing holdout test shows the discipline applied to RemarketingAdvertising to previous visitors or engagers.: a sceptical hypothesis, a primary metric from your own data, guardrails, and a ratio of incremental to reported conversions that makes later reports usable. This exercise applies the same discipline to prospecting, where the audience is everyone and a user-level split is harder to keep clean.
The brand, its spend and its reported return
Puddleby Baby Clothing is an invented brand selling organic baby clothes across the UK. Prospecting has risen in steps this year; a separate remarketing campaign and an email welcome flow run alongside. The owner wants to double prospecting next month.
Puddleby Baby Clothing: spend history and economics
| Prospecting spend a month (two-month average) | Meta-reported purchases | Meta-reported revenue | New-customer orders, whole shop |
|---|---|---|---|
| £6,000 | 295 | £16,225 | 810 |
| £8,000 | 370 | £20,350 | 870 |
| £10,000 | 430 | £23,650 | 905 |
Average order £55; contribution margin 45%. Twelve-month contribution per new customer: £48. The owner will pay up to £40 for an incremental new customer. Most first purchases happen within 14 days of a first ad click. Orders are held by delivery postcode area for two years. Peak trading runs from November to December.
Designing the holdout and reading lift
A prospecting audience cannot be split by hand the way a remarketing list can. The choice is between the platform's own lift study, which randomises people inside Meta but measures only the conversions Meta observes and is run by the party being judged, and a geo holdout, which withholds prospecting from matched regions and reads every new customer in the shop's own records. Lift is the difference in outcomes between treated and control regions after scaling for their baseline, read with its range, never just its midpoint. The result describes the tested spend level only.
Task
- Calculate the average reported return, the break-even return on a first order, and the marginal cost per extra purchase and per extra new customer for each budget step.
- Choose the test design and justify it against the alternative.
- Specify the split, the primary metric, the guardrails and the duration, and say what else stays frozen.
- Set the minimum detectable effect and the decision rule before the test starts.
- Decide how far and how fast prospecting may scale if the test shows lift, and what happens if it does not.
Worked answer
Puddleby's averages and margins
- Average reported ROAS at £10,000: £23,650 ÷ £10,000 ≈ 2.37. First-order break-even: 1 ÷ 0.45 ≈ 2.22.
- Step to £8,000: £2,000 for 75 more reported purchases (≈ £27 each) but 60 more new customers shop-wide (≈ £33).
- Step to £10,000: £2,000 for 60 more reported purchases (≈ £33 each) but 35 more new customers (≈ £57).
- Doubling on the average would forecast about 860 reported purchases; the margin says far fewer.
Design. A geo holdout by postcode area. It counts new customers whether or not Meta saw them, which matters for a brand whose buyers are often gift-givers on several devices. A platform lift study can run alongside as a cross-check.
Split and budget. Pair postcode areas on their history of new-customer orders, randomly assign one of each pair to control, and confirm the pairs moved together before the test. Exclude control areas from prospecting and cut the budget in proportion, so treated areas keep the same spend per head. Without that cut, treated areas receive extra money and the test measures a different spend level.
Metric and guardrails. Primary metric: new-customer orders from the shop's records by delivery area. Guardrails: frequency in treated areas, which should stay flat because spend per area is unchanged, and cost per incremental new customer, the CPACost of each conversion. that matters, judged at the end. Remarketing and email run normally in both groups, and the plan says so: shrinking remarketing pools in control areas are part of what prospecting does.
Duration. A full buying cycle plus a cool-down so delayed purchases land, finishing before peak, with no promotions, price changes or new campaigns meanwhile.
Puddleby's test calendar and threshold
- Test: 6 weeks from early September, then a 2-week cool-down; read in late October.
- Treated spend at the current level: about £7,500 a month, so prospecting must add about 188 new customers a month in treated areas to sit at the £40 limit (£7,500 ÷ £40).
- Control areas: about a quarter of new-customer orders, so roughly 220 a month that the test must read against.
Decision rule. Written in the experiment hypothesis template before launch: the smallest lift worth detecting is the one that puts incremental cost per new customer at the owner's limit. Scale only if the pessimistic end of the range is inside the limit; hold and retest if the range straddles it; cut back if the range includes zero.
Scaling if lift appears. Not to double. The last step already cost more per extra new customer shop-wide than the owner's limit, so the marginal returns guide places the stopping point near current spend. Scale horizontally first with new concepts, then take moderate budget steps as the scaling campaigns guide describes, each held for the lag plus a week and judged on shop-wide new customers. To learn whether a higher level pays, run the budget increase incrementality test rather than assume it. If the test is inconclusive, step back to the level whose marginal return last cleared the limit and retest there. The IncrementalityConversions caused by marketing. factor from the test then discounts reported purchases until the next test.
Common mistakes
- Scaling on the average. The last increment is what the next one resembles.
- Excluding control regions without cutting budget. The treated regions get a spend increase nobody planned.
- Measuring the result in Ads Manager. Control regions have no Meta conversions to compare.
- Setting the threshold after the result. Then the test cannot fail.
- Testing into peak trading. Season moves both groups and swamps the effect.
Check your reasoning against the incrementality and scaling checkpoint, or revisit budget steps on the intermediate Meta Ads level.
