Why this matters
Social creates demand that search then captures, and both platforms claim the same orders, so only withholding social can show what it adds. The incrementality testing guide covers designs and how to read lift; the matched-region method in the bidding strategy geo split test carries over directly. At the intermediate level you tested one audience; here the unit is the whole channel and the decision is the split between channels.
The service and its attribution question
Petalcourt Flowers is an invented online florist delivering bouquets nationwide. Paid social is credited with a large share of orders; the finance lead suspects most of those buyers would have ordered anyway, through brand search or straight to the site.
Petalcourt Flowers: an average quiet-season week
| Channel | Weekly spend | Orders the platform credits |
|---|---|---|
| Paid social, prospecting and remarketing | £3,000 | 200 |
| Paid search, brand | £600 | 150 |
| Paid search, non-brand and Shopping | £2,400 | 170 |
| All orders in the order system, every source | 700 |
- Contribution per order before marketing: £16. Finance will pay at most £22 for an incremental order, allowing for repeat orders.
- Gifting peaks: Valentine's Day, Mother's Day and Christmas. Weeks 14 to 23 hold no peak.
- Orders are stored with the buyer's billing postcode and the recipient's delivery postcode.
Petalcourt Flowers: candidate regions over a 26-week pre-period
| Region | Weekly orders | Delivered to another region |
|---|---|---|
| London | 150 | 45% |
| South East | 95 | 30% |
| North West | 70 | 18% |
| East of England | 60 | 22% |
| West Midlands, Yorkshire, South West (each) | 55 | 15–17% |
| Scotland | 50 | 10% |
| East Midlands | 40 | 19% |
| Wales, North East (each) | 35 | 12–14% |
| Proposed pair | Week-to-week variation in their order ratio |
|---|---|
| North West and East of England | ±4% |
| West Midlands and South West | ±5% |
| Yorkshire and East Midlands | ±5% |
| Scotland and North East | ±11% |
From matched regions to a decision rule
Four choices decide a channel holdout. The outcome region is where the buyer saw the ads, which for gifts is not where the flowers went. The test must see an effect of the size that changes the decision, judged from pre-period wander. Search stays frozen, because search capturing less demand is part of what social adds. And the rule that turns the result into money is written before launch.
Task
- Choose the pairs you keep and the regions you leave out, and say why.
- Choose the outcome measure and where it is counted.
- Estimate the holdout regions' social spend and credited orders, the share of those orders social must cause to pay at finance's ceiling, and whether the pre-period noise lets the test see it.
- Set the timing, the length and the cool-down.
- Name the guardrails and what stays frozen.
- Write the decision rule for each outcome in next quarter's budget terms.
- Say whether mix modelling is worth adding, and when.
Worked answer
Regions. Keep the three pairs with tight pre-period fit and assign one region in each at random to the holdout. Leave out Scotland and North East, whose ratio wanders too much to read. Leave London and the South East out of the read: commuters and cross-border gifting blur them.
Outcome. All orders from the order system, every channel, assigned by billing postcode. A held-out region still receives flowers sent from elsewhere; counting by delivery postcode would hide the effect.
Petalcourt's holdout arithmetic, if North West, South West and East Midlands are drawn
- Holdout regions' weekly orders: 70 + 55 + 40 = 165, about 24% of all orders.
- Social spend withheld: 165 ÷ 700 × £3,000 ≈ £700 a week. Social orders credited there: 200 × 165 ÷ 700 ≈ 47.
- To pay at the ceiling, social must cause £700 ÷ £22 ≈ 32 orders a week: about 68% of what it is credited with, and a fall of 32 ÷ 165 ≈ 19% in holdout orders.
- If it causes 30% of its credit: about 14 orders, an 8.5% fall, costing about £50 per incremental order.
- Pre-period wander of about ±5% a week shrinks to roughly ±2% over six weeks, so a fall of around 4% or more is readable. The test can tell "pays" from "does not pay" comfortably.
- Test weeks 15 to 20, cool-down week 21, read in week 22.
Timing. Six weeks inside the quiet window, ending well before summer weddings, plus a week of cool-down for orders from late exposure. The result describes quiet weeks only; it says nothing yet about peak behaviour.
Guardrails and freezes. Brand search clicks and cost in the holdout regions show demand that social was feeding into search. Watch new-customer share and order value. Freeze search budgets, targets, offers and creative everywhere. Cut the social budget by the withheld share rather than letting the platform spend it in treated regions, which would change the spend level being judged. The holdout removes remarketing as well as prospecting; a later remarketing holdout can split the two.
Decision rule. Calculate lost orders against the control-based prediction and the cost per incremental order across its range. If the whole range sits below the ceiling, keep social and plan a budget-increase test next quarter. If the range straddles it, hold spend and rerun before the autumn. If the whole range sits above it, cut the quarter's social budget by a third, keep half the saving and retest at the lower level. Every outcome is reported with its IncrementalityConversions caused by marketing. factor, so routine AttributionAssigning conversion credit to touchpoints. reports can be discounted until the next test.
Mix modelling. Not yet. Petalcourt is almost entirely digital, its spend has barely varied, and it runs few channels, so the comparison of mix modelling and multi-touch attribution points to tests and clean attribution. Start keeping weekly spend, orders, prices and promotions by region now, and revisit MMMStatistical model of channel impact. if offline channels or several more paid channels join the plan, using this test to calibrate it.
Common mistakes
- Counting orders by delivery address. Gifts cross borders; the buyer saw the ad.
- Letting search bidding react in the holdout. Search capturing less is part of the effect.
- Moving the withheld budget into treated regions. The test then judges a different spend level.
- Deciding the threshold after the read. A rule written later can no longer fail.
- Applying a quiet-season result to Christmas. Retest before treating it as a peak answer.
