Why this matters
Data-driven AttributionAssigning conversion credit to touchpoints. is generous to channels that reach people already on their way to buying, and the data-driven attribution guide explains why it cannot see what would have happened anyway. The incrementality testing guide gives the remedy: withhold the channel from comparable places and measure the difference. With a handful of regions, the design decides whether the test can answer anything at all.
The firm and its regions
Orchardgate Juice Delivery is an invented cold-pressed juice subscription delivering to eight zones. New subscribers sign up online and are recorded by delivery postcode in the subscription system. The owner's attribution report credits paid social with most new subscriptions and she wants to know how many it causes.
Orchardgate Juice Delivery: zones over a twenty-six-week pre-period
| Zone | New subscriptions per week | Paid social spend per week | Notes |
|---|---|---|---|
| Harbourside | 140 | £2,800 | Adjacent to Old Town; heavy commuting between them |
| Old Town | 130 | £2,600 | Adjacent to Harbourside |
| Riverside | 95 | £1,900 | Weekly sign-ups track Hillcrest closely |
| Hillcrest | 90 | £1,800 | Tracks Riverside |
| Westfield | 70 | £1,400 | Weekly sign-ups track Eastmoor closely |
| Eastmoor | 65 | £1,300 | Tracks Westfield |
| Southbay | 50 | £1,000 | Poster campaign booked during the planned test window |
| Northgate | 45 | £900 | Delivery rounds close to full capacity |
- Data-driven attribution credits paid social with 55% of new subscriptions.
- Weekly ratio between each tracking pair wanders within about ±8%.
- Contribution per subscriber per week: £12. Average subscription length: 20 weeks.
- The owner will pay at most half a subscriber's contribution to acquire one.
Choosing cells, duration and the metric
Regions belong in the test only if they can be compared and kept apart. Pairs that moved together in the pre-period make good matches; neighbours with heavy commuting leak exposure across the border; and anything else planned for one region and not its pair, from posters to capacity limits, will be read as the effect. The minimum detectable effect should come from the decision, not from convenience: the smallest loss of sign-ups that would make the spend worth keeping. If the pre-period wander is larger than that effect, no amount of analysis afterwards will rescue the read.
The metric should count every sign-up a region produces, whatever channel gets the credit. That is why a geo design fits better here than a user-level holdout like the remarketing holdout test, which suits audiences the business owns.
Task
- Choose the holdout and control regions, and say which zones stay out of the read and why.
- Calculate the minimum detectable effect from the owner's cost limit and compare it with the pre-period wander.
- Set the primary metric, the guardrails and any stopping rule.
- Set the duration, the cool-down and the timing.
- Write the decision rule for each possible outcome.
- Explain how the result will adjust attribution reporting and later calibrate a mix model.
Worked answer
Regions. Use the two tracking pairs, Riverside with Hillcrest and Westfield with Eastmoor, and toss a coin within each pair for which zone loses paid social. Leave Harbourside and Old Town out of the read: commuting would carry exposure into whichever one is held out. Exclude Southbay because of the posters and Northgate because full delivery rounds would cap sign-ups in its pair. Keep paid social running in excluded zones, and say that the result describes the smaller zones only.
Minimum detectable effect from the owner's limit
Value of a subscriber: £12 × 20 weeks = £240. Maximum cost per incremental subscriber: £120. Suppose the coin picks Hillcrest and Eastmoor. Spend withheld: £1,800 + £1,300 = £3,100 a week. To be worth that spend, paid social must cause at least £3,100 ÷ £120 ≈ 26 sign-ups a week in those zones, out of 90 + 65 = 155: a drop of about 17%.
Pre-period wander: about ±8% in a single week, so a sustained 17% gap over several weeks stands clear of it. If attribution were right, the drop would be near 55%.
Metric and guardrails. Primary: new subscriptions per week by delivery postcode, from the subscription system, held-out zone against the prediction from its pair. Guardrails: brand search and direct sign-ups by zone, to see whether demand shifted channel rather than disappeared; first-month cancellations, so lost sign-ups are not just poor-quality ones; and delivery capacity in the control zones. The only stopping rule is business harm agreed in advance, such as a fall far beyond the attributed share for consecutive weeks; never stop because an early week looks decisive.
Duration. Six weeks of holdout, long enough to cover the time from first seeing an ad to signing up, then two weeks of cool-down with spend restored so delayed sign-ups land in both groups. Avoid the new-year spike. Freeze creative, offers and prices in all four zones throughout.
Decision rule. Calculate incremental sign-ups and incremental cost per sign-up with a range, not a point. If even the pessimistic end of the cost sits under the limit, keep spend and test a higher level. If the midpoint sits above it, cut by a third and retest. Anything between: hold spend and rerun with more weeks or zones.
Calibration. Divide incremental sign-ups by the sign-ups attribution credited to paid social in the comparable zones over the same weeks. That IncrementalityConversions caused by marketing. factor corrects paid social figures in monthly reports, labelled with the test date and spend level, and expires after a set period. When Orchardgate has enough weekly history across channels for a mix model (MMMStatistical model of channel impact.), the result becomes a calibration point on the paid social response curve, as the mix modelling basics guide describes.
Common mistakes
- Including every zone for more data. Leaky or disturbed zones add bias, not power.
- Setting the detectable effect after the result. Then the test cannot fail.
- Counting only platform conversions. The point is total sign-ups by postcode.
- No cool-down. Delayed sign-ups land after the read and the effect looks smaller than it is.
Test this judgement in the causal measurement checkpoint, or revisit attribution models at the intermediate analytics level.
