Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Analytics › Advanced

Designing a Geo Holdout Test for a Juice Delivery Firm

You design a geo holdout for an invented juice subscription whose attribution credits paid social with most new subscribers. The exercise practises choosing and excluding regions, tying the minimum detectable effect to the decision, setting the metric, guardrails, duration and decision rule in advance, and planning how the result calibrates attribution and a future mix model.
ExerciseAdvanced levelModule 2: Causal measurement5 minUpdated

Why this matters

Data-driven Attribution is generous to channels that reach people already on their way to buying, and the data-driven attribution guide explains why it cannot see what would have happened anyway. The incrementality testing guide gives the remedy: withhold the channel from comparable places and measure the difference. With a handful of regions, the design decides whether the test can answer anything at all.

The firm and its regions

Orchardgate Juice Delivery is an invented cold-pressed juice subscription delivering to eight zones. New subscribers sign up online and are recorded by delivery postcode in the subscription system. The owner's attribution report credits paid social with most new subscriptions and she wants to know how many it causes.

Illustrative example — not a real result

Orchardgate Juice Delivery: zones over a twenty-six-week pre-period

ZoneNew subscriptions per weekPaid social spend per weekNotes
Harbourside140£2,800Adjacent to Old Town; heavy commuting between them
Old Town130£2,600Adjacent to Harbourside
Riverside95£1,900Weekly sign-ups track Hillcrest closely
Hillcrest90£1,800Tracks Riverside
Westfield70£1,400Weekly sign-ups track Eastmoor closely
Eastmoor65£1,300Tracks Westfield
Southbay50£1,000Poster campaign booked during the planned test window
Northgate45£900Delivery rounds close to full capacity
  • Data-driven attribution credits paid social with 55% of new subscriptions.
  • Weekly ratio between each tracking pair wanders within about ±8%.
  • Contribution per subscriber per week: £12. Average subscription length: 20 weeks.
  • The owner will pay at most half a subscriber's contribution to acquire one.

Choosing cells, duration and the metric

Regions belong in the test only if they can be compared and kept apart. Pairs that moved together in the pre-period make good matches; neighbours with heavy commuting leak exposure across the border; and anything else planned for one region and not its pair, from posters to capacity limits, will be read as the effect. The minimum detectable effect should come from the decision, not from convenience: the smallest loss of sign-ups that would make the spend worth keeping. If the pre-period wander is larger than that effect, no amount of analysis afterwards will rescue the read.

The metric should count every sign-up a region produces, whatever channel gets the credit. That is why a geo design fits better here than a user-level holdout like the remarketing holdout test, which suits audiences the business owns.

Task

  1. Choose the holdout and control regions, and say which zones stay out of the read and why.
  2. Calculate the minimum detectable effect from the owner's cost limit and compare it with the pre-period wander.
  3. Set the primary metric, the guardrails and any stopping rule.
  4. Set the duration, the cool-down and the timing.
  5. Write the decision rule for each possible outcome.
  6. Explain how the result will adjust attribution reporting and later calibrate a mix model.

Worked answer

Regions. Use the two tracking pairs, Riverside with Hillcrest and Westfield with Eastmoor, and toss a coin within each pair for which zone loses paid social. Leave Harbourside and Old Town out of the read: commuting would carry exposure into whichever one is held out. Exclude Southbay because of the posters and Northgate because full delivery rounds would cap sign-ups in its pair. Keep paid social running in excluded zones, and say that the result describes the smaller zones only.

Illustrative example — not a real result

Minimum detectable effect from the owner's limit

Value of a subscriber: £12 × 20 weeks = £240. Maximum cost per incremental subscriber: £120. Suppose the coin picks Hillcrest and Eastmoor. Spend withheld: £1,800 + £1,300 = £3,100 a week. To be worth that spend, paid social must cause at least £3,100 ÷ £120 ≈ 26 sign-ups a week in those zones, out of 90 + 65 = 155: a drop of about 17%.

Pre-period wander: about ±8% in a single week, so a sustained 17% gap over several weeks stands clear of it. If attribution were right, the drop would be near 55%.

Metric and guardrails. Primary: new subscriptions per week by delivery postcode, from the subscription system, held-out zone against the prediction from its pair. Guardrails: brand search and direct sign-ups by zone, to see whether demand shifted channel rather than disappeared; first-month cancellations, so lost sign-ups are not just poor-quality ones; and delivery capacity in the control zones. The only stopping rule is business harm agreed in advance, such as a fall far beyond the attributed share for consecutive weeks; never stop because an early week looks decisive.

Duration. Six weeks of holdout, long enough to cover the time from first seeing an ad to signing up, then two weeks of cool-down with spend restored so delayed sign-ups land in both groups. Avoid the new-year spike. Freeze creative, offers and prices in all four zones throughout.

Decision rule. Calculate incremental sign-ups and incremental cost per sign-up with a range, not a point. If even the pessimistic end of the cost sits under the limit, keep spend and test a higher level. If the midpoint sits above it, cut by a third and retest. Anything between: hold spend and rerun with more weeks or zones.

Calibration. Divide incremental sign-ups by the sign-ups attribution credited to paid social in the comparable zones over the same weeks. That Incrementality factor corrects paid social figures in monthly reports, labelled with the test date and spend level, and expires after a set period. When Orchardgate has enough weekly history across channels for a mix model (MMM), the result becomes a calibration point on the paid social response curve, as the mix modelling basics guide describes.

Common mistakes

  • Including every zone for more data. Leaky or disturbed zones add bias, not power.
  • Setting the detectable effect after the result. Then the test cannot fail.
  • Counting only platform conversions. The point is total sign-ups by postcode.
  • No cool-down. Delayed sign-ups land after the read and the effect looks smaller than it is.

Test this judgement in the causal measurement checkpoint, or revisit attribution models at the intermediate analytics level.

Pages this lesson applies

Terms used on this page

Also related

Related