Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Comparisons

Geo Holdout Tests vs Conversion Lift Studies

Geography-based holdout tests and platform conversion lift studies compared: how each builds a control group, what each can measure, and when to run which.
Comparisonadvanced9 min readUpdated

Geo holdout test and conversion lift study at a glance

A geo holdout test treats regions as the unit of the experiment. The advertiser picks a set of comparable regions, such as cities, postcode areas or sales territories, switches the advertising off or down in some of them and leaves it running in the rest. After the test period, total outcomes in the treated regions are compared with outcomes in the control regions, adjusted for how the two groups compared before the test. The difference is the incremental outcome. Because it is measured from the business's own regional sales, it does not depend on any platform's tracking, and it can include orders taken in shops, over the phone or through partners. The incrementality testing guide covers how such a test is designed, matched and sized; this page does not repeat that.

A conversion lift study is the platform's own version of the same experiment, run at the level of individual users. The platform takes the audience a campaign would reach, withholds the ads from a random share of it and compares conversions between the exposed and held-back groups. Google Ads Help and the Meta Business Help Centre both document lift measurement of this kind, and both describe eligibility conditions and spend requirements, so a study cannot be switched on for every campaign. The result is a measure of Incrementality inside one platform's universe, read against the conversions that platform records or receives.

The two are not rival theories of measurement. They are two ways of building a control group, and each one trades something to get its advantage. The geo test gives independence and full outcome coverage at the cost of statistical power and effort. The platform study gives clean user-level randomisation and convenience at the cost of independence and a narrower view of outcomes.

Compare table

Geo holdout test vs Conversion lift study

Row labelGeo holdout testConversion lift study
Unit of randomisationRegions, cities or postcode areas, matched on their outcome historyIndividual users within the platform's audience, assigned at random
Who runs itThe advertiser or its analysts, with the platform only delivering ads by locationThe platform, through its own lift product
Channels coveredAny channel that can be switched off or down by region, alone or togetherOne platform, usually one campaign or a set of campaigns on it
Offline outcomesIncluded, provided regional sales data existsOnly what the platform observes or receives through uploads or integrations
CostWithheld spend in control regions, analyst time and data preparationWithheld reach inside the campaign, plus any minimum spend to qualify
Contamination riskCommuters, national delivery and media that ignore borders blur the groupsShared devices and people the platform cannot match across sessions
Readout speedSlower: the test period, a cool-down and a regional analysisFaster: the platform reports once the study period ends
Independence from the platformHigh: outcomes come from the business's own systemsLow: the party being judged designs, runs and reports the study
Statistical powerLimited by the small number of regionsUsually higher, because the sample is many individual users

The rows that decide most choices are offline outcomes and independence. If a meaningful share of sales happens where no platform can see it, a platform study will understate the effect by design. If the result will be used to defend or cut a large budget line, a figure produced by the seller of that line will struggle to carry the decision on its own. The academy checkpoint on evidence for what social channels add tests this choice for social channels beside view-through credit, outside signals, mix models and reporting an organic video channel.

When to choose a geo holdout test

Choose a geo holdout when the outcome the business cares about is not fully visible to the platform. Retailers with shops, businesses that close sales by phone, and brands sold through distributors all have outcomes that no pixel records. A regional comparison of total sales captures them without any tracking change, which is the single largest advantage of the design.

Choose it when the question spans channels. A geo test can withhold paid social and online video together, or reduce a whole regional media plan by a set share, and read the combined effect. No platform study can measure a channel it does not sell. The same logic applies when two platforms each claim the same customers: a geo test on total outcomes is the neutral referee.

Choose it when independence matters more than speed. A finance team asked to fund a large channel will reasonably ask who produced the evidence. A test designed by the advertiser, measured from its own order system, answers that question in a way a platform report cannot.

The constraints are real. Regional targeting has to be reliable, and the geo-targeted campaigns guide shows how location settings decide who actually sees an ad. The market needs enough comparable regions, the business needs regional outcome data, and the team needs the skill to match regions and read the result. Borders leak: people commute, national media reaches everyone and online orders may ship from one warehouse. A result whose interval is wide because there were too few regions is not a failure of the method, but it is not a decision either.

When to choose a conversion lift study

Choose a platform conversion lift study when the question is about one platform and its rules fit. If the campaign qualifies, the platform handles randomisation at the level of individual users, which gives far more units than any regional split and so a tighter estimate for the same spend. Setup is mostly configuration rather than analysis, and the result arrives in the same interface as the campaign.

It is the right first test for a team that has never measured incrementality. It is cheaper to run than a geo test, it needs no regional data, and it shows how far the platform's attributed conversions sit from the caused ones, which is often the most useful single fact a team can learn about a channel. It also suits repeated checks: once a geo test has established how far a platform's own studies can be trusted, regular lift studies can track the channel between independent tests.

The limits follow from who runs it. The study counts the conversions the platform observes or receives, so outcomes outside its view are missing. The method is documented in outline rather than in full, so its assumptions cannot all be audited. And the result describes the channel inside that platform only; it cannot say what would happen if the budget moved elsewhere. Read the platform's own statement of Statistical Significance or confidence alongside the size of the effect, not instead of it.

Illustrative example — not a real result

A tile retailer checks a platform study with a geo test

Glimmerstone Tiles sells online and through fourteen showrooms. It runs a four-week conversion lift study on its paid social campaigns at £30,000 of spend. The platform reports 210 incremental online orders at an average order value of £180, or £37,800 of incremental revenue: an incremental ROAS of about 1.3. That looks marginal. The following quarter, at the same weekly spend, it runs a geo holdout across ten treated and ten matched control regions and measures total revenue from its order system and showroom tills. Online revenue in the treated regions is £36,000 higher than the matched control, close to the platform's figure, but showroom revenue is a further £27,000 higher, which the platform never saw. Total incremental revenue of £63,000 on £30,000 of spend gives an incremental ROAS of 2.1. The retailer keeps the channel, records that platform studies capture a little over half of its effect, and uses that ratio to read future lift studies until the next geo test.

Decision table

SituationChooseWhy
A large share of sales happens in shops, by phone or through partnersGeo holdout testOnly regional totals capture outcomes no platform records
The question is one platform's effect and the campaign qualifiesConversion lift studyUser-level randomisation gives a tighter estimate for less effort
Two or more platforms claim the same customersGeo holdout testA neutral comparison on total outcomes settles overlapping claims
First incrementality test for a team with no analystConversion lift studySetup is configuration, and the readout arrives in the platform
Budget decision on the largest or most contested lineGeo holdout test, ideally after a lift studyIndependent evidence carries a large decision further
Market has few comparable regionsConversion lift studyA geo test would lack the power to read a realistic effect
Campaign falls below the platform's spend or eligibility rulesGeo holdout test, or a larger pooled studyThe platform product is simply not available

Running one to check the other

The two designs work best as a pair rather than as alternatives. Run the platform study first, because it is cheaper and faster, and record its incremental figure. Then run a geo holdout on the same channel at a similar spend level, in a different period so the two designs do not interfere, and measure total outcomes. The ratio between the two answers tells you how much of the channel's effect the platform can see.

If the two agree, the platform's lift studies can be trusted for routine checks on that channel, and the expensive geo test only needs repeating when something material changes: a large budget shift, a new market or a change in how sales happen. If they disagree, the reason is usually visible in the data: outcomes the platform cannot observe, spillover across regional borders or a difference in spend level between the two tests. Either way, the incremental figures from both feed the reporting described in the guide to choosing an attribution approach, and the comparison of marketing mix modelling and multi-touch attribution explains how test results then calibrate the wider budget model. The Incremental ROAS each test produces describes the channel at that spend and in that period, so date both results and keep them side by side. The academy capstone on a social programme governance plan for a swimwear brand practises placing these tests in a year's evidence plan for TikTok, creators and organic video, beside incident readiness, programme audits and accessibility standards.

Common mistakes

  • Treating a platform lift study as an independent audit. It is a sound experiment run by the party being judged, limited to the outcomes that party can see.
  • Running a geo test with too few regions and reading a wide, inconclusive interval as proof that the channel does nothing.
  • Ignoring spillover. Commuters, national media and central fulfilment move effects across borders and shrink the measured difference.
  • Comparing a platform study and a geo test run at different spend levels and calling the gap a measurement error.
  • Measuring only online orders in a geo test when the reason for choosing the design was to capture offline sales.
  • Running both designs on one channel at the same time without restricting one to the other's treated regions.
  • Letting one result stand for years. Both designs describe a channel at one spend level and one moment; retest when either changes.

Frequently asked questions

Is a platform conversion lift study biased because the platform runs it?

Not necessarily biased, but not independent either. The randomisation inside a well-run lift study is sound, so the comparison between exposed and held-back users is a fair one. The limits are elsewhere: the study counts only the conversions the platform can observe or receive, the method details are only partly documented, and the party measuring the effect is the party being paid for it. Treat the result as the platform's estimate of its own effect and check it with an independent geo holdout on the largest budget lines.

How many regions does a geo holdout need?

Enough that the treated and control groups move together before the test and that the smallest effect worth acting on would stand out from their normal week-to-week difference. There is no fixed count: it depends on how noisy regional sales are, how large the expected effect is and how many comparable regions the market has. The incrementality testing guide covers how to size a test; the practical warning is that small markets with few regions often cannot support a readable geo test at all.

Can I run a geo holdout and a platform lift study at the same time?

It is possible but rarely wise on the same channel, because the geo test changes delivery in the very regions the platform is sampling from and each design then contaminates the other. Running them in consecutive periods at a similar spend level is cleaner, and it gives two estimates of the same channel's effect that can be compared directly. If they must overlap, restrict the platform study to the treated regions and say so in the readout.

Terms used on this page

Where this page is used

Part of these learning paths

Also related

Sources

  1. Google Ads Help — Google Ads Help
  2. Meta Business Help Centre — Meta

Related