Geo holdout test and conversion lift study at a glance
A geo holdout test treats regions as the unit of the experiment. The advertiser picks a set of comparable regions, such as cities, postcode areas or sales territories, switches the advertising off or down in some of them and leaves it running in the rest. After the test period, total outcomes in the treated regions are compared with outcomes in the control regions, adjusted for how the two groups compared before the test. The difference is the incremental outcome. Because it is measured from the business's own regional sales, it does not depend on any platform's tracking, and it can include orders taken in shops, over the phone or through partners. The incrementality testing guide covers how such a test is designed, matched and sized; this page does not repeat that.
A conversion lift study is the platform's own version of the same experiment, run at the level of individual users. The platform takes the audience a campaign would reach, withholds the ads from a random share of it and compares conversions between the exposed and held-back groups. Google Ads Help and the Meta Business Help Centre both document lift measurement of this kind, and both describe eligibility conditions and spend requirements, so a study cannot be switched on for every campaign. The result is a measure of IncrementalityConversions caused by marketing. inside one platform's universe, read against the conversions that platform records or receives.
The two are not rival theories of measurement. They are two ways of building a control group, and each one trades something to get its advantage. The geo test gives independence and full outcome coverage at the cost of statistical power and effort. The platform study gives clean user-level randomisation and convenience at the cost of independence and a narrower view of outcomes.
Compare table
Geo holdout test vs Conversion lift study
| Row label | Geo holdout test | Conversion lift study |
|---|---|---|
| Unit of randomisation | Regions, cities or postcode areas, matched on their outcome history | Individual users within the platform's audience, assigned at random |
| Who runs it | The advertiser or its analysts, with the platform only delivering ads by location | The platform, through its own lift product |
| Channels covered | Any channel that can be switched off or down by region, alone or together | One platform, usually one campaign or a set of campaigns on it |
| Offline outcomes | Included, provided regional sales data exists | Only what the platform observes or receives through uploads or integrations |
| Cost | Withheld spend in control regions, analyst time and data preparation | Withheld reach inside the campaign, plus any minimum spend to qualify |
| Contamination risk | Commuters, national delivery and media that ignore borders blur the groups | Shared devices and people the platform cannot match across sessions |
| Readout speed | Slower: the test period, a cool-down and a regional analysis | Faster: the platform reports once the study period ends |
| Independence from the platform | High: outcomes come from the business's own systems | Low: the party being judged designs, runs and reports the study |
| Statistical power | Limited by the small number of regions | Usually higher, because the sample is many individual users |
The rows that decide most choices are offline outcomes and independence. If a meaningful share of sales happens where no platform can see it, a platform study will understate the effect by design. If the result will be used to defend or cut a large budget line, a figure produced by the seller of that line will struggle to carry the decision on its own. The academy checkpoint on evidence for what social channels add tests this choice for social channels beside view-through credit, outside signals, mix models and reporting an organic video channel.
When to choose a geo holdout test
Choose a geo holdout when the outcome the business cares about is not fully visible to the platform. Retailers with shops, businesses that close sales by phone, and brands sold through distributors all have outcomes that no pixel records. A regional comparison of total sales captures them without any tracking change, which is the single largest advantage of the design.
Choose it when the question spans channels. A geo test can withhold paid social and online video together, or reduce a whole regional media plan by a set share, and read the combined effect. No platform study can measure a channel it does not sell. The same logic applies when two platforms each claim the same customers: a geo test on total outcomes is the neutral referee.
Choose it when independence matters more than speed. A finance team asked to fund a large channel will reasonably ask who produced the evidence. A test designed by the advertiser, measured from its own order system, answers that question in a way a platform report cannot.
The constraints are real. Regional targeting has to be reliable, and the geo-targeted campaigns guide shows how location settings decide who actually sees an ad. The market needs enough comparable regions, the business needs regional outcome data, and the team needs the skill to match regions and read the result. Borders leak: people commute, national media reaches everyone and online orders may ship from one warehouse. A result whose interval is wide because there were too few regions is not a failure of the method, but it is not a decision either.
When to choose a conversion lift study
Choose a platform conversion lift study when the question is about one platform and its rules fit. If the campaign qualifies, the platform handles randomisation at the level of individual users, which gives far more units than any regional split and so a tighter estimate for the same spend. Setup is mostly configuration rather than analysis, and the result arrives in the same interface as the campaign.
It is the right first test for a team that has never measured incrementality. It is cheaper to run than a geo test, it needs no regional data, and it shows how far the platform's attributed conversions sit from the caused ones, which is often the most useful single fact a team can learn about a channel. It also suits repeated checks: once a geo test has established how far a platform's own studies can be trusted, regular lift studies can track the channel between independent tests.
The limits follow from who runs it. The study counts the conversions the platform observes or receives, so outcomes outside its view are missing. The method is documented in outline rather than in full, so its assumptions cannot all be audited. And the result describes the channel inside that platform only; it cannot say what would happen if the budget moved elsewhere. Read the platform's own statement of Statistical SignificanceA test result unlikely enough under 'no real difference' to pass a threshold chosen in advance. or confidence alongside the size of the effect, not instead of it.
A tile retailer checks a platform study with a geo test
Glimmerstone Tiles sells online and through fourteen showrooms. It runs a four-week conversion lift study on its paid social campaigns at £30,000 of spend. The platform reports 210 incremental online orders at an average order value of £180, or £37,800 of incremental revenue: an incremental ROAS of about 1.3. That looks marginal. The following quarter, at the same weekly spend, it runs a geo holdout across ten treated and ten matched control regions and measures total revenue from its order system and showroom tills. Online revenue in the treated regions is £36,000 higher than the matched control, close to the platform's figure, but showroom revenue is a further £27,000 higher, which the platform never saw. Total incremental revenue of £63,000 on £30,000 of spend gives an incremental ROAS of 2.1. The retailer keeps the channel, records that platform studies capture a little over half of its effect, and uses that ratio to read future lift studies until the next geo test.
Decision table
| Situation | Choose | Why |
|---|---|---|
| A large share of sales happens in shops, by phone or through partners | Geo holdout test | Only regional totals capture outcomes no platform records |
| The question is one platform's effect and the campaign qualifies | Conversion lift study | User-level randomisation gives a tighter estimate for less effort |
| Two or more platforms claim the same customers | Geo holdout test | A neutral comparison on total outcomes settles overlapping claims |
| First incrementality test for a team with no analyst | Conversion lift study | Setup is configuration, and the readout arrives in the platform |
| Budget decision on the largest or most contested line | Geo holdout test, ideally after a lift study | Independent evidence carries a large decision further |
| Market has few comparable regions | Conversion lift study | A geo test would lack the power to read a realistic effect |
| Campaign falls below the platform's spend or eligibility rules | Geo holdout test, or a larger pooled study | The platform product is simply not available |
Running one to check the other
The two designs work best as a pair rather than as alternatives. Run the platform study first, because it is cheaper and faster, and record its incremental figure. Then run a geo holdout on the same channel at a similar spend level, in a different period so the two designs do not interfere, and measure total outcomes. The ratio between the two answers tells you how much of the channel's effect the platform can see.
If the two agree, the platform's lift studies can be trusted for routine checks on that channel, and the expensive geo test only needs repeating when something material changes: a large budget shift, a new market or a change in how sales happen. If they disagree, the reason is usually visible in the data: outcomes the platform cannot observe, spillover across regional borders or a difference in spend level between the two tests. Either way, the incremental figures from both feed the reporting described in the guide to choosing an attribution approach, and the comparison of marketing mix modelling and multi-touch attribution explains how test results then calibrate the wider budget model. The Incremental ROASRevenue caused by advertising, measured against a holdout, divided by the ad spend. each test produces describes the channel at that spend and in that period, so date both results and keep them side by side. The academy capstone on a social programme governance plan for a swimwear brand practises placing these tests in a year's evidence plan for TikTok, creators and organic video, beside incident readiness, programme audits and accessibility standards.
Common mistakes
- Treating a platform lift study as an independent audit. It is a sound experiment run by the party being judged, limited to the outcomes that party can see.
- Running a geo test with too few regions and reading a wide, inconclusive interval as proof that the channel does nothing.
- Ignoring spillover. Commuters, national media and central fulfilment move effects across borders and shrink the measured difference.
- Comparing a platform study and a geo test run at different spend levels and calling the gap a measurement error.
- Measuring only online orders in a geo test when the reason for choosing the design was to capture offline sales.
- Running both designs on one channel at the same time without restricting one to the other's treated regions.
- Letting one result stand for years. Both designs describe a channel at one spend level and one moment; retest when either changes.
