Definition
A geo experiment is an incrementality test whose unit is a geographic area. Areas are assigned to treatment, where the campaign runs or runs at a heavier weight, or to control, where it is withheld or kept at the usual weight. The control areas act as the holdout. Outcomes are measured for every area from the business's own data, and the lift is the difference between treatment and control after adjusting for how the two sets compared before the test began.
The names vary. The site's guides call it a regional test, a geo test or a geo holdout; matched-market test stresses the matching of areas. All describe the same design: compare comparable places, one set exposed and one not.
When regions are the right unit
Regions are the right unit when people cannot be split one by one, or when splitting them would miss what matters. The guide to measuring offline media places regional tests near the top of its ladder of evidence, because local radio, regional television, regional press and outdoor by city can all be bought by area, and no platform can randomise who hears a radio spot. The guide to buying radio advertising makes the same point for radio specifically: where it can be bought by area, a regional test against comparison areas is the strongest evidence available.
For digital channels the choice is closer. A platform's lift study is easier to run but measures inside that platform's universe and is run by the party being judged; a geo test needs only the ability to target or exclude regions and reads every outcome a region produces. The guide to measuring TikTok's incremental impact sets out that choice for one platform, and the same reasoning applies to others.
In practice: comparable areas, matched weight, own outcomes
The incrementality testing guide owns the design, and the points that define a geo experiment follow from it:
- Comparable areas. Test and control areas are chosen so that their outcome history moves together, and a pre-period comparison confirms it before anything launches.
- Size decided first. The smallest lift worth detecting is set before the test, because regions are few and an underpowered test reads "no effect" when it means "could not tell".
- Clean conditions. No extra promotions, distribution changes or spending in one set that the other does not also get.
- Own outcomes. Orders, leads or bookings by region from the business's records, not platform-reported conversions.
- A full cycle. The test runs across a full purchase cycle, with a cool-down afterwards, and is read as an interval rather than a single figure.
Geo experiments also test plans, not only channels. The tutorial on weighting a media plan by region gives one region heavier weight than the plan's rule and keeps a matched region at the rule's weight, so the regional split itself becomes a hypothesis the results can confirm or overturn.
A driving school tests radio town by town
Veallott Driving Instructors, an invented driving instruction firm, works in 8 towns of similar size whose weekly lesson bookings have tracked each other closely for a year. It buys local radio in 4 of them for 6 weeks and holds the other 4 back, with no other change in either set. Before the test the radio towns averaged 210 bookings a week and the others 205. During the test the radio towns average 244 and the held-back towns 207. Against the towns' own baseline, the radio towns rose by about 32 bookings a week more than the held-back towns, roughly 190 extra bookings over the test, most of them by phone, which the firm's website tracking would never have attributed to radio.
Related terms
A geo experiment is one way to measure IncrementalityConversions caused by marketing., the outcomes that would not have happened without the campaign. Its held-back regions are a Holdout GroupA randomly chosen part of an audience kept from exposure to measure what exposure caused. made of places rather than people. Each region's expected outcome without the campaign is its BaselineAn estimate of what a metric would have done without a given change, campaign or promotion, against which the effect is measured., usually built from its own history and the control regions' movement. And an MMMStatistical model of channel impact., a marketing mix model, benefits from geo experiments twice: the regional variation they create helps the model tell channels apart, and their measured lift gives it points to calibrate against.
Common confusions
- Any before-and-after comparison by region is a geo experiment. Without comparable control areas chosen in advance, it is a pre and post reading, confounded by everything else that changed.
- The two sets need to be identical. They need to move together beforehand; the analysis adjusts for the remaining difference.
- A geo experiment measures the whole channel. It measures the change tested: switching the channel off, or reducing or increasing its weight, at that spend level and over that period.
- More regions are always possible. The number of usable areas is limited by what can be bought, targeted and measured separately, which is why power is the main constraint.
- Spillover does not matter. People travel, media signals cross borders and online orders can come from anywhere, so areas are chosen to keep leakage small.
