Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Glossary term

Geo Experiment

A test that runs a campaign in some regions and holds it back in comparable ones, then compares total outcomes, where people cannot be randomised one by one.
AnalyticsprocessUpdated

Definition

A geo experiment is an incrementality test whose unit is a geographic area. Areas are assigned to treatment, where the campaign runs or runs at a heavier weight, or to control, where it is withheld or kept at the usual weight. The control areas act as the holdout. Outcomes are measured for every area from the business's own data, and the lift is the difference between treatment and control after adjusting for how the two sets compared before the test began.

The names vary. The site's guides call it a regional test, a geo test or a geo holdout; matched-market test stresses the matching of areas. All describe the same design: compare comparable places, one set exposed and one not.

When regions are the right unit

Regions are the right unit when people cannot be split one by one, or when splitting them would miss what matters. The guide to measuring offline media places regional tests near the top of its ladder of evidence, because local radio, regional television, regional press and outdoor by city can all be bought by area, and no platform can randomise who hears a radio spot. The guide to buying radio advertising makes the same point for radio specifically: where it can be bought by area, a regional test against comparison areas is the strongest evidence available.

For digital channels the choice is closer. A platform's lift study is easier to run but measures inside that platform's universe and is run by the party being judged; a geo test needs only the ability to target or exclude regions and reads every outcome a region produces. The guide to measuring TikTok's incremental impact sets out that choice for one platform, and the same reasoning applies to others.

In practice: comparable areas, matched weight, own outcomes

The incrementality testing guide owns the design, and the points that define a geo experiment follow from it:

  • Comparable areas. Test and control areas are chosen so that their outcome history moves together, and a pre-period comparison confirms it before anything launches.
  • Size decided first. The smallest lift worth detecting is set before the test, because regions are few and an underpowered test reads "no effect" when it means "could not tell".
  • Clean conditions. No extra promotions, distribution changes or spending in one set that the other does not also get.
  • Own outcomes. Orders, leads or bookings by region from the business's records, not platform-reported conversions.
  • A full cycle. The test runs across a full purchase cycle, with a cool-down afterwards, and is read as an interval rather than a single figure.

Geo experiments also test plans, not only channels. The tutorial on weighting a media plan by region gives one region heavier weight than the plan's rule and keeps a matched region at the rule's weight, so the regional split itself becomes a hypothesis the results can confirm or overturn.

Illustrative example — not a real result

A driving school tests radio town by town

Veallott Driving Instructors, an invented driving instruction firm, works in 8 towns of similar size whose weekly lesson bookings have tracked each other closely for a year. It buys local radio in 4 of them for 6 weeks and holds the other 4 back, with no other change in either set. Before the test the radio towns averaged 210 bookings a week and the others 205. During the test the radio towns average 244 and the held-back towns 207. Against the towns' own baseline, the radio towns rose by about 32 bookings a week more than the held-back towns, roughly 190 extra bookings over the test, most of them by phone, which the firm's website tracking would never have attributed to radio.

A geo experiment is one way to measure Incrementality, the outcomes that would not have happened without the campaign. Its held-back regions are a Holdout Group made of places rather than people. Each region's expected outcome without the campaign is its Baseline, usually built from its own history and the control regions' movement. And an MMM, a marketing mix model, benefits from geo experiments twice: the regional variation they create helps the model tell channels apart, and their measured lift gives it points to calibrate against.

Common confusions

  • Any before-and-after comparison by region is a geo experiment. Without comparable control areas chosen in advance, it is a pre and post reading, confounded by everything else that changed.
  • The two sets need to be identical. They need to move together beforehand; the analysis adjusts for the remaining difference.
  • A geo experiment measures the whole channel. It measures the change tested: switching the channel off, or reducing or increasing its weight, at that spend level and over that period.
  • More regions are always possible. The number of usable areas is limited by what can be bought, targeted and measured separately, which is why power is the main constraint.
  • Spillover does not matter. People travel, media signals cross borders and online orders can come from anywhere, so areas are chosen to keep leakage small.

More terms to look up

Where this term is used

Related