Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Marketing Analytics › Attribution

Incrementality Testing

How incrementality tests measure what marketing caused: holdout and geo designs, lift studies, cells and duration, reading lift, and budget decisions.
Guideadvanced10 min readUpdated

Why attribution cannot answer the budget question

Every attribution model, including data-driven, works from paths that were recorded. It can compare paths with a touchpoint against paths without one, but the people on those paths were not assigned at random. Someone who clicked a brand search ad was already searching for the brand; someone who saw a retargeting ad had already visited the site. Paths that contain those touches convert more often for reasons that precede the ad, and every model credits the ad. The data-driven attribution guide sets out the mechanism and why it still cannot see the counterfactual.

The budget question is different in kind. "Should we spend more or less on this channel?" asks what would happen to conversions if the spend changed, and the only way to observe that is to change the spend for some and not for others and compare. That is an experiment, and the discipline is the one behind any A/B Test: random assignment, a control, a pre-registered outcome and enough sample to read the difference. Applied to marketing channels it is called incrementality testing, and its output is the number attribution can never produce: conversions that would not have happened without the channel.

The cost of the answer is that it is narrow. A test measures one channel, at one spend level, over one period, for one outcome. It says nothing about the channel at twice the spend, next quarter, or for a different product. That narrowness is why tests are planned around the biggest budget lines and the most contested ones, rather than run everywhere, and why the marketing mix modelling guide exists for the questions a single test cannot reach.

Test designs

Four designs cover most practical cases. They differ in what they randomise, what they can measure and what they need from the platform.

DesignWhat is randomisedMeasuresNeedsWeakness
User-level holdoutIndividual people or households, some shown the ads and some withheldLift in any outcome the business can observe for both groupsA way to withhold ads from a random subset: a suppression list for owned channels, or the platform's own tooling for paid mediaHard to withhold paid ads cleanly outside a platform; contamination through shared devices
Platform lift studyPeople within the platform's audience, assigned by the platformLift in conversions the platform can observe or receiveThe platform's lift product and enough spend to qualifyMeasured inside one platform's universe and run by the party being judged
Geo testRegions, cities or postcodes, treated and controlLift in total outcomes per region, including offline and unobserved conversionsRegional targeting, regional outcome data, enough comparable regionsSpillover across borders; regions are few, so power is limited and matching matters
Pre and postNothing; the channel is switched on or off over timeChange in outcomes before and afterOnly a calendarConfounded by season, promotions, competitors and everything else that changed

The user-level holdout is the cleanest when it is possible, which in practice means owned channels and platform-supported paid media. The geo test is the workhorse for paid channels because it measures every outcome a region produces, online and offline, without depending on any platform's tracking; its challenge is statistical, since the number of regions is small and they must be balanced or matched carefully. The platform lift study is the easiest to run and the hardest to audit. The pre and post comparison is listed because teams run it constantly, usually without naming it; it is a last resort, acceptable only when the change is large, sudden and isolated, and even then it needs a comparison series from a market that did not change.

Variations exist for each. A geo test can hold back a channel entirely or reduce its spend by a set share, which measures the marginal effect of the top slice of budget rather than the whole channel. A holdout can show a public-service announcement to the control group in place of the brand's ad, so that both groups see something and the comparison isolates the message. The design question is always the same: what is the unit of randomisation, what outcome will be compared, and what could leak between the groups?

Choosing cells, size and duration

Cells. The minimum is one treatment cell and one control cell. Add cells only when the test is designed to compare spend levels or creative strategies, and accept that each extra cell divides the sample. A control cell must be comparable before the test starts: for user-level designs, random assignment achieves that; for geo designs, the regions are chosen so that their outcome history moves together, and a pre-period comparison is run to confirm that it does.

Size. Before the test, decide the smallest lift that would change the decision, the minimum detectable effect, and work backwards from the outcome's baseline volume and variance to the sample or the number of regions needed to detect it. Google does not document a formula for this inside its products, and no platform's lift tool should be allowed to decide it after the fact; the calculation belongs to the team, before the test, and it is the number that later decides whether "no significant lift" means "no effect" or "underpowered". The outcome chosen matters here: a frequent outcome such as a lead form or an add-to-basket needs far less sample than a rare one such as a high-value purchase, and a test on a rare outcome often has to measure a proxy and accept the gap.

Duration. Run across at least one full buying cycle so that the control group has time to show what people do without the channel, add a cool-down period after the treatment stops so that delayed conversions land in both groups, and avoid starting or ending inside a seasonal peak or a promotion. Longer is not always better: contamination accumulates as people move between regions or devices, and the business's patience is finite. Freeze other changes to the channel and to the site for the duration, and write down the start, end, cells, outcome and minimum detectable effect before the first day. A test that is changed while running is two half tests. The holdout test readiness checklist turns these decisions into a gate to tick before the first day, while the cells can still be redrawn.

Reading lift and uncertainty

The raw output of a test is the outcome in the treated group, the outcome in the control group adjusted to the treated group's scale, and the difference. The difference is the incremental outcome. Divided by the spend in the treated group it becomes an incremental cost per outcome, and for revenue outcomes an incremental return on ad spend (ROAS), which the ROAS calculator will compute once the incremental revenue is known. Three readings follow.

The point estimate against the attributed figure. The ratio between the incremental conversions and the conversions the platform or the analytics model attributed to the channel over the same period is the channel's incrementality factor. A channel whose attributed conversions were mostly going to happen anyway has a low factor; a channel that creates demand the platform never sees can have a factor above one.

The interval, not just the point. Every test result comes with a range of plausible lift. A result whose interval includes zero has not shown an effect; a result whose interval is wide has shown an effect whose size is uncertain, and the budget decision has to be robust across that range rather than optimised for the midpoint.

The spend level. The lift was measured at the spend the treated group received. It is a statement about that level and the slice of budget tested, not about the channel in general; scaling the spend changes the answer, usually downwards per unit as the channel saturates.

Illustrative example — not a real result

Reading a geo holdout on paid social

A homeware retailer runs a six-week geo test on paid social, withholding the channel from 12 control regions matched to 12 treated regions on their order history. During the test the treated regions generate 2,310 orders and the control regions, scaled to the treated regions' pre-test baseline, 2,040. Incremental orders: 270, with a plausible range of roughly 110 to 430. Paid social spend in the treated regions was £54,000, so the incremental cost per order is £200 at the midpoint and somewhere between £126 and £490 across the range. The platform attributed 610 orders to the same spend in the same regions over the same weeks, so the incrementality factor is about 0.44: a little under half the attributed orders were caused by the ads. The retailer's blended gross profit per order is £95. At the midpoint the channel loses money on first orders but is close to break-even on a twelve-month customer value of £260; at the pessimistic end of the range it is clearly unprofitable. The decision is not to cut but to reduce spend by a third and retest at the new level, because the marginal slice is the one most likely to be wasted.

From lift to budget decisions

A test result enters a budget decision in three ways. The first is direct: the incremental cost per outcome at the tested spend, compared with the outcome's value to the business, says whether the channel at that level is worth keeping, trimming or growing. The marketing ROI hub covers the value side of that comparison, including margin and customer lifetime, which is where tests on first-order outcomes most often mislead.

The second is as a correction factor. The incrementality factor from a clean test can be applied to the channel's attributed conversions in routine reporting until the next test, so that monthly dashboards show estimated incremental outcomes rather than touched ones. The factor decays in value as the channel, the creative and the competition change, which is why the biggest channels are retested on a schedule rather than once. The comparison of iROAS and ROAS shows how to report the tested and the platform figure side by side without either standing in for the other.

The third is as calibration for the broader allocation. The budget allocation guide allocates by marginal return, and a test is the most reliable point on a channel's return curve. Mix models are calibrated against test results for the same reason. A programme that runs a few well-designed tests a year on its largest or most contested lines, feeds the factors into reporting and the points into allocation, and uses attribution for everything tactical in between, has the measurement stack the attribution hub describes.

One caution carries through all three uses. A test measures a channel's marginal contribution given everything else that was running. Cutting two channels that each tested as weakly incremental can remove more than the sum of their individual effects, because each was tested with the other present. Sequence large changes, and retest after them.

Where incrementality tests sit beside attribution models and mix modelling, and which to reach for first, is laid out in the guide to choosing an attribution approach.

Brand lift studies use the same control-group logic, and the measurement plan for awareness shows how to book one before the flight alongside reach and search signals.

When retargeting and brand search dominate every attribution report, retargeting and brand search taking all the credit shows how to compare their credit with new customers before designing a holdout.

When the choice is between building the control group from regions and letting an ad platform randomise its own audience, the comparison of geo holdout tests and conversion lift studies sets out what each design can measure and who controls it.

For a channel whose return looks weakest on last click, the guide to measuring what TikTok adds beyond last click applies these designs to TikTok and orders them by budget and stage.

When the question is an individual's share of a tested result, for a portfolio case or a promotion case, the guide to showing your contribution to marketing results honestly turns a holdout's range into wording that claims no more than the test showed.

Common mistakes

  • Running a pre and post comparison and calling it a test. Season and promotions did the moving.
  • Deciding the minimum detectable effect after seeing the result. The test can no longer fail.
  • Stopping when the number looks good. Early results are the noisiest.
  • Letting regions leak. Commuters, shared devices and national delivery blur the groups.
  • Reading the midpoint and ignoring the interval. The decision has to survive the pessimistic end.
  • Treating one test as permanent truth. Channels, creative and competitors change; retest the big lines.
  • Measuring only the platform's own conversions. A geo test on total orders catches what the platform misses.

The data-driven attribution guide explains why a model's credit is the hypothesis a test checks rather than the answer, and the attribution hub places testing in the wider measurement stack.

Frequently asked questions

How long should an incrementality test run?

Long enough to cover at least one full buying cycle for the product, plus a cool-down period after the treatment ends so that delayed conversions in both groups are counted, and long enough to reach the sample the pre-test power calculation asked for. A test that is stopped early because the result looks good is reporting noise, and one that straddles a seasonal peak without planning for it is reporting the season.

Can I run an incrementality test without a platform's lift study tool?

Yes. A geo test needs nothing from the platform beyond the ability to target or exclude regions, and a holdout for owned channels such as email needs only a randomised suppression list. Platform lift studies are convenient because the platform assigns the control group within its own audience, but they measure within the platform's universe and are run by the party whose spend is being judged, so an independent geo design is worth having alongside.

What does it mean if the test shows no significant lift?

It means the effect, if there is one, was smaller than the test was powered to detect; it does not prove the channel does nothing. Check the minimum detectable effect that was set before the test, whether the groups were contaminated, and whether the channel's spend during the test was typical. If the test was well powered and clean, the honest conclusion is that the channel's effect at that spend level is too small to justify the budget, which is a result.

Terms used on this page

Tools to try

Frameworks

Where this page is used

Part of these learning paths

Related