Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Comparisons

Marketing Mix Modelling vs Incrementality Testing

Marketing mix modelling and incrementality testing compared: a model of every channel over time against an experiment on one, and how each calibrates the other.
Comparisonadvanced8 min readUpdated

Marketing mix modelling and incrementality testing at a glance

A mix model starts from the total. It takes a long weekly history of an outcome such as sales, breaks it into a baseline, the contribution of each channel and the effects of price, promotions and season, and estimates how each channel's contribution responds to its spend. The marketing mix modelling guide covers its inputs and its two core ideas, carry-over and saturation, which together produce a response curve for every channel and show where Diminishing Returns set in. Its output is a plan: what the outcome would be under a different split of the same money.

An incrementality test starts from one decision. It changes spend for some people or places and not for others, then compares outcomes. The incrementality testing guide covers the designs, from user-level holdouts to geographic tests, and how to size and read them. Its output is a single causal number: what this channel added at this spend in this period.

Neither replaces the other because they fail in opposite ways. The model sees everything and is certain of nothing in particular; it can be wrong about any channel whose spend moved with another or whose effect was absorbed by a missing control. The test is certain about one thing and blind to everything else; it says nothing about the channel at double the spend, or about the channels it left alone. The effectiveness work the IPA publishes is a common reference point for planners weighing this kind of evidence, and reading it well means knowing which kind of method produced each finding.

Compare table

Marketing mix modelling vs Incrementality testing

Row labelMarketing mix modellingIncrementality testing
Question answeredHow does each channel contribute to the total, and what would another split deliver?What did this channel cause, at this spend, in this period?
Data neededA long, consistent history of spend by channel, outcomes and controlsRandomisable people or regions, outcome data for both groups and a pre-agreed design
Channels coveredEvery channel in the history at once, including offline mediaOne channel or one change per test
Causal strengthPartial: a model of history, only as good as its controls and specificationStrong within the test's limits: randomisation supplies the counterfactual
Time to resultMonths to assemble data and build; faster on refreshWeeks per test, plus planning and a cool-down
CostData assembly, modelling expertise and ongoing maintenanceWithheld spend or reach, design effort and analyst time per test
Refresh cadencePeriodically, often quarterly or with each planning roundRepeated on the largest lines when spend, creative or market changes
Use in budget allocationDirect: response curves compare the next unit of spend across channelsIndirect: one well-measured point on one channel's curve

The causal strength and use-in-allocation rows pull in opposite directions, and that is the whole argument for running both. The method best placed to allocate a budget is the one least able to prove cause, and the method that proves cause cannot allocate on its own.

When to choose marketing mix modelling

Choose a mix model when the decision is the split of a whole budget. Annual and quarterly planning asks where the next unit of money should go across every channel, and only a method that estimates all the response curves together can compare them. A test can say one channel is worth keeping; it cannot say whether the money would earn more elsewhere.

Choose it when important media leave no trace a test can easily isolate. Television, radio, out-of-home and sponsorship reach everyone in a market at once, which makes clean holdouts hard and sometimes impossible. A model fitted to a long history of their spend, alongside everything else, can still estimate their contribution.

Choose it when the business has the history. A model needs a long, consistent run of weekly data with real variation in each channel's spend, and the controls that explain the rest of the outcome. A business with a short history, a single channel or spend that never changed gives the model nothing to learn from. It also needs an owner: a model that is built once and never refreshed becomes a historical report.

The tutorial on preparing data for a marketing mix model shows how that weekly history is assembled, checked for flat series and channels that move together, and documented in a data log.

When to choose incrementality testing

Choose a test when one budget line is in question. Whether to keep, cut or grow a large or contested channel is a causal question about that channel, and an experiment answers it more directly than any model. Remarketing, branded search and any channel whose reported return looks too good to question are the usual first candidates.

Choose it when the business is too small or too young for a model. A few digital channels, a short history and a small team can still run one well-designed test a quarter on the largest line, and that produces more decision-ready evidence than a model built on thin data.

Choose it to check a model. Any channel where the model's estimate is uncertain, surprising or about to drive a large reallocation deserves a test before the money moves. That is the bridge to the next section.

Illustrative example — not a real result

A pet food maker calibrates its model with one test

Pondhollow Pet Foods sells through supermarkets and online and spends £1.2 million a year across television, online video, paid social and search. Its first mix model estimates that online video returns £1.60 of revenue for each £1 of spend, but with a range from £0.40 to £2.80, because video spend has always risen and fallen with paid social. Before moving money into video, the company runs an eight-week geographic holdout, withholding video from a matched set of regions. In the treated regions video spend is £80,000 and incremental revenue £96,000, a return of £1.20. The modellers feed that result into the next refit as a calibration point. Video's estimate settles at £1.30 with a range of £0.90 to £1.70, and some of the credit the first model had given video moves to paid social. The planned shift into video is halved, and paid social, now the more uncertain of the two, is scheduled for the next test.

Decision table

SituationChooseWhy
Splitting next year's budget across many channels, offline includedMarketing mix modellingOnly a model compares every channel's next unit of spend at once
Deciding whether to keep or cut one large digital lineIncrementality testingThe question is causal and specific to that line
Short history, few channels, small teamIncrementality testingA model would have too little variation to learn from
Model's estimate for a channel is wide or surprisingTest that channel, then refit the modelThe test turns an uncertain estimate into a calibrated one
Television or sponsorship carries a large share of spendMarketing mix modelling, with geographic tests where feasibleMass media are hard to hold out cleanly at the level of individuals
Finance asks whether marketing's returns are realBoth, with tests as the evidence the model is anchored toTests prove cause; the model extends it to the whole budget

Calibrating the model with tests

Calibration is the work that turns two separate methods into one system. A test gives a measured point on one channel's response curve: at this spend, in this period, the channel added this much. A model calibrated to that point is constrained to agree with it, either by using the result as a prior on the channel's effect or by checking the fitted curve against it and adjusting the specification until it passes through. The other channels' estimates then improve too, because credit the model had wrongly given to the tested channel has to go somewhere.

Choose what to test from the model. The best candidates are channels with the widest uncertainty, the largest spend, the most surprising marginal return or a planned reallocation large enough to matter. A channel whose estimate is tight and whose budget is stable can wait. This keeps testing focused on the lines where an error would cost the most.

Match the test to the model's grain. A model works at channel level by week, so a test is most useful when it measures the same channel definition, on the same outcome, at a spend level the model has seen. A test on one campaign inside a channel, or on an outcome the model does not use, is hard to feed back. Record every test with its dates, spend level, outcome and interval, so each refit can use the full set. The academy checkpoint on mix model data, review and calibration tests how a mix model and an incrementality test calibrate each other and which channel to test first.

Agreement with past tests is one of the checks in the guide to evaluating a marketing mix model, alongside the model's data, specification, stability and uncertainty.

Keep the loop running. Each planning round starts from the latest calibrated model, moves money in steps rather than all at once, tests the channels that moved most, and refits. The guide to choosing an attribution approach places this loop alongside attribution, which handles decisions below the model's grain, and the comparison of marketing mix modelling and multi-touch attribution covers how the model and path-based credit divide the remaining work.

Common mistakes

  • Treating a mix model's output as proven cause without testing the channels whose estimates drive the largest moves.
  • Generalising one test to the whole channel, at any spend level and in any season.
  • Testing a channel definition the model does not use, so the result cannot be fed back into it.
  • Moving the whole budget on one refit instead of in steps that can be tested and corrected.
  • Building a model and never refreshing it, so a one-off report ends up guiding years of plans.
  • Discarding either result when they disagree, instead of asking why and refitting with the test as a calibration point.
  • Starting with a model when the business lacks history or variation, when a test on the largest line would have answered the real question sooner.

Frequently asked questions

If I run incrementality tests, do I still need a mix model?

Only once the budget is spread across enough channels that testing each one in turn would take too long. A test answers one channel at one spend level in one period; a plan for the whole budget needs every channel's response at once, including channels that are hard to test such as television or sponsorship. A business with two or three digital channels can often run on tests and good attribution; one with a broad mix, including offline media, usually needs the model as well.

What happens when the model and a test disagree?

Trust the test for the channel, period and spend level it measured, then ask why the model differed. Common reasons are a channel whose spend moved in step with another, a missing control such as a promotion, or a test run at a spend level the model has rarely seen. The fix is to feed the test into the model as a calibration point and refit, not to discard either result. Repeated disagreement on the same channel is a sign the model's specification needs work.

Is a marketing mix model causal?

Partly. It estimates how outcomes moved with spend over time, adjusted for the controls it includes, and that is closer to cause than attribution gets. But it cannot rule out everything that moved with spend, and channels that rise and fall together are hard to separate. Calibrating it with experiments, which are causal by design, is what lets its estimates carry budget decisions with more confidence.

Terms used on this page

Where this page is used

Part of these learning paths

Also related

Sources

  1. IPA — Institute of Practitioners in Advertising

Related