Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Marketing Analytics › Attribution

Evaluating a Marketing Mix Model

How to judge a marketing mix model before acting on it: data and specification, fit and stability, whether the story makes sense, calibration and uncertainty.
Guideadvanced11 min readUpdated

Why a model needs a reviewer

A mix model is fitted to a few years of weekly totals, which is not much data, and asked to separate the effects of many channels, prices, promotions and seasons that often moved together. There are usually many combinations of channel effects that explain the history about equally well. The model picks one, guided by its specification and its assumptions, and presents it with the same confidence whether the data pinned it down or not.

That is not a flaw to be ashamed of; it is the nature of the method, and the basics guide is clear that two competent modellers given the same data will not produce identical curves. It does mean that a model's outputs should be treated as a claim to be examined. The examination has three parts: is the model built on the right data for the decision, does it behave like a model that has learned something real, and is its uncertainty small enough for the decision at hand. A reviewer who asks those questions every time, in the same order, will catch most of the models that should not move money.

The review is also the moment to separate the model from its sponsor. Whoever built it has made dozens of choices, and whoever buys a channel has an interest in how that channel comes out. Neither makes the model wrong. Both make an independent read worth having.

The brief and the data

Start with what the model was for, then what it was fed.

The brief. Which decision is the model meant to inform, and does its outcome, channel list and history match that decision? A model built to split a digital budget may have lumped all offline media into one series; it cannot then be used to decide between radio and outdoor.

The outcome. Is it taken from the system of record, such as orders or revenue from the order system or ledger, and does it include every route to purchase? A model of one platform's conversions or of website sales alone cannot see the effects that make mix modelling worth doing.

The grain and the history. Weekly data over a history long enough to contain more than one full seasonal cycle is the basic expectation the basics guide sets out. Ask how the history was chosen and whether it crosses breaks such as a change of order system or business model.

The channel definitions and the controls. Ask for the mapping from campaigns and invoices to channels, and for the list of controls with the reason for each. Then ask what was left out. An omitted price change or promotion calendar is the most common reason a media channel is over-credited.

The data log. A competent model comes with a record of every source, definition, gap and fix; the guide to preparing data for a mix model describes what it should contain. No data log is a finding in itself: nobody, including the modeller, can then say what the model was fitted to.

Specification choices

The specification is the set of modelling decisions that turn the data into estimates. A reviewer does not need to choose them, but should see them written down with a reason for each.

  • Carry-over form. How each channel's effect is spread over later weeks, whether it decays from the week of spend or builds to a peak after a delay, and whether the rate was estimated per channel or fixed.
  • Saturation form. The curve that turns carried-over spend into effect and produces Diminishing Returns, and over what range of spend it was actually observed.
  • Priors and constraints. In models that use them, the assumptions about each channel's likely effect before seeing the data, and any constraints such as effects that cannot be negative. Ask where each prior came from: experiments, earlier models, industry convention or the media plan. A prior that is never challenged by the data simply returns as the answer.
  • Trend and seasonality. How the baseline is allowed to change over time. A baseline that is too rigid pushes slow growth into the media channels that grew alongside it.
  • Grouping and transformations. Which channels were combined, which were split, and whether the outcome or inputs were transformed.

The question for each choice is not whether it is the right one but whether the conclusions depend on it. Ask what changes when a reasonable alternative is used. A channel whose contribution survives a different carry-over form and a weaker prior is better supported than one that only appears under one set of choices.

Fit, holdout and stability

Fit is the first check and the least informative one. A model that cannot track the history week by week has missed something, usually a control, and its residuals, the weeks it gets most wrong, will often point to what. But a close fit says little about whether the split between channels is right, because many splits fit equally well.

Two checks say more. The holdout check fits the model on the history without its most recent weeks and asks it to predict them. A model that has learned real relationships predicts unseen weeks reasonably; one that has memorised the history does not. Ask which weeks were held out and why, and compare the prediction error with the error on fitted weeks.

The stability check refits the model with small, reasonable changes: a few weeks dropped, a quarter of new data added, a prior loosened, an alternative carry-over form. Then it compares channel contributions and marginal returns across the refits. Stable estimates mean the data is doing the work; estimates that swing when a few weeks move mean the model is fragile for that channel, and the decision should not rest on it. A modeller who has not run these checks can usually run them quickly, and the request is a reasonable part of any brief.

Does the story make sense?

A model's outputs tell a story about how the business works. Read it the way a sceptical channel manager would.

  • The baseline. For an established brand most of the outcome sits in the baseline, as the basics guide explains. A model that gives media most of the outcome for a mature business with loyal customers is making an unusual claim that needs unusual evidence.
  • Carry-over. Do the estimated decay rates fit what the channel teams know? Search and retargeting usually act quickly; television and outdoor may act over weeks. A search effect that lingers for months, or a television effect that vanishes within a week, deserves a question.
  • Curves. Is the channel the paid search team says is saturated shown as saturated? Is a channel at a fraction of its historical spend shown on the steep part of its curve? Curves that contradict long operational experience may be right, but they need corroboration.
  • Signs and sizes. A media channel with a negative contribution, a tiny channel with an enormous return, or a channel whose contribution tracks its spend almost exactly are all signs the model has absorbed something else, often an omitted control or a channel it could not separate from another.
  • The optimised allocation. The basics guide describes it as a direction to test, not an instruction. Check how far it pushes each channel beyond the spend levels the history contains.

Calibration with experiments

The strongest check on a model is an experiment it did not see. A well-run test measures the Incrementality of one channel at one spend level in one period; the model should agree with it within its uncertainty. The incrementality testing guide covers how such tests are run, and the comparison of mix modelling and incrementality testing explains how results are used to calibrate a model.

Ask three questions. Has the model been compared with any experiment, and did it agree? Were experiment results used in the model, for example as priors or constraints, and if so, is that disclosed, since agreement with a test that was fed into the model is not independent? And which channels have never been tested? Those are the ones where the model's estimate is resting on history and assumptions alone, and the largest of them is the natural next test. The academy exercise on reviewing an agency's marketing mix model for a mattress brand practises these checks on a model that credits podcasts with a return far above every other channel, ending with the result to calibrate first.

Uncertainty and the decision

Every estimate the model produces has a range of plausible values: a Confidence Interval in a classical model, or a credible interval in a Bayesian one. Ask for intervals on every channel's contribution and marginal return, not only the central estimates, and read the decision against the whole range.

The decision rule should be written before the results are read. A sound rule moves budget only where the direction of the move holds across the plausible range: the channel receiving money has a higher Marginal ROAS than the channel losing it at the pessimistic ends of both intervals, not only at their midpoints. Where the intervals overlap, the model has not settled the question, and the right response is a test rather than a reallocation. Moves should also stay within the spend levels the history covers, since curves beyond them are extrapolation. The budget allocation guide describes allocating by marginal return, and the budget reallocation memo template is where the evidence, its uncertainty and the review date are written down before money moves. The academy checkpoint on mix model data, review and calibration tests deciding under uncertainty with ranges rather than point estimates, beside specification, fit, holdout and stability.

A review scorecard

The scorecard turns the questions into a decision. Score each line as met, partly met or not met, and decide the overall verdict from the pattern rather than a total.

AreaQuestionMet when
BriefDoes the model's outcome, channel list and history match the decision?The decision is named and the model's scope covers it
DataIs the outcome from the system of record, and is there a data log?Finance confirms the outcome; every source and fix is documented
ControlsAre price, promotions, season and known events included?The controls list is complete and each omission is justified
SpecificationAre carry-over, saturation, priors and trend stated with reasons?Written down, with sensitivity to reasonable alternatives shown
Fit and holdoutDoes it track history and predict held-out weeks?Residuals show no obvious missing driver; holdout error is close to fitted error
StabilityDo estimates hold when refitted with small changes?Contributions and marginal returns move little across refits
PlausibilityDo baseline, carry-over and curves fit what the business knows?Surprises are few and each has an explanation
CalibrationDoes it agree with independent experiments?Agreement within uncertainty on the channels tested
UncertaintyAre intervals reported, and does the decision hold across them?The proposed moves survive the pessimistic ends of the intervals

A model that meets every line can inform the decision directly. A model with gaps in calibration or uncertainty can inform the direction of a decision, confirmed by a test before the full amount moves. A model with gaps in the brief, the data or the controls should be rebuilt before anyone reads its results, however plausible they look.

Illustrative example — not a real result

Oatlantern Cereals reviews an agency's model

Oatlantern Cereals, an invented breakfast cereal brand, receives a model from its media agency built on 143 weeks of retail sales. It credits outdoor posters with 58 per cent of the growth over the period and recommends moving £1.2 million from paid social and in-store promotion into outdoor. The marketing analyst scores it against the scorecard. Brief, fit and specification are met: the agency's documentation is thorough and the model predicts the twelve held-out weeks well. Controls are not met. The model has no price series, and the analyst finds that the two years of outdoor growth coincided with a recipe change and a 9 per cent price cut on the core range, both of which the outdoor series moved with. Uncertainty is not met either: the interval on outdoor's marginal return runs from 0.4 to 3.1, wide enough to include losing money. Calibration is not met because outdoor has never been tested. The verdict is test first. The agency adds the price series and refits, which cuts outdoor's share of growth to 31 per cent with an interval that still overlaps paid social's. Oatlantern books a regional outdoor test across matched cities for the next spring, and no budget moves until the test reads out.

The guide to selecting, briefing and reviewing agencies covers how to write the review into an agency's scope, so that the data log, the specification and the stability checks are deliverables rather than favours. The attribution hub sets the model beside attribution and experiments in the wider measurement approach.

Common mistakes

  • Accepting a fit statistic as proof. Many channel splits fit the same history.
  • Reviewing the results before the data. A missing control invalidates every number after it.
  • Taking priors on trust. An unchallenged prior comes back as the answer.
  • Reading midpoints. The decision has to survive the whole interval.
  • Acting on the optimiser's split. It extrapolates beyond the spend the history contains.
  • Counting agreement with a test the model was fed. It is not independent evidence.
  • Letting the model's sponsor be its only reviewer. Interest shapes specification, however honestly.
  • Never refreshing the review. A model that passed last year has to pass again after each refit.

Frequently asked questions

What fit statistic tells me a marketing mix model is good?

None on its own. A model can fit the history closely and still give the wrong split between channels, because several different splits can explain the same total equally well, especially when channels moved together. Fit is a necessary check, not a sufficient one. The more telling questions are whether the model predicts weeks it was not fitted on, whether its channel estimates hold steady when it is refitted with a few weeks removed or a different reasonable specification, and whether its estimates agree with experiments. Be wary of anyone who offers a single fit figure as proof of accuracy.

Should the modeller be the same team that buys the media?

It can be, but then the review matters even more. A team that is judged on a channel's results has an interest, however honest, in a model that flatters it, and specification choices give plenty of room for that interest to show. Whoever builds the model, ask for the data log, the specification and the priors in writing, have someone without a stake in the answer review them, and calibrate against experiments that the buying team did not design alone.

What should happen when the model and an experiment disagree?

Treat the experiment as the stronger evidence for the channel and conditions it tested, and treat the disagreement as information about the model. A well-run test measures what the channel caused at one spend level in one period; the model estimates the same thing from history and assumptions. Ask the modeller to calibrate the channel to the test result and refit, and look at what else moves when they do. If calibrating one channel makes others implausible, the model's split between channels was not well determined by the data in the first place.

Terms used on this page

Frameworks

Where this page is used

Part of these learning paths

Also related

Related