Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Marketing Analytics and Reporting › Expert

Reviewing an Agency's Marketing Mix Model for a Mattress Brand

An agency's mix model for an invented mattress brand credits podcasts with a return far above every other channel and recommends moving budget there. You check the dataset behind it, review the specification, fit, holdout and stability, test the story against what the brand knows and read the uncertainty, then decide which result to calibrate with an experiment and what the brand does with the model meanwhile.
ExerciseExpert levelModule 2: Mix models finance can trust5 minUpdated

Why this matters

The advanced marketing analytics level forecast spend and turned single tests into evidence finance accepts. A delivered mix model (MMM) claims more: every channel at once, with a recommended split. The guide to evaluating a marketing mix model is written for a reviewer who did not build it and must decide whether to act. Its first questions concern the dataset, which the tutorial on preparing data for a marketing mix model assembles, and that is where most models that should not move money fail.

The mattress brand and the model

Gruzzemop Sleep is an invented direct-to-consumer mattress brand selling online with a home trial and advertising on television, podcasts, paid social and search. Its media agency built the model and buys the media.

Illustrative example — not a real result

Gruzzemop Sleep: the agency's model

  • History: 130 weeks. Outcome: purchases recorded by the website's analytics tag. The order system holds 12% more orders over the period, and 9% of orders are returned during the trial.
  • Podcast spend is taken from the network's invoices, each dated the first week of a quarter. The brand's quarterly sale runs in the first two weeks of each quarter.
  • Controls: season, holidays and a sale-week flag. Discount depth is not included; the deepest sales, at 25% off rather than the usual 10%, all carried podcast promo codes.
  • A list of sources but no data log. The podcast prior came from the agency's other clients.
  • Average error on fitted weeks: 4%. On the last 12 weeks, held out: 15%.
  • Podcast marginal ROAS when refitted without the first 13 weeks: 3.1; with its prior loosened: 2.4.
ChannelAnnual spendMarginal ROASIntervalRecommendation
Television£1,100,0001.91.3 to 2.6Cut by £150,000
Paid social£780,0001.61.2 to 2.0Cut by £150,000
Paid search£520,0002.21.8 to 2.7Hold
Podcasts£240,0006.80.6 to 12.4Raise to £540,000

Annual podcast spend has never exceeded £240,000, and no channel has ever been tested.

Data, fit, story and uncertainty

Review in the guide's order, writing the decision rule before reading the recommendation: data first, since a misdated series invalidates everything after it; then fit and stability, the story, calibration and uncertainty. The Diminishing Returns curves mean something only once their inputs are sound.

Task

  1. Check the outcome, the podcast series, the controls and the documentation against what a model needs.
  2. Review the specification, fit, holdout and stability, and say what each tells you.
  3. Test the story: which outputs are plausible and which need corroboration.
  4. Write a decision rule, then read the uncertainty against it.
  5. Score the model on the guide's scorecard and give a verdict.
  6. Say which result to calibrate with an experiment, and how to shape the test.
  7. Write what the brand should and should not do now.

Worked answer

The dataset. The outcome comes from the analytics tag, which misses orders and ignores trial returns; it should come from the order system, with an explicit decision on returns, confirmed by finance. Podcast spend is dated by invoice, so it lands in the sale weeks rather than the weeks episodes ran; it must be rebuilt from the booking schedule and air dates. Discount depth is missing, so the deepest sales lend their effect to whichever series moved with them, and the misdated podcast series moved with them exactly. Without a data log nobody knows what was fitted.

Specification, fit and stability. The close fit proves little, since quite different splits between channels can reproduce one history. Held-out error far above fitted error suggests memorising. The refits settle it: podcasts lose over half their return when early weeks are dropped or the prior is loosened, so the estimate rests on a borrowed prior, not on Gruzzemop's data.

The story. A small channel with a return far above the rest is one of the guide's signs that a series has absorbed something else: here, the sales. The optimiser then pushes podcasts beyond any spend the history contains, which is extrapolation, not evidence.

Uncertainty and the rule. The rule, written first: move money only where the move holds at the pessimistic ends of both channels' intervals, not only at the midpoints. The podcast interval is wide enough to include losing money, so the move fails.

Verdict. On the guide's scorecard, fit and holdout is not met, because the held-out error is far above the fitted error. Plausibility is partly met. Data, controls, specification, stability, calibration and uncertainty are not met. Brief is met only if the agency names the budget decision the model was built for. Gaps in data and controls mean rebuild before anyone reads the results.

Calibration. After the rebuild, podcasts get the first Incrementality test: the largest proposed move, the widest interval, never tested. Shape it to the model's grain, as the comparison of mix modelling and incrementality testing describes: all podcast spend, the order-system outcome, a spend level the history contains, recorded for every refit. If no controlled test can be designed, planned pauses staggered away from the sales give the refit variation.

What to do now. Hold every budget. Ask the agency, as deliverables, for the rebuilt outcome and podcast series, a discount-depth control, a data log, holdout and stability results and intervals on every marginal return; the guide to selecting, briefing and reviewing agencies writes these into scope. Someone with no stake in the media buy reviews the refit. Do not quote the podcast return to the board, and do not discard the model: rebuilt and calibrated, it alone weighs television beside the rest, as the marketing mix modelling basics explain. Record the decision and review date in the budget reallocation memo template.

Illustrative example — not a real result

Gruzzemop Sleep: the numbers behind the verdict

Held-out error, 15%, is nearly four times the fitted 4%. Podcast marginal ROAS of 6.8 falls to 3.1 or 2.4 under small changes, a drop of between 54% and 65%. The pessimistic end of the podcast interval, 0.6, is below the pessimistic ends for television (1.3) and paid social (1.2), and returns 60p per pound spent. The proposed £540,000 is 2.25 times the highest annual podcast spend in the history.

Common mistakes

  • Reading the results before the data. The misdated podcast series invalidates everything after it.
  • Taking the fit as proof. The holdout and refits disagree.
  • Moving half the budget as a compromise. It is still unsupported.
  • Accepting a prior borrowed from other clients. Gruzzemop's own data never got the chance to disagree with it.
  • Letting the buying agency review itself. The team that buys the podcasts should not be the only one to judge their return.

Pages this lesson applies

Terms used on this page

Also related

Related