Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Analytics › Advanced

Checkpoint: Incrementality, Mix Modelling and Data-Driven Attribution

Scenario questions set in an invented car rental firm test whether you can match causal and modelled measurement to the decisions they serve. They cover holdout against geo designs, contamination and duration, adstock and saturation, the blind spots of data-driven attribution, calibrating a mix model and choosing a method per budget decision.
CheckpointAdvanced levelModule 2: Causal measurement4 minUpdated

What this checkpoint covers

Durlston Car Rental is an invented car rental firm with town and airport branches. Customers book online or by phone, often weeks before collecting a car, and the firm advertises on radio, paid search, paid social and by email to past renters. The questions draw on the incrementality testing guide, the data-driven attribution guide and the marketing mix modelling basics guide.

Task

Answer each question in a sentence or two and give the reason. Then compare with the explanations, count a right action with a wrong reason as a miss, and reread the linked page for each miss.

Questions

  1. Durlston wants to know what two things add: its reminder emails to past renters, and its regional radio. Which test design fits each, and why not the same one?
  2. A geo test on paid social uses booking branch as the outcome region. Many customers book from home for collection at an airport in another region. What goes wrong, and how long should the test run?
  3. A mix model gives radio a long carry-over and shows paid search with a high average return but an almost flat curve at current spend. What does each finding mean for next year's budget?
  4. Data-driven Attribution gives radio no credit and paid search a great deal. Which journeys can the model not see, and where does their credit land?
  5. A clean geo test finds paid social causes far fewer bookings than the mix model's curve implies at the same spend. What should happen to the model?
  6. The mix model's optimiser suggests tripling radio. The head of marketing wants to do it next quarter. What do you advise?
  7. A holdout on paid social shows no significant lift, with a wide interval. A colleague reports that paid social does nothing. Is that right?
  8. Durlston has three decisions: the annual split between radio, search and social; whether to keep remarketing; and which keywords get more budget. Which method serves each?

Answers explained

  1. A user-level holdout for email, a geo test for radio. Email goes to an owned list, so a random share can be withheld and both groups' bookings compared. Radio cannot be withheld from individuals, but it can be bought by region, and a geo test counts every booking a region produces. See test designs.
  2. The outcome is measured in the wrong place. Exposure happens where people live; booking branch moves bookings into other regions and mixes test and control. Count bookings by the booker's home postcode. Run at least one full booking cycle, which here includes long lead times, then a cool-down so delayed bookings land in both groups. See choosing cells, size and duration.
  3. Radio works over weeks; search is near saturation. Long adstock means radio's effect outlasts its weeks on air, so short-window reports under-credit it and tests need patience. A flat curve means the next unit of search spend buys little, whatever the average. Plan on marginal return, report on average. See adstock and saturation.
  4. Anything without a recorded click. Radio, phone bookings, consent-declined visits, cross-device journeys and touches outside the lookback window never enter the paths. Their effect lands on the next recorded touch, usually brand search or direct. See what it cannot observe.
  5. Calibrate it to the test. The test is a measured point on the paid social response curve at that spend level, and the model should pass through it rather than average the two answers. Record the test's spend, period and regions so the calibration is not stretched beyond them. See mix modelling, attribution and incrementality together.
  6. Test before tripling. Response curves are well estimated only within the spend range observed; tripling radio is a guess about territory the model has never seen. Move part of the way and run a geo test on the extra spend. The MMM output is a direction to test, not an instruction. See outputs and what they mean.
  7. No. An interval that includes zero has not shown an effect, but a wide one has not ruled out a large one either. Compare the interval with the minimum detectable effect set before launch: if the test could not detect an effect that would change the decision, it was underpowered. Rerun with more units or time. See reading lift and uncertainty.
  8. Mix model, test, attribution. The annual channel split is a mix-model question, calibrated by tests. Keeping remarketing is a budget decision on a contested line, so it needs an Incrementality test. Keyword allocation sits below the model's grain and suits attribution. See from lift to budget decisions.

Where to go next

If you missed the design questions, redo the geo holdout exercise for a juice delivery firm. Then continue to metric governance with the metric definitions exercise for a training provider. To revisit the attribution models themselves, return to the intermediate analytics level.

Pages this lesson applies

Terms used on this page

Also related

Related