What this checkpoint covers
Ottervale Kitchenware is an invented online retailer of pans, knives and small appliances. It advertises through Google Shopping, Meta prospecting and remarketing, and sponsored listings on a marketplace that can only be switched on or off nationally. The questions draw on the incrementality testing guide, the matched-region method in the bidding strategy geo split test and the comparison of mix modelling and multi-touch attribution, and they assume the flower delivery holdout exercise.
Task
Write an answer and its reason for every question before reading the explanations; it counts only if the reasoning matches. Then name the largest budget line in a programme you know and the design you would use to test it.
Questions
- A geo test shows Meta prospecting caused roughly two-fifths of the orders Meta credited to it over the same weeks. The head of ecommerce proposes splitting next year's budget by each channel's share of credited orders. What should budgets follow instead?
- Ottervale wants to know what three lines add: remarketing to recent visitors, non-brand Shopping, and the marketplace listings. Which design suits each?
- For a Shopping geo test, the analyst assigns regions at random without looking at history. The treated group ends up with the two fastest-growing regions, and the ratio between groups was drifting before launch. What do you change?
- Six weeks into a test planned for six, the result looks promising and Black Friday is three weeks away. The owner wants to run on through the peak "to get more data" and agree the threshold once the result is in. What do you advise?
- The finance director wants a mix model built now from one year of weekly data in which spend barely moved, just as Ottervale starts a printed catalogue. What can a model answer here, and what does it need first?
- The Shopping test ends with a range of lift running from a small loss to a large gain, centred above zero. One colleague calls it a win, another a failure. How do you read it?
- Remarketing and the marketplace listings each tested as weakly incremental. Finance wants to cut both next month. What is the risk?
Answers explained
- Incremental cost per order at the tested spend. Credited orders include buyers who would have ordered anyway, and every platform claims its own share of the same customers. Budgets follow what each channel causes, judged against what an order is worth; platform AttributionAssigning conversion credit to touchpoints. stays useful for optimising inside each channel, discounted by the factor the test found (from lift to budget decisions).
- A user holdout, a geo split and a switchback. Remarketing audiences can be split by person, so hold out a random share. Shopping can be targeted by region, so match regions and withhold or reduce spend in some. The marketplace cannot be split either way, so alternate on and off periods against a comparison series and accept a weaker read; a plain before-and-after is the last resort (test designs).
- Match on history, then randomise within pairs. Pair regions whose orders moved together in the pre-period, assign one of each pair at random, and confirm the ratio was stable before launch. Leave out regions whose growth or behaviour no partner matches (design and pitfalls).
- Stop as planned and keep the peak out. The threshold belongs in the plan before launch; set afterwards, it cannot fail. A peak changes demand, prices and competition unevenly across regions, so extending into it mixes two different tests. Read the planned weeks after the cool-down, and plan a separate peak test if one is needed (duration).
- Little yet. A mix model answers how to split budget across channels, offline ones such as the catalogue included, and where each saturates. It needs a long history in which spend varied; a flat year gives it nothing to learn from. Vary spend deliberately, keep running tests, and build MMMStatistical model of channel impact. once the history exists, calibrated against those tests (when to choose mix modelling).
- Neither. A range that includes zero has not shown an effect, and a wide one may simply be too small a test. Compare it with the minimum effect the plan said it could detect; if the test was underpowered, rerun with more regions, a longer read or a deeper cut. Meanwhile hold spend at a level whose cost would be tolerable at the pessimistic end (reading lift).
- Each was tested with the other running. Their combined IncrementalityConversions caused by marketing. can exceed the sum of the two results, because each may have been catching demand the other would otherwise have caught. Cut one, retest, then decide on the second (sequencing large changes).
Where to go next
If you missed the design questions, rework the region and timing steps of the flower delivery exercise. If you missed the mix modelling question, reread the decision table in the mix modelling comparison. Then continue to profit targets with the pet supplement exercise, or revisit budget planning on the intermediate performance marketing level.
