Why this matters
"Which channels work?" hides several questions, each with a different cost of being wrong. The guide to choosing an attribution approach argues for starting from the decision: the more money and the less reversible the choice, the stronger the evidence it needs. The comparison of mix modelling and multi-touch attribution sets out what each method can see. This exercise turns both into a working table for one business.
The broker and its questions
Harbourstone Insurance is an invented broker selling home insurance direct to consumers. Most policies are bought online after a quote; some are bought by phone. It advertises on paid search, on regional television and through listings on price comparison sites, and it emails existing customers before renewal.
Harbourstone's spend, data and questions
| Channel | Annual spend | What tracking can see |
|---|---|---|
| Paid search | £640,000 | Clicks, quotes and online sales by campaign and keyword |
| Regional television | £520,000 | Nothing at user level; airtime by region and week |
| Price comparison sites | £780,000 | Sales reported by each site, paid per sale |
| Renewal email | £30,000 | Opens, clicks and renewals of emailed customers |
Weekly history of spend, quotes and policies sold by region exists for 150 weeks. Television has run in four of the broker's nine regions, with gaps over summer. Phone sales are about 20% of new policies and carry no campaign data. Around a third of online visitors decline analytics consent.
The questions, as the marketing director wrote them:
| No. | Question |
|---|---|
| A | Which search keywords should get more of the search budget this month? |
| B | Is television worth what we pay for it? |
| C | Should we cut comparison-site spend by half? |
| D | How should next year's budget split across search, television and comparison sites? |
| E | Does television make people search for us by name? |
| F | Does the renewal email keep customers who would otherwise leave? |
| G | What return on marketing do we report to the board? |
Matching questions to methods
For each question, note the decision, how soon the answer is needed and whether the channel is visible at user level. Then apply the division of labour the comparison sets out: AttributionAssigning conversion credit to touchpoints. for fast, detailed choices inside digital channels; mix modelling (MMMStatistical model of channel impact.) for channel-level planning, including media no tag can see; and IncrementalityConversions caused by marketing. experiments for up-or-down decisions on a single line, and for checking the other two.
Task
- For each question, write the decision it informs, who makes it and how often.
- Mark whether the channels involved are visible to user-level tracking, and whether the decision is cheap to reverse.
- Assign one primary method to each question, and a second where it would check the first.
- Write the caveat the answer must carry when it is reported.
- Recommend which piece of measurement Harbourstone should start first, and why.
Worked answer
| Question | Decision and time frame | Method | Caveat to report |
|---|---|---|---|
| A: search keywords | Bids and budgets inside search, weekly; cheap to reverse | Platform attribution, read beside analytics | Credit is association along tracked paths, not cause; phone sales are missing |
| B: is television worth it | Keep, cut or change television, annually; expensive to reverse | Mix model, checked by a regional test | The estimate depends on history and controls and carries an interval |
| C: halve comparison sites | Up-or-down on the largest line, this year; expensive to reverse | Experiment | A result holds for the spend level and period tested |
| D: next year's split | Allocation across channels, annually | Mix model, calibrated by tests | Response curves are reliable only within spend levels already seen |
| E: television and brand search | How television and search budgets are judged together | Regional experiment, or a model with brand search as an outcome | Brand search is an outcome of television, not a control |
| F: renewal email | Keep or change the programme, quarterly | Randomised holdout of customers | Clicks and opens show attention, not retention |
| G: board return | How marketing's return is reported | Sales and policies from the policy system, split by the methods above | Platform-reported sales are claims and are never summed |
The reasoning behind the harder rows:
- B and D need a mix model because television produces no clicks; no path-based method can credit it. Television running in only some regions, with gaps, gives the model the variation it needs.
- C needs an experiment because comparison sites report every sale they touch, including people who would have bought anyway. Whether the sites can withhold or reduce listings in selected regions or weeks is a question for the commercial agreement; if they can, a geo or time-based test, designed with the budget increase incrementality test as a starting pattern, gives the evidence. Halving the largest line on attributed figures alone is the case the decision rule forbids.
- E is a test or a model, not attribution. Attribution would credit the brand search click and never see the advert that prompted it. In a mix model, treating brand search as a control would hand television's effect to search, as the mix modelling basics guide warns.
- F is the simplest experiment available. Withhold the email from a random share of renewing customers and compare renewal rates.
The note to the marketing director
Start with question C. Comparison sites take £780,000 a year, the largest line, and the only evidence for them is their own reporting. A test this year protects more money than any other measurement can. In parallel, assemble the 150 weeks of regional data into a clean weekly table, so a mix model for B and D can be built for next year's planning round with the comparison test as its calibration point. Keep attribution for weekly search decisions, and run the renewal email holdout, which costs almost nothing.
Common mistakes
- Answering a budget question with attribution. Credit shares are hypotheses about roles, not evidence of cause.
- Forgetting phone sales. Any method fed only online data understates channels that drive calls.
- Using brand search as a control. It moves because of television, so it steals television's credit.
- Reporting answers without their caveats. Unlabelled figures get compared with ones produced differently.
Next, check the module in the measurement approach checkpoint, or return to the intermediate marketing analytics level.
