Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Marketing Analytics and Reporting › Intermediate

Choosing a Measurement Approach for an Insurance Broker

An exercise in starting from the decision rather than the method: sort an invented insurance broker's questions by the decision each informs and how soon it is needed, assign each to attribution, marketing mix modelling or an experiment, and write the caveat that travels with every answer. The worked answer shows why the broker needs all three, each in its own place.
ExerciseIntermediate levelModule 2: Choosing a measurement approach5 minUpdated

Why this matters

"Which channels work?" hides several questions, each with a different cost of being wrong. The guide to choosing an attribution approach argues for starting from the decision: the more money and the less reversible the choice, the stronger the evidence it needs. The comparison of mix modelling and multi-touch attribution sets out what each method can see. This exercise turns both into a working table for one business.

The broker and its questions

Harbourstone Insurance is an invented broker selling home insurance direct to consumers. Most policies are bought online after a quote; some are bought by phone. It advertises on paid search, on regional television and through listings on price comparison sites, and it emails existing customers before renewal.

Illustrative example — not a real result

Harbourstone's spend, data and questions

ChannelAnnual spendWhat tracking can see
Paid search£640,000Clicks, quotes and online sales by campaign and keyword
Regional television£520,000Nothing at user level; airtime by region and week
Price comparison sites£780,000Sales reported by each site, paid per sale
Renewal email£30,000Opens, clicks and renewals of emailed customers

Weekly history of spend, quotes and policies sold by region exists for 150 weeks. Television has run in four of the broker's nine regions, with gaps over summer. Phone sales are about 20% of new policies and carry no campaign data. Around a third of online visitors decline analytics consent.

The questions, as the marketing director wrote them:

No.Question
AWhich search keywords should get more of the search budget this month?
BIs television worth what we pay for it?
CShould we cut comparison-site spend by half?
DHow should next year's budget split across search, television and comparison sites?
EDoes television make people search for us by name?
FDoes the renewal email keep customers who would otherwise leave?
GWhat return on marketing do we report to the board?

Matching questions to methods

For each question, note the decision, how soon the answer is needed and whether the channel is visible at user level. Then apply the division of labour the comparison sets out: Attribution for fast, detailed choices inside digital channels; mix modelling (MMM) for channel-level planning, including media no tag can see; and Incrementality experiments for up-or-down decisions on a single line, and for checking the other two.

Task

  1. For each question, write the decision it informs, who makes it and how often.
  2. Mark whether the channels involved are visible to user-level tracking, and whether the decision is cheap to reverse.
  3. Assign one primary method to each question, and a second where it would check the first.
  4. Write the caveat the answer must carry when it is reported.
  5. Recommend which piece of measurement Harbourstone should start first, and why.

Worked answer

QuestionDecision and time frameMethodCaveat to report
A: search keywordsBids and budgets inside search, weekly; cheap to reversePlatform attribution, read beside analyticsCredit is association along tracked paths, not cause; phone sales are missing
B: is television worth itKeep, cut or change television, annually; expensive to reverseMix model, checked by a regional testThe estimate depends on history and controls and carries an interval
C: halve comparison sitesUp-or-down on the largest line, this year; expensive to reverseExperimentA result holds for the spend level and period tested
D: next year's splitAllocation across channels, annuallyMix model, calibrated by testsResponse curves are reliable only within spend levels already seen
E: television and brand searchHow television and search budgets are judged togetherRegional experiment, or a model with brand search as an outcomeBrand search is an outcome of television, not a control
F: renewal emailKeep or change the programme, quarterlyRandomised holdout of customersClicks and opens show attention, not retention
G: board returnHow marketing's return is reportedSales and policies from the policy system, split by the methods abovePlatform-reported sales are claims and are never summed

The reasoning behind the harder rows:

  • B and D need a mix model because television produces no clicks; no path-based method can credit it. Television running in only some regions, with gaps, gives the model the variation it needs.
  • C needs an experiment because comparison sites report every sale they touch, including people who would have bought anyway. Whether the sites can withhold or reduce listings in selected regions or weeks is a question for the commercial agreement; if they can, a geo or time-based test, designed with the budget increase incrementality test as a starting pattern, gives the evidence. Halving the largest line on attributed figures alone is the case the decision rule forbids.
  • E is a test or a model, not attribution. Attribution would credit the brand search click and never see the advert that prompted it. In a mix model, treating brand search as a control would hand television's effect to search, as the mix modelling basics guide warns.
  • F is the simplest experiment available. Withhold the email from a random share of renewing customers and compare renewal rates.
Illustrative example — not a real result

The note to the marketing director

Start with question C. Comparison sites take £780,000 a year, the largest line, and the only evidence for them is their own reporting. A test this year protects more money than any other measurement can. In parallel, assemble the 150 weeks of regional data into a clean weekly table, so a mix model for B and D can be built for next year's planning round with the comparison test as its calibration point. Keep attribution for weekly search decisions, and run the renewal email holdout, which costs almost nothing.

Common mistakes

  • Answering a budget question with attribution. Credit shares are hypotheses about roles, not evidence of cause.
  • Forgetting phone sales. Any method fed only online data understates channels that drive calls.
  • Using brand search as a control. It moves because of television, so it steals television's credit.
  • Reporting answers without their caveats. Unlabelled figures get compared with ones produced differently.

Next, check the module in the measurement approach checkpoint, or return to the intermediate marketing analytics level.

Pages this lesson applies

Terms used on this page

Also related

Related