Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Marketing Analytics › Attribution

Measuring Direct Mail

How to measure direct mail with a holdout drawn from the mailing list, response codes and tagged addresses, and why matching buyers to the list overstates it.
Guideintermediate11 min readUpdated

Why addressed mail can be measured differently

The guide to measuring TV, radio, podcasts and print works up a ladder of evidence for media whose audience is unknown: response mechanisms that count a floor, response around airings, search lift and regional tests. Each rung exists because nobody can split the audience into people who saw the advert and people who did not. With addressed mail that split is in the business's hands. The mailing file names every recipient, and the order system or CRM records what every customer on the list bought, by any route, before and after the drop.

That knowledge supports the design the incrementality testing guide calls the user-level holdout, with a randomised list of people withheld, which it describes as the cleanest design when it is possible and names for owned channels. A mailing to existing customers or prospects is one of the few places outside email where it is fully possible: the business decides who receives each piece, so it can decide at random who does not.

The same knowledge produces the commonest mistake. Because every buyer can be looked up on the list, it is easy to count every listed buyer as a response, and that count is always flattering. The rest of this guide builds the measurement around the holdout and treats the other numbers as supporting evidence.

EvidenceWhat it showsWhere it sits
Random holdout from the mailing listWhat the mailing changed in total purchasesTop: a controlled comparison
Codes, tagged addresses and askingWho used a route that names the mailingFloors, useful for comparing versions
MatchbackEvery listed customer who boughtAn upper limit, not an effect

What to record for every mailing

Measurement usually fails because the facts of a mailing were never written down. Record these before the drop, in the campaign record the guide to campaign records in the CRM says a direct mail drop deserves, named with the same convention as the rest of the business's campaigns:

FieldWhy it matters
Selection rules and the list sourceDefines who could have been mailed, and so who the result applies to
People removed before the draw, and whySeparates people who may not be mailed from the holdout, which is a different group
The holdout flag for every person on the final listThe only way to compare mailed and unmailed customers later
Version, offer and segment for every mailed personLets codes, addresses and outcomes be read per version
Drop date and the expected arrival periodStarts the response window
Codes, short addresses and phone routes per versionConnects responses to the version that carried them
Costs from the business's own invoicesFeeds the costing at the end
The response window and the reading date, decided nowStops the result being read when it looks best

Customer status matters as much as the version. A mailing to lapsed customers, one to active customers and one to prospects who have never bought behave differently, and the guide to new and returning customers explains how to define status from the business's own records rather than from analytics, which cannot know who bought by post or by phone.

Who may be mailed at all, how objections are handled and what goes on a Suppression List of people who asked not to be contacted are privacy questions, not measurement ones. The privacy and consent guide and the UK Information Commissioner's Office's direct marketing guidance, listed in the sources, cover that ground; this guide does not restate them and nothing here is legal advice. For measurement, the point is only that those people are removed before the holdout is drawn.

Response routes: codes, tagged addresses and asking

Response routes connect an order to the mailing that prompted it. Each is worth having and each misses people.

Codes. A code printed on one version of one mailing catches buyers on any device and by any route, including the phone, if order-takers are asked to record it. It misses buyers who forget it, and a generous offer changes what is measured, because some of the response is to the discount rather than the mailing. The offline media guide's warning about codes travelling to deal sites applies to mail too.

Tagged addresses. A short, memorable address on the business's own domain for each version, redirecting to the landing page with campaign parameters attached, makes the visits that use it arrive tagged. Google's URL builder documentation, listed in the sources, describes the parameters; the UTM and naming conventions guide sets the convention so that "mail" is a medium like any other and not a new spelling every season. Test each redirect before the drop, as the offline media flight launch checklist does for print, because a redirect that drops its parameters reports the visits as direct. Never put a customer number, name or address into the link: the guide to keeping personal data out of analytics treats identifiers in web addresses as a leak.

Asking. A question at checkout or on the phone, "where did you hear about us" or "what prompted this order", with the catalogue as an answer, catches people who used neither code nor address. The self-reported attribution guide covers wording, coding and the biases to read around.

Read every route as a floor. They are best for comparing versions sent to comparable people at the same time, where the leaks are similar on both sides, and poor for judging the mailing as a whole. When the choice is between two pieces for the same segment, such as a catalogue and a cheaper postcard, the direct mail format test sets out a design that mails each piece to a random part of the list beside a held-back group and reads every arm from the order records rather than its codes.

Matchback and why it overstates

Matchback underlies most mailing reports. Take everyone who bought during a period after the drop, match them to the mailed list by customer record, name or address, and credit the mailing with every match. It is a form of Attribution, and like click-based attribution it describes who was touched before buying, not what the touch changed.

It overstates for reasons that compound:

  • Selection. Lists are chosen for likelihood to buy. Recent and frequent customers are mailed first because they respond best, and they are also the customers most likely to buy without a catalogue.
  • The usual purchases. Every mailed customer's ordinary purchases in the window are credited, including orders placed before the piece could have arrived if the window is drawn loosely.
  • Other channels. The same customers receive email, see paid search and visit the site. Matchback credits the mailing with orders that email or search also claim, so the channel totals add up to more than the sales.
  • No counterfactual. Nothing in a matchback says what the same people would have bought unmailed.

Matchback still earns its place. It counts how many listed customers bought, it is an upper limit on the mailing's effect, and its gap from the holdout result is worth reporting, because that gap is the amount the business has been over-crediting the channel in every earlier report. What it cannot do is decide whether a segment is worth mailing.

A holdout drawn from the mailing list

The holdout turns the mailing into a controlled comparison. The steps follow the user-level design in the incrementality testing guide:

  • Finish the list first. Apply the selection rules and remove everyone who may not be mailed. The holdout is drawn only from people who could have been mailed and were chosen to be.
  • Draw at random. Use a random number per customer, not the last digit of an account number that might correlate with age of account, and not "the people the printer could not fit in". Within each segment, so that every segment has its own comparison.
  • Size it from the business's own volumes. How large the holdout must be depends on how often the segment buys unmailed, how much that varies and how small a difference would still change the decision. The incrementality guide's section on cells, size and duration explains the calculation; do it before the drop and accept a larger holdout for segments with few buyers.
  • Keep everything else equal. Held-out customers receive the same emails, see the same advertising and can buy the same products. Only the mailing differs.
  • Flag it in the records. The holdout flag sits with each customer in the CRM, so the comparison can be run from the order system without anyone rebuilding the list.

Evan Miller's article on running A/B tests, listed in the sources, explains why a test read repeatedly and stopped when it looks good reports noise; the incrementality testing guide applies the same rule to holdouts. Decide the reading date before the drop and keep to it. The holdout test readiness checklist gathers the draw, the flag and the reading date into one gate to tick before the mailing goes out.

A persistent holdout, never mailed, gives a long view at the cost of every mailing to those customers. Redrawing for each campaign spreads that cost but cannot measure the cumulative effect of years of mailings.

Reading the result over the response window

Mail is read over days and weeks, not in the hour after an airing. Set the response window before the drop, from the business's own history: how long orders from earlier mailings kept arriving before purchases returned to their Baseline. The offline media guide gives the same advice for every offline medium.

At the reading date, compare the two groups on what the business cares about, from its own order records and across every channel:

  • Buyers per person in each group, and the difference.
  • Revenue and gross margin per person, because a mailing can raise order size as well as order count.
  • The incremental total: the difference per person multiplied by the number mailed.
  • The same split by segment and customer status, where the holdout was drawn within each segment.

Put a range around the difference rather than quoting one number. The guide to reading small numbers explains why differences built on modest counts move a long way by chance, and a segment whose range runs from a loss to a gain has not shown anything yet. Write the result up with the experiment results report template, including the matchback figure beside the incremental one so the gap is visible.

For customers mailed again and again, read both groups for a while after the window closes too: if mailed customers only bought earlier rather than more, the gap narrows.

Costing a mailing against its incremental margin

The return on a mailing is the incremental margin it added minus what it cost. Costs come from the business's own invoices for design, print, postage, list rental and handling; margin comes from its own accounts, not from revenue, because a mailing that adds revenue at a thin margin can lose money. Incremental margin here is the mailed group's margin per customer less the held-back group's, before any of the mailing's costs. The break-even calculator turns those into the incremental orders or margin a mailing must produce to pay for itself.

The break-even calculator walk-through for direct mail costs one segment's share of the bill and reads the result as the incremental orders and order-rate lift the held-back comparison must show.

Do the costing per segment. A mailing usually pays for itself in some segments and not in others, and the decision is which segments to mail next time, not whether mail "works". Lapsed customers brought back by a mailing may buy again later, so their value can extend beyond the window; the lifecycle marketing guide treats reactivation as a stage with its own measures, and a later read of repeat purchases in both groups shows whether that extra value is real. Mailings to named business accounts, which the account-based marketing guide describes, are usually too few to read alone; judge them on pipeline over a longer period, with the uncertainty stated.

Inserts in parcels go to customers who have already bought and have no list of non-recipients in the same sense; the packaging and order inserts guide covers testing one insert against another. The attribution hub sets out the other methods in this section.

A seed company tests its spring catalogue (illustrative)

Illustrative example — not a real result

Pemberqua Seeds holds back part of its catalogue list

Pemberqua Seeds, an invented mail-order seed company, mails its spring catalogue to 24,000 lapsed customers in 2 segments and holds back a random 10% of each. The response window is 10 weeks, set from the previous 3 springs.

SegmentMailedHeld backMailed who orderedHeld back who orderedIncremental orders
Lapsed 1 to 2 seasons9,0001,0001,350 (15%)90 (9%)about 540
Lapsed 3 or more seasons12,6001,400819 (6.5%)77 (5.5%)about 126, range from roughly 0 to 300
Total21,6002,4002,169167about 666

Matchback credits the catalogue with 2,169 orders, more than 3 times the incremental estimate. At a gross margin of £21 an order, matchback values the catalogue at about £45,500 against a print and postage bill of £20,520.

On the incremental figures, the first segment adds about £11,300 of margin against £8,550 of cost, and the second adds about £2,600 against £11,970, with a range that includes no effect at all. The autumn catalogue goes only to customers lapsed 1 to 2 seasons, with a fresh 10% holdout, and the older segment gets a cheaper postcard tested the same way.

Common mistakes

  • Reporting matchback as the result. Every listed buyer is credited, including those who would have bought anyway.
  • Holding back the wrong people. A holdout of customers who could not be mailed, or of a different segment, is not a comparison.
  • Mixing suppression and holdout. People who may not be mailed are removed first; the holdout comes from the rest.
  • Counting only coded orders. Codes and tagged addresses are floors; the holdout comparison uses every order.
  • Identifiers in web addresses. Customer numbers in a link send personal data to analytics.
  • Reading early. A result checked weekly and announced when it looks good is noise with a headline.
  • Costing on revenue. Margin decides whether a mailing paid; revenue flatters thin-margin offers.
  • One verdict for the whole list. Segments differ, and the decision is which ones to mail next time.

Frequently asked questions

What is matchback and why is it not enough?

Matchback takes everyone who bought during a period after a mailing, matches them against the list of people who were mailed and credits the mailing with every match. It is easy and it always looks good, because the people chosen for a mailing are usually the ones most likely to buy anyway, and matchback cannot tell their usual purchases from the ones the mail caused. It is a useful count of who responded and an upper limit on what the mailing did; the incremental effect comes from comparing mailed customers with a random group from the same list who were not mailed.

Do we lose sales by holding part of the list back?

Some, if the mailing works, and that is the price of knowing whether it does. The lost sales are the mailing's incremental effect on the held-back group, which is exactly the number the test measures. Without the holdout, the business keeps paying for every piece on the strength of matchback, including the segments where the mail changed nothing. The holdout's size can be set from the business's own volumes so that it is no larger than the decision needs.

Can we put customer numbers in personalised web addresses to track responses?

Not into anything analytics collects. A web address that carries a customer number or a name sends personal data into analytics with every visit, which the site's guide on personal data in analytics treats as a leak to stop at source. Use a short address per mailing version that redirects to a page with campaign parameters, and connect responses to customers in the business's own order and CRM records, where customer identifiers belong.

Terms used on this page

Tools to try

Frameworks

Where this page is used

Also related

Sources

  1. [GA4] URL builders: Collect campaign data with custom URLs — Google Analytics Help
  2. How Not To Run an A/B Test — Evan Miller
  3. Direct marketing guidance — Information Commissioner's Office (UK)

Related