Why addressed mail can be measured differently
The guide to measuring TV, radio, podcasts and print works up a ladder of evidence for media whose audience is unknown: response mechanisms that count a floor, response around airings, search lift and regional tests. Each rung exists because nobody can split the audience into people who saw the advert and people who did not. With addressed mail that split is in the business's hands. The mailing file names every recipient, and the order system or CRM records what every customer on the list bought, by any route, before and after the drop.
That knowledge supports the design the incrementality testing guide calls the user-level holdout, with a randomised list of people withheld, which it describes as the cleanest design when it is possible and names for owned channels. A mailing to existing customers or prospects is one of the few places outside email where it is fully possible: the business decides who receives each piece, so it can decide at random who does not.
The same knowledge produces the commonest mistake. Because every buyer can be looked up on the list, it is easy to count every listed buyer as a response, and that count is always flattering. The rest of this guide builds the measurement around the holdout and treats the other numbers as supporting evidence.
| Evidence | What it shows | Where it sits |
|---|---|---|
| Random holdout from the mailing list | What the mailing changed in total purchases | Top: a controlled comparison |
| Codes, tagged addresses and asking | Who used a route that names the mailing | Floors, useful for comparing versions |
| Matchback | Every listed customer who bought | An upper limit, not an effect |
What to record for every mailing
Measurement usually fails because the facts of a mailing were never written down. Record these before the drop, in the campaign record the guide to campaign records in the CRM says a direct mail drop deserves, named with the same convention as the rest of the business's campaigns:
| Field | Why it matters |
|---|---|
| Selection rules and the list source | Defines who could have been mailed, and so who the result applies to |
| People removed before the draw, and why | Separates people who may not be mailed from the holdout, which is a different group |
| The holdout flag for every person on the final list | The only way to compare mailed and unmailed customers later |
| Version, offer and segment for every mailed person | Lets codes, addresses and outcomes be read per version |
| Drop date and the expected arrival period | Starts the response window |
| Codes, short addresses and phone routes per version | Connects responses to the version that carried them |
| Costs from the business's own invoices | Feeds the costing at the end |
| The response window and the reading date, decided now | Stops the result being read when it looks best |
Customer status matters as much as the version. A mailing to lapsed customers, one to active customers and one to prospects who have never bought behave differently, and the guide to new and returning customers explains how to define status from the business's own records rather than from analytics, which cannot know who bought by post or by phone.
Who may be mailed at all, how objections are handled and what goes on a Suppression ListA list of addresses or contacts excluded from every send, such as unsubscribes, hard bounces and spam complaints. of people who asked not to be contacted are privacy questions, not measurement ones. The privacy and consent guide and the UK Information Commissioner's Office's direct marketing guidance, listed in the sources, cover that ground; this guide does not restate them and nothing here is legal advice. For measurement, the point is only that those people are removed before the holdout is drawn.
Response routes: codes, tagged addresses and asking
Response routes connect an order to the mailing that prompted it. Each is worth having and each misses people.
Codes. A codeA code a customer enters at checkout to receive an offer, also used to identify sales from a creator, partner or channel that links cannot capture. printed on one version of one mailing catches buyers on any device and by any route, including the phone, if order-takers are asked to record it. It misses buyers who forget it, and a generous offer changes what is measured, because some of the response is to the discount rather than the mailing. The offline media guide's warning about codes travelling to deal sites applies to mail too.
Tagged addresses. A short, memorable address on the business's own domain for each version, redirecting to the landing page with campaign parameters attached, makes the visits that use it arrive tagged. Google's URL builder documentation, listed in the sources, describes the parameters; the UTM and naming conventions guide sets the convention so that "mail" is a medium like any other and not a new spelling every season. Test each redirect before the drop, as the offline media flight launch checklist does for print, because a redirect that drops its parameters reports the visits as direct. Never put a customer number, name or address into the link: the guide to keeping personal data out of analytics treats identifiers in web addresses as a leak.
Asking. A question at checkout or on the phone, "where did you hear about us" or "what prompted this order", with the catalogue as an answer, catches people who used neither code nor address. The self-reported attribution guide covers wording, coding and the biases to read around.
Read every route as a floor. They are best for comparing versions sent to comparable people at the same time, where the leaks are similar on both sides, and poor for judging the mailing as a whole. When the choice is between two pieces for the same segment, such as a catalogue and a cheaper postcard, the direct mail format test sets out a design that mails each piece to a random part of the list beside a held-back group and reads every arm from the order records rather than its codes.
Matchback and why it overstates
Matchback underlies most mailing reports. Take everyone who bought during a period after the drop, match them to the mailed list by customer record, name or address, and credit the mailing with every match. It is a form of AttributionAssigning conversion credit to touchpoints., and like click-based attribution it describes who was touched before buying, not what the touch changed.
It overstates for reasons that compound:
- Selection. Lists are chosen for likelihood to buy. Recent and frequent customers are mailed first because they respond best, and they are also the customers most likely to buy without a catalogue.
- The usual purchases. Every mailed customer's ordinary purchases in the window are credited, including orders placed before the piece could have arrived if the window is drawn loosely.
- Other channels. The same customers receive email, see paid search and visit the site. Matchback credits the mailing with orders that email or search also claim, so the channel totals add up to more than the sales.
- No counterfactual. Nothing in a matchback says what the same people would have bought unmailed.
Matchback still earns its place. It counts how many listed customers bought, it is an upper limit on the mailing's effect, and its gap from the holdout result is worth reporting, because that gap is the amount the business has been over-crediting the channel in every earlier report. What it cannot do is decide whether a segment is worth mailing.
A holdout drawn from the mailing list
The holdout turns the mailing into a controlled comparison. The steps follow the user-level design in the incrementality testing guide:
- Finish the list first. Apply the selection rules and remove everyone who may not be mailed. The holdout is drawn only from people who could have been mailed and were chosen to be.
- Draw at random. Use a random number per customer, not the last digit of an account number that might correlate with age of account, and not "the people the printer could not fit in". Within each segment, so that every segment has its own comparison.
- Size it from the business's own volumes. How large the holdout must be depends on how often the segment buys unmailed, how much that varies and how small a difference would still change the decision. The incrementality guide's section on cells, size and duration explains the calculation; do it before the drop and accept a larger holdout for segments with few buyers.
- Keep everything else equal. Held-out customers receive the same emails, see the same advertising and can buy the same products. Only the mailing differs.
- Flag it in the records. The holdout flag sits with each customer in the CRM, so the comparison can be run from the order system without anyone rebuilding the list.
Evan Miller's article on running A/B tests, listed in the sources, explains why a test read repeatedly and stopped when it looks good reports noise; the incrementality testing guide applies the same rule to holdouts. Decide the reading date before the drop and keep to it. The holdout test readiness checklist gathers the draw, the flag and the reading date into one gate to tick before the mailing goes out.
A persistent holdout, never mailed, gives a long view at the cost of every mailing to those customers. Redrawing for each campaign spreads that cost but cannot measure the cumulative effect of years of mailings.
Reading the result over the response window
Mail is read over days and weeks, not in the hour after an airing. Set the response window before the drop, from the business's own history: how long orders from earlier mailings kept arriving before purchases returned to their BaselineAn estimate of what a metric would have done without a given change, campaign or promotion, against which the effect is measured.. The offline media guide gives the same advice for every offline medium.
At the reading date, compare the two groups on what the business cares about, from its own order records and across every channel:
- Buyers per person in each group, and the difference.
- Revenue and gross margin per person, because a mailing can raise order size as well as order count.
- The incremental total: the difference per person multiplied by the number mailed.
- The same split by segment and customer status, where the holdout was drawn within each segment.
Put a range around the difference rather than quoting one number. The guide to reading small numbers explains why differences built on modest counts move a long way by chance, and a segment whose range runs from a loss to a gain has not shown anything yet. Write the result up with the experiment results report template, including the matchback figure beside the incremental one so the gap is visible.
For customers mailed again and again, read both groups for a while after the window closes too: if mailed customers only bought earlier rather than more, the gap narrows.
Costing a mailing against its incremental margin
The return on a mailing is the incremental margin it added minus what it cost. Costs come from the business's own invoices for design, print, postage, list rental and handling; margin comes from its own accounts, not from revenue, because a mailing that adds revenue at a thin margin can lose money. Incremental marginThe margin an activity added: margin per customer assigned to it minus margin per comparable customer held back, before the activity's own costs are taken off. here is the mailed group's margin per customer less the held-back group's, before any of the mailing's costs. The break-even calculator turns those into the incremental orders or margin a mailing must produce to pay for itself.
The break-even calculator walk-through for direct mail costs one segment's share of the bill and reads the result as the incremental orders and order-rate lift the held-back comparison must show.
Do the costing per segment. A mailing usually pays for itself in some segments and not in others, and the decision is which segments to mail next time, not whether mail "works". Lapsed customers brought back by a mailing may buy again later, so their value can extend beyond the window; the lifecycle marketing guide treats reactivation as a stage with its own measures, and a later read of repeat purchases in both groups shows whether that extra value is real. Mailings to named business accounts, which the account-based marketing guide describes, are usually too few to read alone; judge them on pipeline over a longer period, with the uncertainty stated.
Inserts in parcels go to customers who have already bought and have no list of non-recipients in the same sense; the packaging and order inserts guide covers testing one insert against another. The attribution hub sets out the other methods in this section.
A seed company tests its spring catalogue (illustrative)
Pemberqua Seeds holds back part of its catalogue list
Pemberqua Seeds, an invented mail-order seed company, mails its spring catalogue to 24,000 lapsed customers in 2 segments and holds back a random 10% of each. The response window is 10 weeks, set from the previous 3 springs.
| Segment | Mailed | Held back | Mailed who ordered | Held back who ordered | Incremental orders |
|---|---|---|---|---|---|
| Lapsed 1 to 2 seasons | 9,000 | 1,000 | 1,350 (15%) | 90 (9%) | about 540 |
| Lapsed 3 or more seasons | 12,600 | 1,400 | 819 (6.5%) | 77 (5.5%) | about 126, range from roughly 0 to 300 |
| Total | 21,600 | 2,400 | 2,169 | 167 | about 666 |
Matchback credits the catalogue with 2,169 orders, more than 3 times the incremental estimate. At a gross margin of £21 an order, matchback values the catalogue at about £45,500 against a print and postage bill of £20,520.
On the incremental figures, the first segment adds about £11,300 of margin against £8,550 of cost, and the second adds about £2,600 against £11,970, with a range that includes no effect at all. The autumn catalogue goes only to customers lapsed 1 to 2 seasons, with a fresh 10% holdout, and the older segment gets a cheaper postcard tested the same way.
Common mistakes
- Reporting matchback as the result. Every listed buyer is credited, including those who would have bought anyway.
- Holding back the wrong people. A holdout of customers who could not be mailed, or of a different segment, is not a comparison.
- Mixing suppression and holdout. People who may not be mailed are removed first; the holdout comes from the rest.
- Counting only coded orders. Codes and tagged addresses are floors; the holdout comparison uses every order.
- Identifiers in web addresses. Customer numbers in a link send personal data to analytics.
- Reading early. A result checked weekly and announced when it looks good is noise with a headline.
- Costing on revenue. Margin decides whether a mailing paid; revenue flatters thin-margin offers.
- One verdict for the whole list. Segments differ, and the decision is which ones to mail next time.
