Hypothesis
If customers in one segment of the final mailing list are split at random between a catalogue, a postcard and no mailing, the postcard will bring at least as much incremental gross margin per listed customer after the cost of each piece as the catalogue, because for that segment the reminder does more of the work than the range shown, unless the catalogue's fuller range adds orders the postcard cannot. The decision behind it is concrete: mail the segment the catalogue, mail it the postcard, or stop mailing it, and keep whichever leaves the most margin once the piece is paid for.
The guide to measuring direct mail sets out how to judge one mailing: finish the list, hold back a random part within each segment, read the order records over a window fixed before the drop and cost the result on incremental margin. It notes that a mailing usually pays in some segments and not others, and that codes and short addresses are best for comparing versions sent to comparable people at the same time. What it leaves to a sentence is the next question a team asks once a segment looks marginal: would a cheaper piece do the same job? That is a comparison between formats, and it needs both pieces and a held-back group drawn from the same list on the same day. Write the arms, the expected result and the limits on each guardrail into the experiment hypothesis template before the list goes to the printer.
Metrics and guardrails
The primary metric is net incremental margin per listed customer. For each arm, take the Gross MarginShare of revenue left after the direct cost of the goods sold. on every order placed by its customers during the response window, by post, phone, website or shop, from the business's own order records, and divide by the number of customers assigned to the arm. Subtract the held-back group's figure, which is what the segment buys unmailed. Then subtract the cost per person of the piece itself: design, print, postage and handling, from the business's own invoices. What remains is the margin each piece added after paying for itself, and it is the number the decision rests on.
Two features make the metric honest. The denominator is everyone assigned, so customers who never noticed the piece count against it, as they would in a real mailing. And the reading comes from the order records, not from codes, so buyers who ordered by phone or typed the address without the code are counted in the right arm. That is the measurement of IncrementalityConversions caused by marketing. the direct mail guide puts at the top of its evidence table, extended from one piece to two.
Two guardrails protect against a piece that wins for the wrong reason:
- Purchases per person in a follow-on window. The direct mail guide warns that mailed customers may only buy earlier rather than more, which narrows the gap once the window closes. Read the same purchases in a second window after the response window, for every arm, so a piece that pulled orders forward is caught.
- Requests to stop receiving post per person mailed. A piece that prompts more customers to ask not to be mailed again is costing future mailings. Record each request against the customer's arm and add it to the Suppression ListA list of addresses or contacts excluded from every send, such as unsubscribes, hard bounces and spam complaints. as usual.
Design
Randomise at customer level, within the segment, before the drop. Build the final list first: apply the selection rules, then remove everyone who may not be mailed. Only then assign each remaining customer at random to the catalogue, the postcard or the held-back group, using a random number per customer rather than anything that might correlate with the customer's history, and flag the arm on the customer record in the CRM. The guide to campaign records in the CRM explains why a mail drop deserves a record of its own, and the arm flag belongs in it.
| Design element | Choice | Reason |
|---|---|---|
| Unit of randomisation | Customer on the final list, assigned within one segment | Segments behave differently; the comparison must be like for like |
| Arms | Catalogue; postcard; held back | The held-back groupA randomly chosen part of an audience kept from exposure to measure what exposure caused. turns two piece counts into two incremental effects |
| Held constant | Offer, code structure, drop date, emails, advertising | Only the piece differs, so only the piece can explain the difference |
| Codes and addresses | One code and one short address per piece | Lets routes be compared between the mailed arms, as a floor |
| Primary metric | Net incremental margin per listed customer, from order records | Counts every route and the full cost of each piece |
| Guardrails | Follow-on purchases; requests to stop post | Catches orders brought forward and damage to future mailings |
Both pieces carry the same offer, or none, and the same kind of Discount CodeA code a customer enters at checkout to receive an offer, also used to identify sales from a creator, partner or channel that links cannot capture. differing only in the code itself. Give each piece its own short address on the business's own domain, redirecting to the landing page with campaign parameters attached, as Google's URL builder documentation describes and the UTM and naming conventions guide standardises. Never put a customer number in that address: the guide to keeping personal data out of analytics treats identifiers in web addresses as a leak. Held-back customers receive exactly the emails, advertising and offers they would otherwise have received.
Who may be mailed at all is a privacy question, not a measurement one. The direct mail guide leaves it to the UK Information Commissioner's Office's direct marketing guidance, and the point for this design is only that people who have objected or may not be contacted are removed before the draw, never placed in the held-back group. This is general practice, not legal advice.
Sample size reasoning
The comparison that sets the size is the one between the pieces, not between either piece and no mail. Both pieces add something over the held-back group, and the gap between them is usually smaller than that, so it needs more customers to separate from chance. Three arms also divide one segment's list three ways. The inputs are how often the segment buys unmailed, how much margin per person varies, the cost difference between the pieces and the smallest difference in net margin that would make the team switch. The section on cells, size and duration in the incrementality testing guide explains how those combine for a holdout, and the break-even calculator turns each piece's cost into the incremental margin it must produce to pay for itself.
The direct mail guide to the break-even calculator shows why a cheaper piece needs a smaller lift over the held-back group: the lift each piece needs is its cost divided by the margin per order.
If the segment cannot support three arms at a size worth reading, change the question before the drop. Test the cheaper piece against no mail first, which needs fewer customers because its effect is larger, and keep the catalogue for a later season. Or combine adjacent segments only if they were mailed the same way before and the team accepts a single answer for both.
Duration
Drop both pieces on the same day, through the same printer and postal service where possible, so they arrive in the same week. Set the response window before the drop from the business's own history: how long orders from earlier mailings kept arriving before purchases returned to normal. Then set a follow-on window by the same rule for the guardrail. Write both dates into the plan. The direct mail guide cites Evan Miller's article on running A/B tests to explain why a result read repeatedly and stopped when it looks good reports noise, and the same applies here: the reading happens once, on the date set in advance.
Reading the results and the decision rule
Read the test once, at the end of the response window. Compare each piece's net incremental margin per listed customer with the held-back group and with each other, with the interval around each difference rather than a single figure; the guide to reading small numbers explains why differences built on modest counts move a long way by chance. Check the follow-on window and the requests to stop post next. Only then read codes and short addresses, as a floor that shows which routes each piece's buyers used. Write the decision rule into the plan before the drop:
| Result | Decision |
|---|---|
| The postcard's net margin is reliably no lower than the catalogue's, within the margin set before the drop, and its follow-on window shows no larger fall-back | Mail the postcard to this segment |
| The catalogue earns reliably more net margin after its extra cost | Keep the catalogue for this segment |
| Neither piece beats the held-back group after its cost | Stop mailing this segment, or try a different offer in a separate test |
| A piece wins but its follow-on purchases fall below the held-back group's | Treat the win as orders brought forward and keep the cheaper option |
The hierarchy of evidence is the reason for this order: the randomised comparison decides, and the codes, addresses and any matchback count only as supporting evidence. Record the result, including the matchback figure beside the incremental one, in the experiment results report template.
Pitfalls
Suppressions applied after the draw are the first trap. If people who may not be mailed are removed from the mailed arms but stay in the held-back group, the groups are no longer comparable; the holdout test readiness checklist puts the final list before the draw as a hard stop. The second is uneven delivery. A postcard sent second class and a catalogue sent first class, or two pieces dropped a week apart, measure the timing as much as the format. The third is a better offer on one piece, which turns a format test into an offer test.
Reading on codes alone is the fourth. Codes miss buyers who forget them or order by phone, and they miss more for a piece that is kept on a shelf than for one that is used at once, so a code count can favour the wrong piece. The order records decide. Finally, the general testing traps apply: check that each arm received the share of the list the plan intended, because a broken assignment rule produces a sample ratio mismatch, and resist reading early. The guide to common testing pitfalls covers both. For lapsed customers who might need a discount rather than a reminder, the win-back offer test is the design; for the size of an offer, the discount depth test is.
Example: an invented mail-order tea merchant tests a postcard against its catalogue (illustrative)
Frempadory Tea tests a postcard against its autumn catalogue
Frempadory Tea, an invented mail-order tea merchant, mails an autumn catalogue every year. Its customers who last ordered one to two years ago look marginal, so this year it tests that segment. After suppressions the segment holds 12,000 customers, split at random into 5,000 for the catalogue, 5,000 for a postcard showing four best sellers, and 2,000 held back. Both pieces carry the same free-delivery code structure and drop on the same day. The catalogue costs £1.10 per person printed and posted, the postcard £0.40. The response window is eight weeks, set from the previous three autumns, with an eight-week follow-on window after it.
| Arm | Customers | Buyers in window | Gross margin per customer | Above held back | Cost per customer | Net incremental margin |
|---|---|---|---|---|---|---|
| Catalogue | 5,000 | 460 | £2.58 | £1.15 | £1.10 | £0.05 |
| Postcard | 5,000 | 410 | £2.13 | £0.70 | £0.40 | £0.30 |
| Held back | 2,000 | 110 | £1.43 | — | — | — |
The catalogue brings more buyers and larger orders, but its cost absorbs almost all of what it adds. The postcard leads on net incremental margin by about £0.26 per customer, with an interval of roughly £0.02 to £0.50. In the follow-on window both mailed arms buy at about the same rate as the held-back group, and requests to stop post are 12 for the catalogue and 9 for the postcard. Codes caught only 190 of the catalogue arm's 460 buyers. Frempadory mails this segment a postcard next autumn, with a fresh held-back group, and plans a separate test of the catalogue for its active customers.
Common mistakes
Format tests fail more often on their list and their reading than on their statistics. The experiment library lists the other designs, including the offer tests that sit beside this one.
- Leaving out the held-back group. Two piece counts say which piece sold more, not which one paid for itself.
- Drawing before the list is final. Suppressed customers end up in one group and the comparison breaks.
- Changing the offer with the format. The test then measures generosity, not the piece.
- Reading on codes. Codes are a floor that leaks unevenly between pieces; the order records decide.
- Ignoring the cost of the piece. A catalogue that sells more can still leave less margin than a postcard.
- Applying one segment's answer to every segment. Lapsed and active customers respond differently; test each segment the decision covers.
