Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Experiment Library

Direct Mail Format Test

An experiment design for choosing which piece to mail a segment: customers split between a catalogue, a postcard and no mail, read on net incremental margin.
Experimentintermediate10 min readUpdated

Hypothesis

If customers in one segment of the final mailing list are split at random between a catalogue, a postcard and no mailing, the postcard will bring at least as much incremental gross margin per listed customer after the cost of each piece as the catalogue, because for that segment the reminder does more of the work than the range shown, unless the catalogue's fuller range adds orders the postcard cannot. The decision behind it is concrete: mail the segment the catalogue, mail it the postcard, or stop mailing it, and keep whichever leaves the most margin once the piece is paid for.

The guide to measuring direct mail sets out how to judge one mailing: finish the list, hold back a random part within each segment, read the order records over a window fixed before the drop and cost the result on incremental margin. It notes that a mailing usually pays in some segments and not others, and that codes and short addresses are best for comparing versions sent to comparable people at the same time. What it leaves to a sentence is the next question a team asks once a segment looks marginal: would a cheaper piece do the same job? That is a comparison between formats, and it needs both pieces and a held-back group drawn from the same list on the same day. Write the arms, the expected result and the limits on each guardrail into the experiment hypothesis template before the list goes to the printer.

Metrics and guardrails

The primary metric is net incremental margin per listed customer. For each arm, take the Gross Margin on every order placed by its customers during the response window, by post, phone, website or shop, from the business's own order records, and divide by the number of customers assigned to the arm. Subtract the held-back group's figure, which is what the segment buys unmailed. Then subtract the cost per person of the piece itself: design, print, postage and handling, from the business's own invoices. What remains is the margin each piece added after paying for itself, and it is the number the decision rests on.

Two features make the metric honest. The denominator is everyone assigned, so customers who never noticed the piece count against it, as they would in a real mailing. And the reading comes from the order records, not from codes, so buyers who ordered by phone or typed the address without the code are counted in the right arm. That is the measurement of Incrementality the direct mail guide puts at the top of its evidence table, extended from one piece to two.

Two guardrails protect against a piece that wins for the wrong reason:

  • Purchases per person in a follow-on window. The direct mail guide warns that mailed customers may only buy earlier rather than more, which narrows the gap once the window closes. Read the same purchases in a second window after the response window, for every arm, so a piece that pulled orders forward is caught.
  • Requests to stop receiving post per person mailed. A piece that prompts more customers to ask not to be mailed again is costing future mailings. Record each request against the customer's arm and add it to the Suppression List as usual.

Design

Randomise at customer level, within the segment, before the drop. Build the final list first: apply the selection rules, then remove everyone who may not be mailed. Only then assign each remaining customer at random to the catalogue, the postcard or the held-back group, using a random number per customer rather than anything that might correlate with the customer's history, and flag the arm on the customer record in the CRM. The guide to campaign records in the CRM explains why a mail drop deserves a record of its own, and the arm flag belongs in it.

Design elementChoiceReason
Unit of randomisationCustomer on the final list, assigned within one segmentSegments behave differently; the comparison must be like for like
ArmsCatalogue; postcard; held backThe held-back group turns two piece counts into two incremental effects
Held constantOffer, code structure, drop date, emails, advertisingOnly the piece differs, so only the piece can explain the difference
Codes and addressesOne code and one short address per pieceLets routes be compared between the mailed arms, as a floor
Primary metricNet incremental margin per listed customer, from order recordsCounts every route and the full cost of each piece
GuardrailsFollow-on purchases; requests to stop postCatches orders brought forward and damage to future mailings

Both pieces carry the same offer, or none, and the same kind of Discount Code differing only in the code itself. Give each piece its own short address on the business's own domain, redirecting to the landing page with campaign parameters attached, as Google's URL builder documentation describes and the UTM and naming conventions guide standardises. Never put a customer number in that address: the guide to keeping personal data out of analytics treats identifiers in web addresses as a leak. Held-back customers receive exactly the emails, advertising and offers they would otherwise have received.

Who may be mailed at all is a privacy question, not a measurement one. The direct mail guide leaves it to the UK Information Commissioner's Office's direct marketing guidance, and the point for this design is only that people who have objected or may not be contacted are removed before the draw, never placed in the held-back group. This is general practice, not legal advice.

Sample size reasoning

The comparison that sets the size is the one between the pieces, not between either piece and no mail. Both pieces add something over the held-back group, and the gap between them is usually smaller than that, so it needs more customers to separate from chance. Three arms also divide one segment's list three ways. The inputs are how often the segment buys unmailed, how much margin per person varies, the cost difference between the pieces and the smallest difference in net margin that would make the team switch. The section on cells, size and duration in the incrementality testing guide explains how those combine for a holdout, and the break-even calculator turns each piece's cost into the incremental margin it must produce to pay for itself.

The direct mail guide to the break-even calculator shows why a cheaper piece needs a smaller lift over the held-back group: the lift each piece needs is its cost divided by the margin per order.

If the segment cannot support three arms at a size worth reading, change the question before the drop. Test the cheaper piece against no mail first, which needs fewer customers because its effect is larger, and keep the catalogue for a later season. Or combine adjacent segments only if they were mailed the same way before and the team accepts a single answer for both.

Duration

Drop both pieces on the same day, through the same printer and postal service where possible, so they arrive in the same week. Set the response window before the drop from the business's own history: how long orders from earlier mailings kept arriving before purchases returned to normal. Then set a follow-on window by the same rule for the guardrail. Write both dates into the plan. The direct mail guide cites Evan Miller's article on running A/B tests to explain why a result read repeatedly and stopped when it looks good reports noise, and the same applies here: the reading happens once, on the date set in advance.

Reading the results and the decision rule

Read the test once, at the end of the response window. Compare each piece's net incremental margin per listed customer with the held-back group and with each other, with the interval around each difference rather than a single figure; the guide to reading small numbers explains why differences built on modest counts move a long way by chance. Check the follow-on window and the requests to stop post next. Only then read codes and short addresses, as a floor that shows which routes each piece's buyers used. Write the decision rule into the plan before the drop:

ResultDecision
The postcard's net margin is reliably no lower than the catalogue's, within the margin set before the drop, and its follow-on window shows no larger fall-backMail the postcard to this segment
The catalogue earns reliably more net margin after its extra costKeep the catalogue for this segment
Neither piece beats the held-back group after its costStop mailing this segment, or try a different offer in a separate test
A piece wins but its follow-on purchases fall below the held-back group'sTreat the win as orders brought forward and keep the cheaper option

The hierarchy of evidence is the reason for this order: the randomised comparison decides, and the codes, addresses and any matchback count only as supporting evidence. Record the result, including the matchback figure beside the incremental one, in the experiment results report template.

Pitfalls

Suppressions applied after the draw are the first trap. If people who may not be mailed are removed from the mailed arms but stay in the held-back group, the groups are no longer comparable; the holdout test readiness checklist puts the final list before the draw as a hard stop. The second is uneven delivery. A postcard sent second class and a catalogue sent first class, or two pieces dropped a week apart, measure the timing as much as the format. The third is a better offer on one piece, which turns a format test into an offer test.

Reading on codes alone is the fourth. Codes miss buyers who forget them or order by phone, and they miss more for a piece that is kept on a shelf than for one that is used at once, so a code count can favour the wrong piece. The order records decide. Finally, the general testing traps apply: check that each arm received the share of the list the plan intended, because a broken assignment rule produces a sample ratio mismatch, and resist reading early. The guide to common testing pitfalls covers both. For lapsed customers who might need a discount rather than a reminder, the win-back offer test is the design; for the size of an offer, the discount depth test is.

Example: an invented mail-order tea merchant tests a postcard against its catalogue (illustrative)

Illustrative example — not a real result

Frempadory Tea tests a postcard against its autumn catalogue

Frempadory Tea, an invented mail-order tea merchant, mails an autumn catalogue every year. Its customers who last ordered one to two years ago look marginal, so this year it tests that segment. After suppressions the segment holds 12,000 customers, split at random into 5,000 for the catalogue, 5,000 for a postcard showing four best sellers, and 2,000 held back. Both pieces carry the same free-delivery code structure and drop on the same day. The catalogue costs £1.10 per person printed and posted, the postcard £0.40. The response window is eight weeks, set from the previous three autumns, with an eight-week follow-on window after it.

ArmCustomersBuyers in windowGross margin per customerAbove held backCost per customerNet incremental margin
Catalogue5,000460£2.58£1.15£1.10£0.05
Postcard5,000410£2.13£0.70£0.40£0.30
Held back2,000110£1.43———

The catalogue brings more buyers and larger orders, but its cost absorbs almost all of what it adds. The postcard leads on net incremental margin by about £0.26 per customer, with an interval of roughly £0.02 to £0.50. In the follow-on window both mailed arms buy at about the same rate as the held-back group, and requests to stop post are 12 for the catalogue and 9 for the postcard. Codes caught only 190 of the catalogue arm's 460 buyers. Frempadory mails this segment a postcard next autumn, with a fresh held-back group, and plans a separate test of the catalogue for its active customers.

Common mistakes

Format tests fail more often on their list and their reading than on their statistics. The experiment library lists the other designs, including the offer tests that sit beside this one.

  • Leaving out the held-back group. Two piece counts say which piece sold more, not which one paid for itself.
  • Drawing before the list is final. Suppressed customers end up in one group and the comparison breaks.
  • Changing the offer with the format. The test then measures generosity, not the piece.
  • Reading on codes. Codes are a floor that leaks unevenly between pieces; the order records decide.
  • Ignoring the cost of the piece. A catalogue that sells more can still leave less margin than a postcard.
  • Applying one segment's answer to every segment. Lapsed and active customers respond differently; test each segment the decision covers.

Frequently asked questions

Why include a no-mail arm when the question is which piece to send?

Because without it the test can only say which piece produced more orders, not which one paid for itself. Many customers in any segment buy without being mailed, and both pieces are credited with those orders. The held-back group shows what the segment buys anyway, so each piece can be judged on what it added, after its own cost. It also answers a question the team may not have asked: whether the segment is worth mailing at all.

Can the two pieces carry different offers?

Not in this test. If the postcard carries a stronger offer than the catalogue, the result measures the offer as well as the format, and nobody can say which did the work. Give both pieces the same offer and the same kind of code, differing only in the code itself so each piece can be traced. If the offer is in question too, settle the format first and test the offer afterwards with a design built for offers.

Is it acceptable to mail some customers and not others on purpose?

Holding back part of a list is the standard way to learn what a mailing adds, and the held-back customers keep every other message, offer and route to buy. The questions that matter are the ones about who may be mailed at all, which are privacy questions rather than measurement ones: people who have objected or who may not be contacted are removed before anyone is assigned. This is general direction, not legal advice; take advice on the rules that apply where the business mails.

Terms used on this page

Tools to try

Frameworks

Where this page is used

Also related

Sources

  1. Direct marketing guidance — Information Commissioner's Office (UK)
  2. [GA4] URL builders: Collect campaign data with custom URLs — Google Analytics Help
  3. How Not To Run an A/B Test — Evan Miller

Related