Hypothesis
If lapsed customers receive a win-back message, more of them will buy again than a held-out group, and an added offer will be worth its cost only if it reactivates enough customers who would not have returned for the message alone. The decision behind it: send lapsed customers a win-back offer, a message without an offer, or nothing.
The hypothesis has two parts because a win-back programme makes two separate claims. The first is that getting in touch matters at all: that a reminder brings back customers who would otherwise have drifted away. The second is that the discount matters on top of the reminder. Many programmes never test the second claim, and discount every lapsed customer by default, which trains customers to wait for the offer and gives margin to people who would have returned anyway. The repeat purchase and loyalty guide explains why the Repeat Purchase RateShare of customers who placed more than one order in the period. and the natural repurchase interval should shape the lapse definition. Write the lapse definition, the offer and the arms into the experiment hypothesis template before anyone is assigned. For an app, where some lapsed users can be reached by push, some only by email or SMS and some only through paid media, the guide to re-engaging lapsed app users sets the lapse from the app's own return curves and splits the holdout by channel. The tutorial on setting an app's lapse threshold builds those return curves from event data.
Metrics and guardrails
The primary metric is incremental gross profit per lapsed customer: gross profit from each arm's purchases within the window, after discounts, divided by the number of customers assigned to the arm, minus the same figure for the holdout. The denominator is everyone assigned, so customers who never open the message still count. Profit rather than revenue is essential here, because the offer arm's revenue includes the discount it gave away. The email metrics and testing guide explains why opens and clicks are poor proxies for a purchase outcome; they can be reported as context only.
The guardrails catch costs the primary metric spreads thin. Discount cost per reactivated customer shows how much margin each returning customer took. Unsubscribe and complaint rates, per assigned customer, show how lapsed customers react to being contacted again; lapsed lists often contain old addresses and fading consent, and complaints affect the sender's reputation for every list. Every message must carry a working opt-out, as the FTC's CAN-SPAM compliance guide sets out for commercial email in the United States, and the ICO's direct marketing guidance gives the UK direction on contacting customers. The third guardrail, a second order after reactivation, separates customers who came back from customers who came back once for the discount.
Design
Assign each customer once, at the moment they first meet the lapse definition, to the offer arm, the message-only arm or the holdout. Enrolment then runs continuously as customers lapse, which spreads the test across weeks and seasons. The offer and message-only arms receive the same sequence with the same subject lines, timing and creative; the only difference is the offer. The holdout receives no win-back contact but stays eligible for the business's ordinary campaigns, exactly as the other arms are.
| Design element | Choice | Reason |
|---|---|---|
| Unit of randomisation | Lapsed customer, assigned once on meeting the definition | Each customer lapses once per test |
| Arms | Offer, message only, holdout with no win-back contact | Separates the value of contact from the value of the discount |
| Content | Identical sequence; the offer is the only difference | Isolates the discount |
| Other campaigns | Same eligibility in all three arms | Prevents the holdout being contacted differently |
| Primary metric | Incremental gross profit per assigned customer | Counts discount cost and natural returns |
| Second orders | Read in a further window after the first | Separates loyalty from one-off bargain hunting |
The e-commerce email and SMS flows guide covers how to build the win-back sequence itself; this test takes a built sequence and decides whether it and its offer earn their place. The abandoned cart discount test uses a similar logic for an incentive at the cart, where intent is fresh rather than faded.
Sample size reasoning
Reactivation is rare, so each arm needs many lapsed customers to be separated from the holdout, and three arms divide the list further. Two comparisons matter: each message arm against the holdout, and the offer arm against the message-only arm. The second is usually the harder one to detect, because both arms benefit from the contact. The incrementality testing guide explains how to size a holdout and why a small one leaves the incremental figure uncertain, and the sample size and significance guide covers how splitting into several arms raises the number needed per arm. If the lapsed population is small, drop to two arms, message against holdout, and test the offer later.
Duration
Enrol until the planned sample is reached, then follow the last customers enrolled for a window matched to the usual repurchase interval, plus a further window to read second orders. A lapse definition based on the repurchase interval means enrolment itself takes weeks; plan the calendar from the first enrolment to the final second-order read before launch, and avoid starting just before a sale period that would bring lapsed customers back in every arm at once.
Reading the results and the decision rule
Read once, after the second-order window closes for the last enrolled customer. First compare each message arm with the holdout on incremental gross profit per customer, with confidence intervals. Then compare the offer arm with the message-only arm. Then check the guardrails. The rule, written before launch:
| Result | Decision |
|---|---|
| Offer arm adds profit over the message-only arm, and guardrails hold | Send the offer |
| Message-only arm beats the holdout, but the offer adds nothing over it | Send the message without the offer |
| Neither message arm beats the holdout | Stop win-back contact and invest earlier in the lifecycle |
| Complaints or unsubscribes break a limit | Pause contact to the affected segment and review the lapse definition |
Reading Churn RateShare of customers at the start of a period who left during it. and lapse together, as the customer retention strategy guide does, keeps the result in proportion: win-back recovers some lost customers, but keeping them is cheaper. When neither message arm beats the holdout, the diagnosis for win-back campaigns that bring few customers back helps find out why, from the reason the messages give to the timing, the exit and the channels used. The academy exercise on auditing retention and testing win-back offers for a baking supplies shop practises writing this test's lapse definition, arms, primary metric, guardrails, windows and decision rule before launch, after scoring the store's retention system with an audit.
An outdoor clothing retailer prices its win-back discount
Duskmoor, an invented outdoor clothing retailer, treats a customer as lapsed after 12 months without an order and assigns 18,000 lapsing customers, 6,000 per arm, over four months. In the 90 days after assignment the holdout earns £1.20 of gross profit per customer, the message-only arm £1.70 and the 20%-off offer arm £1.85. Both message arms beat the holdout, but the offer adds only £0.15 per customer over the message, with a 95% interval from roughly −£0.20 to £0.50, and customers reactivated by the offer place fewer second orders. Unsubscribes are similar in both contacted arms. Duskmoor keeps the win-back message and drops the discount.
Pitfalls
A lapse definition that includes seasonal buyers is the first pitfall. Customers who buy once a year for a particular season are not lapsed between seasons, and messaging them early inflates every arm's purchases; set the definition from the repurchase interval and exclude known seasonal patterns. A holdout reached by other campaigns is the second. If the holdout receives a newsletter or a sale email that the win-back arms are suppressed from, or the reverse, the comparison measures that difference. Keep ordinary campaign eligibility identical. Reading reactivations without the holdout is the third, and the commonest: every post-message purchase is then credited to the programme.
Discount leakage matters too. A code shared on voucher sites reaches holdout customers and other shoppers; use single-use codes tied to the customer.
Common mistakes
Win-back tests are judged wrongly far more often than they are run wrongly. The lifecycle email programme playbook shows where the decision goes next, and the experiment library lists other retention designs.
- Counting every returning customer as reactivated. Subtract the holdout's natural return.
- Judging on revenue. The offer arm's revenue hides the discount; use gross profit.
- Skipping the message-only arm. Without it, the discount takes credit for the contact.
- Letting other campaigns differ between arms. The holdout must stay eligible for ordinary sends.
- Reusing a shared discount code. Leaked codes contaminate the holdout.
- Stopping at the first purchase. Second orders show whether the customer has really returned.
