Why this matters
The advanced ecommerce level valued customer cohorts and judged a loyalty scheme on its cost: one decision about one idea. The expert question is the system. Does the business know where its customers go and why, and does its follow-up add anything customers would not have done anyway?
The customer retention audit answers the first by rating the whole system that keeps customers, with cohort reporting and reasons for leaving as gates. The win-back offer test answers the second for lapsed customers, pricing the contact and the discount separately against a holdoutA randomly chosen part of an audience kept from exposure to measure what exposure caused.. The lifecycle email programme playbook supplies the ownership around both: a journey inventory, a named owner for every journey and a holdout wherever a journey can afford one.
The baking supplies shop and its lapsed customers
Muvird Baking Supplies is an invented online shop selling baking ingredients, tins and decorating kits, with a small monthly decorating kit subscription. It is early autumn, and the build for the end-of-year peak starts next month.
Muvird Baking Supplies: the evidence gathered for the audit
Team: the owner; an operations manager who owns packing and the order data; an email and content marketer; two customer service advisers, one of them the service lead; a part-time developer; a freelance analyst a few days a month.
The figure the owner reads monthly: returning customers' share of orders, between 38% and 41% every month for two years.
Cohorts rebuilt by the freelance analyst from the order export for the audit:
| Joining quarter | New customers | Ordered again within 90 days |
|---|---|---|
| Spring, last year | 1,100 | 31% |
| Summer, last year | 1,000 | 24% |
| Autumn, last year | 1,300 | 22% |
| Winter peak, last year | 2,600 | 12% |
| Spring, this year | 1,200 | 20% |
| Days from first order to second, customers with a second order | Share of second orders |
|---|---|
| 0–30 | 34% |
| 31–60 | 29% |
| 61–90 | 18% |
| 91–120 | 7% |
| 121–180 | 6% |
| Over 180 | 6% |
What the owner believes: customers leave for cheaper flour at the supermarket. Service contacts are free text with no tags; the auditor hand-read 420 from last quarter: melted or bloomed chocolate and decorations in warm weather 96, dented or broken tins 71, items out of stock or substituted 64, delivery charge on heavy flour orders 38, price found cheaper elsewhere 22, other 129. Kit subscription: joined online, cancelled only by emailing service; 85 cancellations last year, no reason recorded.
Other evidence: product pages promise "packed to arrive as it left us"; a welcome email with a sign-up code and a review request after delivery, nothing pointing to a second order; refunds given readily, causes never traced; email, phone and chat with a used help centre; customers chasing a substitution repeat their order number by email and again by phone, and the help centre answer on delivery charges was added after complaints; a monthly newsletter with a code to every consented subscriber; a post-delivery star rating reported monthly, open text unread; no loyalty scheme, by the owner's choice; nobody has walked the journey as a customer.
Win-back today: a "we miss you" email every month to everyone 60 days past their last order, with no end. Customers now past 90 days without an order: 6,900, of whom 1,700 order only in the six weeks before the end-of-year celebrations and 1,300 bought as guests with no marketing consent. Last year 7% of customers sent the monthly email ordered within 60 days of it; there was never a holdout.
The proposal: 15% off in every win-back email. Gross profit after goods, packing and delivery: 40% of order value. Average order from a returning lapsed customer: £38. Contactable, non-seasonal customers reaching 90 days without an order: about 600 a month. No sale is planned between January and the next peak.
Audit before offer
The audit is explicit about order. A red band, any gate scored missing, pauses new retention offers until retention is defined once and read by CohortA group of customers who share a starting event and period, tracked together over time., and contacts and cancellations are tagged by reason. A discount added now would be judged on the blended figure that hid the problem, and nobody could say whether it brought anyone back.
That does not stop the test being written. The lapse definition needs the same cohort view the first fix builds, and the win-back message is stronger when it can answer the reason customers left. The expert move is sequencing: fixes start now, the test is designed now, and it launches after the peak once the gates have been rescored. The win-back email also belongs in the playbook's journey inventory, with an owner, a trigger, an exit and a holdout. Nobody at Muvird can name any of those, which is why it has run every month with no end.
Task
- Before reading the results, record what the owner believes about why customers leave, then score the two gates and say what the cohorts show that the blended figure hides.
- Score the other rows missing, partial or solid, give the band, and name anything that must stop at once rather than wait for the fix list.
- Rank the fixes in the audit's order, each with an owner, the guide that explains it and a rescore date.
- Set the lapse definition from the gap between orders, and say who is excluded before anyone is randomised.
- Design the arms, the primary metric and the guardrails, and work out what the offer must achieve just to match the message alone.
- Size the test and set its windows and read date against the shop's calendar.
- Write the decision rule, including what an inconclusive result means, and enter the win-back journey in the inventory.
Worked answer
The gates. Retention read by cohort scores missing. The returning share of orders held steady because the customer base kept growing, while each spring cohort returned less than the one before; the churn and retention metrics guide shows how a blended churn rateShare of customers at the start of a period who left during it. can hide exactly this. Reasons for leaving scores missing: free text only, and subscription cancellations untagged. The hand-read sample is provisional evidence, not a score, but it already contradicts the owner: damaged goods and stock-outs far outnumber price.
The other rows and the band. Survey scores tested: missing. Onboarding: partial, since nothing points a new customer to a second order. Promises kept: missing, as the packing promise meets warm-weather damage. Customer effort: partial, since the delivery-charge answer came after complaints while substitution chasers still repeat their order number. Service recovery: partial, generous refunds with causes never traced. Support channels: solid. Cancellation: missing, because the subscription is joined online and left only by email. Win-back with a stopping rule: missing. Loyalty design: solid, a deliberate absence. Lifecycle timing and voice of the customer: partial. Journey walked: missing. The band is red. Two things stop today, outside the fix list: guest buyers with no marketing consent come off the lapsed email, and the endless monthly send is switched off, because the win-back row asks for contactable leavers only and a defined end.
The fix list, ordered as the audit ranks it and, within each group, by how many customers each fix touches:
| Fix | Owner | Explained by |
|---|---|---|
| One definition of active and lapsed; joining cohorts read monthly | Operations manager with the analyst | The churn and retention metrics guide |
| Reason tags on every service contact and every kit cancellation | Service lead | The tutorial on tagging support contacts by reason |
| Warm-weather packing and tin protection, the promise made true | Operations manager | The tutorial on checking promises against the experience |
| A post-purchase path to the second order | Email and content marketer | The repeat purchase and loyalty guide |
| Online cancellation for the kit subscription | Owner and developer | The guide to designing the cancellation experience |
| Win-back rebuilt with a stopping rule, then tested | Email and content marketer | The win-back offer test |
| Open text read monthly; the journey walked each quarter | Service lead and owner | The voice of the customer guide |
The rescore falls in January, once the peak's contacts carry tags, and the test launches only if both gates have moved off missing. Real tags may reorder the middle of the list; that is the point of rescoring.
Lapse and exclusions. The gap table shows the old trigger firing while a large share of second orders were still due, and the curve flattening only a month later. The lapse point moves to where it flattens, as the repeat purchase and loyalty guide sets it from the time between orders. Seasonal bakers are not lapsed between seasons, so they leave the test and get a reminder of their own before the peak. Guests without consent, anyone unsubscribed and active kit subscribers are removed before randomising. Each customer is assigned once, on meeting the definition, and flagged in the customer records. The backlog already past the lapse point stays out: months of the old email have contaminated it, and its treatment follows the result.
Arms, metric and guardrails. Equal thirds: a single-use code tied to the customer, the same sequence without it, and a holdout with no win-backMessages aimed at customers who have stopped buying or engaging, sent once they pass their usual gap between purchases. contact that stays on the newsletter like everyone else. Both contacted arms get one short sequence over a month, then stop. Its content returns with something new, such as the new warm-weather packing, as the repeat purchase guide advises. Success is read as incremental gross profit per assigned customer, set against the holdout and counted from the order system. Guardrails: discount cost per reactivated customer, unsubscribes and complaints per assigned customer, and second orders.
Muvird Baking Supplies: break-even, sizing and windows
- Lapse point: 63% of second orders arrive within 60 days, the old trigger, and 81% within 90, after which the curve flattens. New definition: 90 days without an order.
- Gross profit on a £38 order at full price: £38 × 0.40 = £15.20. With 15% off: £15.20 − £5.70 = £9.50.
- The offer beats the message alone only when offer rate × £9.50 > message rate × £15.20, so the offer must bring back more than 1.6 times as many customers. If the message alone brings back 7%, the offer needs above 11.2%.
- Enrolment January to April at about 600 a month: 2,400 customers, 800 per arm.
- Rough detectable gap, planning on the 7% seen last year: standard error √(2 × 0.07 × 0.93 ÷ 800) ≈ 1.3 points, so about 3.6 points at conventional confidence and power. The offer's break-even gap of about 4 points is just within reach.
- First-order window 60 days from assignment, second-order window a further 60: the last enrolment's windows close at the end of June and the end of August. One read in September, before the peak build in October.
- The owner's guardrail, set before launch: pause a contacted arm if complaints pass 0.3% of its assigned customers.
The holdout test readiness checklist is run before the first assignment: the decision named, the smallest gap that matters written down, the holdout flagged so the comparison can be rerun, other campaigns identical across arms and the read date fixed. The lapse definition, arms, rule and read date are written into the experiment hypothesis template before launch.
The decision rule, written before launch:
| Result at the September read | Decision |
|---|---|
| The offer beats the message alone on gross profit per assigned customer, guardrails hold | Send the single-use offer |
| The message beats the holdout; the offer adds nothing distinguishable | Send the message without the offer |
| No contacted arm earns more than the holdout | End win-back sends and move the effort to the second-order path |
| A contacted arm breaks the complaint limit | Pause it and review the lapse definition and consent |
An offer that cannot be told apart from the message loses, because the discount is the claim that has to prove itself.
The inventory row. Win-back journey: lapse stage; trigger the new lapse point, non-seasonal and contactable; exit on an order or at the end of the sequence; goal incremental gross profit against the holdout; owner the email and content marketer; the analyst owns the lapse definition and the holdout flag; the owner signs off the consent basis. After the test a standing holdout stays on the journey.
Common mistakes
- Pricing the offer before reading cohorts. The blended figure would have flattered any discount.
- Taking the owner's belief as the reason. The sample points at packing and stock, not price.
- Firing win-back inside the buying cycle. Customers still due to return inflate every arm.
- Counting seasonal bakers as lapsed. They return on their own calendar.
- Leaving guests without consent in the arms. Remove them before randomising, not after.
- Reading an inconclusive offer as a win. The discount carries the burden of proof.
- Running the test into the peak. Peak buying returns lapsed customers to all three arms together, so the difference between them disappears.
The capstone for this level puts this retention programme beside the peak and the Shopping programme in one year's trading plan.
