Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

E-commerce Marketing › Expert

Auditing Retention and Testing Win-Back Offers for a Baking Supplies Shop

You audit an invented baking supplies shop's retention system before anyone adds a discount to its win-back emails. You score the gates first, rank the fixes with owners and a rescore date, and design a three-arm test of an offer, a message alone and no contact, with every rule written before the first lapsed customer is assigned.
ExerciseExpert levelModule 3: Retention as an owned programme10 minUpdated

Why this matters

The advanced ecommerce level valued customer cohorts and judged a loyalty scheme on its cost: one decision about one idea. The expert question is the system. Does the business know where its customers go and why, and does its follow-up add anything customers would not have done anyway?

The customer retention audit answers the first by rating the whole system that keeps customers, with cohort reporting and reasons for leaving as gates. The win-back offer test answers the second for lapsed customers, pricing the contact and the discount separately against a holdout. The lifecycle email programme playbook supplies the ownership around both: a journey inventory, a named owner for every journey and a holdout wherever a journey can afford one.

The baking supplies shop and its lapsed customers

Muvird Baking Supplies is an invented online shop selling baking ingredients, tins and decorating kits, with a small monthly decorating kit subscription. It is early autumn, and the build for the end-of-year peak starts next month.

Illustrative example — not a real result

Muvird Baking Supplies: the evidence gathered for the audit

Team: the owner; an operations manager who owns packing and the order data; an email and content marketer; two customer service advisers, one of them the service lead; a part-time developer; a freelance analyst a few days a month.

The figure the owner reads monthly: returning customers' share of orders, between 38% and 41% every month for two years.

Cohorts rebuilt by the freelance analyst from the order export for the audit:

Joining quarterNew customersOrdered again within 90 days
Spring, last year1,10031%
Summer, last year1,00024%
Autumn, last year1,30022%
Winter peak, last year2,60012%
Spring, this year1,20020%
Days from first order to second, customers with a second orderShare of second orders
0–3034%
31–6029%
61–9018%
91–1207%
121–1806%
Over 1806%

What the owner believes: customers leave for cheaper flour at the supermarket. Service contacts are free text with no tags; the auditor hand-read 420 from last quarter: melted or bloomed chocolate and decorations in warm weather 96, dented or broken tins 71, items out of stock or substituted 64, delivery charge on heavy flour orders 38, price found cheaper elsewhere 22, other 129. Kit subscription: joined online, cancelled only by emailing service; 85 cancellations last year, no reason recorded.

Other evidence: product pages promise "packed to arrive as it left us"; a welcome email with a sign-up code and a review request after delivery, nothing pointing to a second order; refunds given readily, causes never traced; email, phone and chat with a used help centre; customers chasing a substitution repeat their order number by email and again by phone, and the help centre answer on delivery charges was added after complaints; a monthly newsletter with a code to every consented subscriber; a post-delivery star rating reported monthly, open text unread; no loyalty scheme, by the owner's choice; nobody has walked the journey as a customer.

Win-back today: a "we miss you" email every month to everyone 60 days past their last order, with no end. Customers now past 90 days without an order: 6,900, of whom 1,700 order only in the six weeks before the end-of-year celebrations and 1,300 bought as guests with no marketing consent. Last year 7% of customers sent the monthly email ordered within 60 days of it; there was never a holdout.

The proposal: 15% off in every win-back email. Gross profit after goods, packing and delivery: 40% of order value. Average order from a returning lapsed customer: £38. Contactable, non-seasonal customers reaching 90 days without an order: about 600 a month. No sale is planned between January and the next peak.

Audit before offer

The audit is explicit about order. A red band, any gate scored missing, pauses new retention offers until retention is defined once and read by Cohort, and contacts and cancellations are tagged by reason. A discount added now would be judged on the blended figure that hid the problem, and nobody could say whether it brought anyone back.

That does not stop the test being written. The lapse definition needs the same cohort view the first fix builds, and the win-back message is stronger when it can answer the reason customers left. The expert move is sequencing: fixes start now, the test is designed now, and it launches after the peak once the gates have been rescored. The win-back email also belongs in the playbook's journey inventory, with an owner, a trigger, an exit and a holdout. Nobody at Muvird can name any of those, which is why it has run every month with no end.

Task

  1. Before reading the results, record what the owner believes about why customers leave, then score the two gates and say what the cohorts show that the blended figure hides.
  2. Score the other rows missing, partial or solid, give the band, and name anything that must stop at once rather than wait for the fix list.
  3. Rank the fixes in the audit's order, each with an owner, the guide that explains it and a rescore date.
  4. Set the lapse definition from the gap between orders, and say who is excluded before anyone is randomised.
  5. Design the arms, the primary metric and the guardrails, and work out what the offer must achieve just to match the message alone.
  6. Size the test and set its windows and read date against the shop's calendar.
  7. Write the decision rule, including what an inconclusive result means, and enter the win-back journey in the inventory.

Worked answer

The gates. Retention read by cohort scores missing. The returning share of orders held steady because the customer base kept growing, while each spring cohort returned less than the one before; the churn and retention metrics guide shows how a blended churn rate can hide exactly this. Reasons for leaving scores missing: free text only, and subscription cancellations untagged. The hand-read sample is provisional evidence, not a score, but it already contradicts the owner: damaged goods and stock-outs far outnumber price.

The other rows and the band. Survey scores tested: missing. Onboarding: partial, since nothing points a new customer to a second order. Promises kept: missing, as the packing promise meets warm-weather damage. Customer effort: partial, since the delivery-charge answer came after complaints while substitution chasers still repeat their order number. Service recovery: partial, generous refunds with causes never traced. Support channels: solid. Cancellation: missing, because the subscription is joined online and left only by email. Win-back with a stopping rule: missing. Loyalty design: solid, a deliberate absence. Lifecycle timing and voice of the customer: partial. Journey walked: missing. The band is red. Two things stop today, outside the fix list: guest buyers with no marketing consent come off the lapsed email, and the endless monthly send is switched off, because the win-back row asks for contactable leavers only and a defined end.

The fix list, ordered as the audit ranks it and, within each group, by how many customers each fix touches:

FixOwnerExplained by
One definition of active and lapsed; joining cohorts read monthlyOperations manager with the analystThe churn and retention metrics guide
Reason tags on every service contact and every kit cancellationService leadThe tutorial on tagging support contacts by reason
Warm-weather packing and tin protection, the promise made trueOperations managerThe tutorial on checking promises against the experience
A post-purchase path to the second orderEmail and content marketerThe repeat purchase and loyalty guide
Online cancellation for the kit subscriptionOwner and developerThe guide to designing the cancellation experience
Win-back rebuilt with a stopping rule, then testedEmail and content marketerThe win-back offer test
Open text read monthly; the journey walked each quarterService lead and ownerThe voice of the customer guide

The rescore falls in January, once the peak's contacts carry tags, and the test launches only if both gates have moved off missing. Real tags may reorder the middle of the list; that is the point of rescoring.

Lapse and exclusions. The gap table shows the old trigger firing while a large share of second orders were still due, and the curve flattening only a month later. The lapse point moves to where it flattens, as the repeat purchase and loyalty guide sets it from the time between orders. Seasonal bakers are not lapsed between seasons, so they leave the test and get a reminder of their own before the peak. Guests without consent, anyone unsubscribed and active kit subscribers are removed before randomising. Each customer is assigned once, on meeting the definition, and flagged in the customer records. The backlog already past the lapse point stays out: months of the old email have contaminated it, and its treatment follows the result.

Arms, metric and guardrails. Equal thirds: a single-use code tied to the customer, the same sequence without it, and a holdout with no win-back contact that stays on the newsletter like everyone else. Both contacted arms get one short sequence over a month, then stop. Its content returns with something new, such as the new warm-weather packing, as the repeat purchase guide advises. Success is read as incremental gross profit per assigned customer, set against the holdout and counted from the order system. Guardrails: discount cost per reactivated customer, unsubscribes and complaints per assigned customer, and second orders.

Illustrative example — not a real result

Muvird Baking Supplies: break-even, sizing and windows

  • Lapse point: 63% of second orders arrive within 60 days, the old trigger, and 81% within 90, after which the curve flattens. New definition: 90 days without an order.
  • Gross profit on a £38 order at full price: £38 × 0.40 = £15.20. With 15% off: £15.20 − £5.70 = £9.50.
  • The offer beats the message alone only when offer rate × £9.50 > message rate × £15.20, so the offer must bring back more than 1.6 times as many customers. If the message alone brings back 7%, the offer needs above 11.2%.
  • Enrolment January to April at about 600 a month: 2,400 customers, 800 per arm.
  • Rough detectable gap, planning on the 7% seen last year: standard error √(2 × 0.07 × 0.93 ÷ 800) ≈ 1.3 points, so about 3.6 points at conventional confidence and power. The offer's break-even gap of about 4 points is just within reach.
  • First-order window 60 days from assignment, second-order window a further 60: the last enrolment's windows close at the end of June and the end of August. One read in September, before the peak build in October.
  • The owner's guardrail, set before launch: pause a contacted arm if complaints pass 0.3% of its assigned customers.

The holdout test readiness checklist is run before the first assignment: the decision named, the smallest gap that matters written down, the holdout flagged so the comparison can be rerun, other campaigns identical across arms and the read date fixed. The lapse definition, arms, rule and read date are written into the experiment hypothesis template before launch.

The decision rule, written before launch:

Result at the September readDecision
The offer beats the message alone on gross profit per assigned customer, guardrails holdSend the single-use offer
The message beats the holdout; the offer adds nothing distinguishableSend the message without the offer
No contacted arm earns more than the holdoutEnd win-back sends and move the effort to the second-order path
A contacted arm breaks the complaint limitPause it and review the lapse definition and consent

An offer that cannot be told apart from the message loses, because the discount is the claim that has to prove itself.

The inventory row. Win-back journey: lapse stage; trigger the new lapse point, non-seasonal and contactable; exit on an order or at the end of the sequence; goal incremental gross profit against the holdout; owner the email and content marketer; the analyst owns the lapse definition and the holdout flag; the owner signs off the consent basis. After the test a standing holdout stays on the journey.

Common mistakes

  • Pricing the offer before reading cohorts. The blended figure would have flattered any discount.
  • Taking the owner's belief as the reason. The sample points at packing and stock, not price.
  • Firing win-back inside the buying cycle. Customers still due to return inflate every arm.
  • Counting seasonal bakers as lapsed. They return on their own calendar.
  • Leaving guests without consent in the arms. Remove them before randomising, not after.
  • Reading an inconclusive offer as a win. The discount carries the burden of proof.
  • Running the test into the peak. Peak buying returns lapsed customers to all three arms together, so the difference between them disappears.

The capstone for this level puts this retention programme beside the peak and the Shopping programme in one year's trading plan.

Pages this lesson applies

Terms used on this page

Also related

Related