Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Experiment Library

Cancellation Save Offer Test

A design for testing a save offer shown to subscribers who start cancelling, judged on retained revenue net of the discount, with easy-exit guardrails.
Experimentadvanced7 min readUpdated

Hypothesis

If subscribers who start to cancel are offered a pause or a discount before confirming, more will still be paying at a fixed horizon and the retained revenue will exceed the discount cost, as long as cancelling stays as easy as before. The decision behind it: show a pause or discount offer in the cancellation flow, and which one, or let people cancel directly.

The cancellation moment is unusual. Everyone who reaches it has already decided to leave, so the offer can only help the subset whose reason is temporary or price-related: a busy month, a holiday, a budget squeeze. For everyone else it is an obstacle, and an obstacle at the exit is the fastest way to turn a lost subscriber into a complaint. The hypothesis therefore has a condition built in, and the guardrails enforce it. The customer retention strategy guide explains why most churn is better prevented earlier, through onboarding, value and service; a save offer is a last intervention, not a retention strategy. For an online store's product subscriptions, the subscribe-and-save programmes guide puts skip, date change and pause controls in every delivery reminder, so fewer subscribers reach the cancel step at all. The retention experimentation loop helps decide whether this test deserves a place ahead of earlier-stage ones. Record the offer, its terms and the guardrail limits in the experiment hypothesis template before launch.

Metrics and guardrails

The primary metric is net retained revenue per cancellation starter at a fixed horizon: revenue collected from each arm's starters between entering the flow and the horizon, minus the cost of any discount given, divided by the number of starters assigned to the arm. The denominator includes everyone who entered the flow, so subscribers who decline the offer and cancel still count. The horizon must run past the end of the offer, so that a discount which merely delays cancellation is seen as such. Report the share of starters still paying at the horizon alongside it, read by entry Cohort as the churn and retention metrics guide describes.

The guardrails protect the subscriber's right to leave and the business's reputation. Complaints about cancelling, logged by support and by any public review channel with the arm attached, catch frustration that the revenue figure never shows. Steps and time to cancel, measured from entering the flow to confirmation for those who do cancel, must not rise beyond the single extra screen the design allows. Cancellation after the offer ends tracks whether saved subscribers leave as soon as the pause or discount runs out. The Competition and Markets Authority and the Federal Trade Commission, in its business guidance, treat clear subscription terms and easy cancellation as consumer protection matters; use their guidance as general direction when setting limits, and take qualified advice on the flow itself. The academy checkpoint on trial and save offer experiments tests reading a save offer on retained revenue net of the discount and the easy-exit guardrails that stop it trapping customers.

Design

Randomise at the level of the subscriber who starts the cancellation flow, at the moment they enter it, and keep them in that arm if they return to the flow later. The control is direct cancellation as it works today. The challenger shows one offer screen, a pause or a discount, with a decline action as prominent as the accept action; declining completes the cancellation with no further screens. Everything else, including any reason survey already in the flow, stays identical.

Design elementChoiceReason
Unit of randomisationSubscriber who starts the cancellation flowOnly people at the exit can be saved
ArmsDirect cancellation versus one offer stepEach extra offer arm divides a small group
Decline actionOne action, equal prominence, cancellation completesEase of exit is held constant
Offer termsFixed end date and price after it, stated on the screenSubscribers know what they are agreeing to
Primary metricNet retained revenue per starter at a fixed horizonCounts the discount and the delayed leavers
Repeat visitsSubscriber keeps their first armPrevents shopping for an offer

If the business wants to compare a pause with a discount, run the winner of this test against the other offer as a second test, unless cancellation volume is large enough for three arms. The win-back offer test covers the other side of the same problem: customers who have already left.

Sample size reasoning

Only people who start cancelling enter, so the sample builds slowly. The inputs are the share of starters still paying at the horizon without an offer, the smallest improvement in net retained revenue that would justify the offer's cost and the confidence required; the sample size and significance guide shows how they combine. Revenue per starter is skewed, because most starters leave and contribute nothing, so it needs more starters than a simple retention rate would. A small subscriber base should accept a long enrolment rather than read early.

Duration

Enrol to the planned sample, then wait for a horizon long enough to see whether saved subscribers cancel when the offer ends: at least one full billing period after the longest offer would finish for the last starter enrolled. Decide the end date before launch. Avoid a price rise or a large product change during enrolment, since either would change why people cancel.

Reading the results and the decision rule

Read once, at the horizon. Compare net retained revenue per starter with its confidence interval, then the share still paying, then each guardrail. Look at cancellation reasons by arm only after the primary read. The rule, written before launch:

ResultDecision
Net retained revenue per starter is higher after the discount, and complaints and time to cancel do not riseKeep the offer
Revenue is higher but complaints or time to cancel riseRemove the offer or redesign it, then retest
Saved subscribers mostly cancel when the offer ends, leaving no net gainRemove the offer; it delays churn at a cost
No reliable differenceLet people cancel directly

The retention and payback guide shows how a change in retained revenue alters LTV and the Payback Period of the customers each channel acquires, which is where the result earns its place in planning.

Illustrative example — not a real result

A houseplant subscription tests a pause option

Pebblewick, an invented monthly houseplant subscription, assigns 6,000 cancellation starters evenly between direct cancellation and an offer screen proposing a pause of up to two months, with a "No thanks, cancel" button the same size as the pause button. At 120 days from entering the flow, net retained revenue per starter is £1.90 in the control arm and £4.70 in the offer arm, with a 95% interval for the difference of roughly £1.80 to £3.80. Median time to cancel, for those who cancel, rises from 40 to 52 seconds, inside the one-screen limit; complaints are level. Of subscribers who paused, about two thirds resumed paying. Pebblewick keeps the pause offer and plans a later test of a discount against it.

Pitfalls

Making cancellation harder in the offer arm is the pitfall that can turn a retention test into a regulatory and reputational problem. Extra screens, a hidden decline link, a forced call or a confusing button order all inflate the save rate by obstruction. Keep the decline action single and equal. Reading saves at the moment of the offer is the second pitfall: acceptance looks impressive and says little until the offer ends. Leaving the discount cost out is the third. A discount retains revenue at a lower price, and a pause retains none during the pause; only the net figure says whether either was worth giving.

Watch for subscribers who learn the offer. If people discover that starting to cancel yields a discount, some will start the flow on purpose; a fixed arm per subscriber and a limit of one offer per subscriber per period keep that in check.

The test judges one screen in the flow. The rest of the exit, from the confirmation and final charge to one optional exit question and what marketing may send afterwards, is designed in the guide to designing the cancellation experience.

Common mistakes

Save-offer tests fail when they are allowed to make leaving harder or are read too soon. The experiment library lists the other lifecycle designs.

  • Adding friction and calling it an offer. One screen, one decline action, then cancellation.
  • Counting accepted offers as saves. Read who is still paying after the offer ends.
  • Ignoring the discount's cost. Judge net retained revenue, not gross.
  • Re-offering on every visit to the flow. Fix the arm and limit offers per subscriber.
  • Testing two offers at once on a small base. Test one against direct cancellation first.
  • Treating regulator guidance as permission. It is direction; take qualified advice on the flow.

Frequently asked questions

Is a save offer the same as making cancellation harder?

It must not be. A save offer is one extra screen that a subscriber can decline in a single action, after which cancellation completes exactly as before. Regulators such as the Competition and Markets Authority and the Federal Trade Commission treat easy cancellation of subscriptions as a consumer protection matter; their guidance is general direction, not legal advice, and the design should be checked with someone qualified before launch.

Why not count saves at the moment the offer is accepted?

Because many accepted offers only delay the cancellation. A subscriber who takes a discount and cancels when it ends has cost the business the discount and bought nothing. Reading net revenue at a horizon that runs past the end of the offer counts those people correctly.

Should the test compare a pause with a discount?

Only once a single offer has shown that the moment is worth intervening in, or when the subscriber base is large enough for a second offer arm. Each extra arm divides an already small group of cancellation starters, so most teams first test one offer against direct cancellation, then test the alternative offer against the winner.

Terms used on this page

Frameworks

Where this page is used

Part of these learning paths

Also related

Sources

  1. Competition and Markets Authority — GOV.UK
  2. FTC business guidance — Federal Trade Commission

Related