Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Marketing Frameworks › The Experimentation Loop and ICE

Experimentation Loop for Retention

Running the experimentation loop for retention: persistent holdout groups, long readouts by cohort, novelty effects and choosing leading metrics you can trust.
Objective guideintermediate7 min readUpdated

Why a quick split test misjudges retention

The loop's usual test ends when the planned sample has seen the change and done something measurable. For retention, the measurable thing is that a customer is still active or still paying after some weeks or months, and that outcome is the product of many small decisions along the way. A test that stops when the sample is reached has measured the first few days of a long story.

There are three further complications. People enter the test as they sign up, so any calendar date catches some at their first day and others at their fortieth. The first weeks of any change carry curiosity that fades. And retention programmes are made of many messages, emails, push notifications, in-app prompts, that interact: a winning email may only win because the push it was paired with was poor. The customer retention strategy guide sets out the levers worth testing; the loop decides how to test them honestly.

Decisions that change for retention

Loop stepIn the generic loopIn a retention loopDecision it forces
Test designShort split, ended at the planned sampleA holdout kept for the length of the effectHow long the holdout lasts, and what it costs
Readout timingA calendar dateFixed cohort ages, such as day seven and day thirtyWhich ages are the decision point
InterpretationEffect at the end of the testEffect at early and later ages comparedWhen an early lift counts as lasting
MetricThe outcome, observed directlyA leading metric during the test, the outcome laterWhich leading metric is proven to predict staying
Unit of changeOne variant of one assetA programme of messages tested togetherWhat is held constant elsewhere in the lifecycle
LearnWrite-up after the testWrite-up updated at each cohort ageWho owns the later updates

Use persistent holdouts instead of short splits

Retention changes accumulate: one more session in week one makes a habit slightly likelier in week two, and so on. Measuring that requires a group that never receives the change for as long as the effect takes to show. A persistent holdout costs something, because those customers miss what may be a better experience, and it tests patience. The alternative, a short split ended at the planned sample, measures the first-week bump and calls it retention. Choose a holdout sized for the cohort age that matters and agree in advance how long it runs. This is Incrementality measurement applied to the customer base, and the email metrics and testing guide covers holdouts for automated flows specifically.

Read results at fixed cohort ages

Put each customer into the test at a defined start, usually sign-up or first purchase, and read the outcome at the same age for everyone: still active at day seven, day thirty, day ninety. Reading by calendar date mixes customers who joined at the start of the test with those who joined last week and dilutes the effect. The cohort analysis guide explains building the table; the loop decides in advance which age is the decision point and which ages are reported for context. The trade-off is time: a day-ninety decision takes a quarter. Choose the earliest age at which past cohorts show the curve has largely settled.

Separate novelty from lasting change

A new onboarding sequence, a redesigned home screen or a fresh notification style often lifts engagement in the first days because it is different. The lift can fade, stay or even reverse. The way to tell is to compare the gap between test and holdout at an early cohort age with the gap at a later one. A gap that shrinks towards nothing was novelty; one that holds or widens is a real change. Choose to report both ages in every write-up, and to treat a decision made only on the early age as provisional.

Validate leading metrics before deciding on them

Because the true outcome is slow, teams want an early signal: completing a setup step, a second session, a first repeat purchase. Such signals help only if they predict staying, and if the change being tested does not break the link. Check on past cohorts that customers who hit the signal retain better, and check after the test that the signal and the later retention moved together in the test group too. A signal that rises because the change nags people into it, without changing why they stay, will mislead. The AARRR framework and the KPI tree for retention are both ways of locating the activation step that deserves to be the leading metric.

Test programmes, not single sends

A retention customer receives a welcome series, push notifications, in-app prompts and perhaps replenishment emails, all at once. Testing one email in isolation measures it against whatever else happened that week, and the result may flip when the rest of the programme changes. Test the programme as a unit where possible: the whole welcome sequence, or the full mix of channels for a stage. The welcome series tutorial, the push and in-app messaging guide and the email and SMS flows guide for ecommerce cover the components. Choose single-send tests only for creative details inside a programme already proven against a holdout.

Decision points in a worked retention test

Illustrative example — not a real result

A new onboarding sequence at an invented meditation app, Quietfern

Quietfern offers guided meditation by subscription after a 14-day trial. Research: of users who never finished a second session in their first three days, only 9% were still active at day thirty, against 38% of those who did. Hypothesis: replacing the current single welcome email with a sequence of three emails and two push prompts, each pointing to a short second session, will raise day-thirty retention.

Loop artefactContent
DesignNew sign-ups assigned 80% to the new sequence, 20% to a holdout that keeps the old email; holdout kept for 60 days
UnitThe whole sequence, with other lifecycle messages frozen for both groups
Leading metricSecond session within 3 days; validated on 6 months of past cohorts
Decision pointActive at day 30, from each user's sign-up
Sample11,000 sign-ups in four weeks, 2,200 in the holdout
Day 7 readoutActive: 52% new sequence, 44% holdout; second session: 61% against 47%
Day 30 readoutActive: 33% new sequence, 29% holdout
Day 60 readoutActive: 24% new sequence, 22% holdout

The gap shrank from 8 points at day seven to 4 at day thirty and 2 at day sixty: part novelty, part lasting. Trial-to-paid conversion rose from 18% to 20%. The sequence was adopted, and a 5% holdout kept to watch whether the day-sixty gap survives.

A team reading the day-seven numbers alone would have claimed an effect four times larger than the one that lasted. The fixed cohort ages, the holdout and the frozen lifecycle messages are what made the smaller, real result visible.

Measuring the loop: KPIs and cadence for retention tests

The decision KPI is retention at a fixed cohort age, compared between test and holdout, with the gap at earlier and later ages reported beside it. Supporting measures are the validated leading metric, revenue retention or lifetime value (LTV) at the later ages, and guardrails such as unsubscribe and notification opt-out rates, which rise when a sequence becomes pushy. For the programme, measure how many tests reached their final cohort age before a decision was taken, and how often the early and late readouts agreed. Review leading metrics weekly, cohort-age readouts as each age is reached, and the holdout list monthly so no holdout runs on forgotten. An exploration in an analytics tool can supply the cohort table for signed-in users; billing data is usually more reliable for paid retention. Reserve the term A/B Test for short tests inside the programme, and call the long comparison what it is, a holdout.

Where retention tests mislead

Holdouts leak: customers in the holdout may receive similar messages from other systems, or hear about the feature from users who got it, which shrinks the measured gap. Cohort ages can reflect a season when the whole test ran in one. A leading metric can be gamed by the very change being tested. And an onboarding effect measured on one acquisition channel's customers may not hold for another channel's, so a change in the acquisition mix can make a proven programme look worse.

Common mistakes

  • Ending at the planned sample. Retention effects appear after the sample is reached, not before.
  • Reading by calendar date. Mixing cohort ages dilutes the effect and hides novelty.
  • Deciding on day seven. Early gaps are the most flattering and the least durable.
  • Trusting an unvalidated leading metric. A step people are pushed into may not predict staying.
  • Testing one email at a time. The rest of the lifecycle changes the result.
  • Forgetting the holdout. Long-running holdouts need an owner and an end date.

Frequently asked questions

How large should a persistent retention holdout be?

Large enough that the retention difference the programme is expected to make can be detected at the cohort age being read, and small enough that the business accepts the cost of withholding the programme from that group. The calculation is the same sample-size reasoning as any test, run on the retention rate at the read age rather than on a click rate, and it often argues for pooling several weeks of new customers into each group.

Can retention tests be run on existing customers, or only on new ones?

Both, but they answer different questions. Tests on new customers, such as onboarding, have clean cohorts that start together. Tests on existing customers mix people at very different ages and habits, so the groups need to be balanced on tenure and activity, and the readout should be split by tenure band rather than reported as one figure.

What should happen to the holdout group when the test ends?

Release it into the winning programme if the result is clear, but consider keeping a small long-term holdout for any programme the business will run indefinitely. A standing holdout keeps showing whether the programme still earns its place as the product, the audience and the competition change around it.

Terms used on this page

Frameworks

Where this page is used

Part of these learning paths

Also related

Sources

  1. How Not To Run an A/B Test — Evan Miller
  2. [GA4] Get started with Explorations — Google Analytics Help

Related