Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Experiment Library

Inactive Subscriber Sunset Test

An experiment design for inactive subscribers: re-engage then suppress, or keep mailing, judged on revenue and complaint signals together.
Experimentadvanced7 min readUpdated

Hypothesis

If inactive subscribers receive a short re-engagement sequence and non-responders are suppressed, revenue from that group will fall only slightly while complaints fall, because the people who no longer want the emails stop receiving them. The decision is whether to re-engage and then suppress inactive subscribers, or to keep mailing them at the normal cadence.

The argument for suppression is about the list as a whole: mailbox providers weigh what recipients do, and a long tail of people who never interact dilutes the signal for everyone. The argument against is about money: some inactive subscribers still buy now and then, and a suppressed address cannot be sold to. The test puts a price on the second argument and watches the first. The segmentation guide sets out engagement tiers and how suppression differs from deletion; this design takes the inactive tier it defines and asks what suppressing it costs. Record the inactivity rule, the sequence and the window in the experiment hypothesis template before the split.

Metrics and guardrails

The primary metric is revenue per inactive subscriber over the test window: all revenue attributable to each arm's members, from email and, where the data allows, from any channel, divided by the number assigned to the arm at the split. Counting revenue from all channels matters here, because a suppressed subscriber who still buys through search or the app has not been lost, and an email-only view would overstate the cost of suppression.

Three guardrails complete the picture. The Spam Complaint Rate per arm is the benefit side of the hypothesis: Google's email sender guidelines set requirements for bulk senders that include keeping reported spam low, and the deliverability guide explains how complaints shape placement for the whole list. The reactivation rate, the share of the suppression arm that responds to the sequence and becomes active again, shows how many people the sequence rescues. The third guardrail sits outside the test: the engagement of active subscribers, who are not randomised, is watched for sender-level effects, because a reputation change caused by either arm reaches them too. Read unsubscribes per arm as a diagnostic; the sequence will usually prompt some, which is part of the point.

Design

Define inactivity first, on behaviour that mail clients cannot fake: no clicks, purchases, replies or visits from email within a chosen window. Take everyone who meets the rule on the start date and split them at random into two arms. The suppression arm receives a short re-engagement sequence asking for a positive signal, and everyone who does not give one is moved to the suppression list at the sequence's end. The control arm keeps receiving the normal cadence. Active subscribers are not part of either arm and their mail does not change.

Design elementChoiceReason
Unit of randomisationSubscriber within the inactive segmentThe decision applies to this segment only
ArmsRe-engage then suppress, against keep mailingCompares the policy with the status quo
Inactivity ruleClicks, purchases, replies or visits, not opens aloneOpens are unreliable signals
Held constantContent and cadence for active subscribers; transactional mail for everyoneIsolates the policy
Primary metricRevenue per assigned inactive subscriber, all channels where possibleCounts revenue lost and revenue that moves channel
Shared effectActive-subscriber engagement watched, not randomisedReputation is domain-wide

Segmentation decides what the test can say afterwards. Name in advance any sub-segments you expect to behave differently, such as former buyers against people who never bought, so that the decision rule can keep mailing one of them without hunting through the data for a winner.

Sample size reasoning

Inactive subscribers buy rarely, so revenue per subscriber is a small, skewed number and the test needs a large inactive segment or a long window. Most members of either arm will spend nothing, a few will buy, and the difference between arms will be hard to separate from chance unless the segment is big. The sample size and significance guide explains the trade-off between segment size, spread and window length. A test that cannot be powered should be replaced by a judgement call with the complaint data: if the segment is small, its revenue is small too, and the deliverability case is usually enough on its own.

Duration

Run the re-engagement sequence plus a window long enough for occasional buyers to purchase, typically several purchase cycles. Inactive subscribers who still buy do so rarely, and a short window will show the suppression arm losing nothing simply because nobody in either arm bought yet. Fix the window before launch, avoid starting just before the business's seasonal peak unless the window spans it fully, and keep the inactivity rule frozen for the whole period so that people do not drift between definitions.

Reading the results and the decision rule

Read once at the planned end. Compare revenue per inactive subscriber between arms with its interval, then read complaints and reactivation, then check active-subscriber engagement for any sender-level change. Read the pre-named sub-segments last. The rule, written before launch:

ResultDecision
Suppression arm gives up little revenue and complaints fallAdopt the sunset policy for the whole inactive segment
Suppression arm loses revenue concentrated in one pre-named sub-segmentSuppress the rest; keep mailing that sub-segment at a lower cadence
Suppression arm loses substantial revenue across the segmentKeep mailing, but tighten the inactivity rule and re-test
Test underpowered, no reliable differenceDecide on the complaint data and the deliverability guidance; suppression is the default

Pitfalls

Sender reputation is shared by both arms. Mailbox providers judge the domain, not the experiment, so fewer complaints from the suppression arm can improve placement for the control arm and for active subscribers alike. The comparison therefore understates the benefit of suppression, and a result showing no difference in complaints may still hide a domain-wide gain. M3AAWG's sender best practices treat list hygiene as part of running a responsible sending operation, and the troubleshooting guide for emails going to spam explains how placement problems spread across a list.

Defining inactivity by opens alone is the second pitfall. Mail clients that fetch images automatically register opens nobody made, so an opens-only rule keeps people who never read and can suppress people who click from a client that blocks images. The third is consent records not updated after suppression. A suppressed subscriber must not be re-imported into a campaign by a later list upload, and their status should be recorded alongside their consent record, as the list building and consent guide describes. The ICO's direct marketing guidance is the reference for how UK consent and opt-outs are handled; suppression is a sending decision, not a withdrawal of the person's consent, and should be recorded that way.

Illustrative example — not a real result

A bookshop tests a sunset policy

Pebblewhistle Books, an invented online bookshop, has 18,000 subscribers who have not clicked, bought or replied in nine months. It splits them into two arms of 9,000. The suppression arm receives two re-engagement emails; 540 respond and stay, and 8,460 are suppressed. Over the following 16 weeks, revenue per assigned subscriber is £0.38 in the suppression arm and £0.44 in the control arm, with an interval that includes no difference. Spam complaints from the control arm run at several times the suppression arm's level, and the open and click rates of Pebblewhistle's active subscribers improve slightly over the period. The loss is concentrated among former buyers. Pebblewhistle adopts the sunset policy and moves former buyers in the inactive tier to one email a month instead.

Common mistakes

Sunset tests fail on their definitions and their windows. The email frequency test changes cadence for the whole list rather than deciding the fate of one segment, and the experiment library lists every design.

  • Defining inactivity by opens. Automatic image fetching makes opens unreliable.
  • Counting only email revenue. Suppressed subscribers may still buy through other channels.
  • Ending before occasional buyers buy. A short window shows no loss by accident.
  • Reading reputation effects as arm differences. The domain's reputation is shared.
  • Deleting instead of suppressing. The consent record and the way back are lost.
  • Running an underpowered test anyway. Use the complaint data and decide.

Frequently asked questions

Why not simply suppress inactive subscribers without a test?

Often that is the right call, and the deliverability case for it is well documented. The test is worth running when inactive subscribers still produce some revenue and the team disagrees about how much would be lost. If the inactive segment is too small to power the test, make the decision on the complaint data and the deliverability guidance instead.

Does the suppression arm's improvement show up in the control arm too?

Partly, and that is the honest limit of this design. Mailbox providers judge the sending domain as a whole, so fewer complaints from the suppression arm can help placement for everyone, including the control arm. The test therefore understates the reputation benefit, which is why the engagement of active subscribers is watched as a guardrail.

What counts as inactive?

Define it before the split, on signals that cannot be faked by mail clients: no clicks, purchases, site visits from email or replies within a chosen window. Opens alone are unreliable because some clients fetch images automatically, so an opens-only rule keeps people who never read anything and can drop people who do.

Terms used on this page

Frameworks

Where this page is used

Also related

Sources

  1. Email sender guidelines — Google Workspace Admin Help
  2. Messaging, Malware and Mobile Anti-Abuse Working Group — M3AAWG
  3. Direct marketing guidance — Information Commissioner's Office (UK)

Related