Hypothesis
If inactive subscribers receive a short re-engagement sequence and non-responders are suppressed, revenue from that group will fall only slightly while complaints fall, because the people who no longer want the emails stop receiving them. The decision is whether to re-engage and then suppress inactive subscribers, or to keep mailing them at the normal cadence.
The argument for suppression is about the list as a whole: mailbox providers weigh what recipients do, and a long tail of people who never interact dilutes the signal for everyone. The argument against is about money: some inactive subscribers still buy now and then, and a suppressed address cannot be sold to. The test puts a price on the second argument and watches the first. The segmentation guide sets out engagement tiersEmail subscribers grouped by how recently and often they interacted with mail, such as active, warm, cooling or inactive, with windows set by cadence. and how suppression differs from deletion; this design takes the inactive tier it defines and asks what suppressing it costs. Record the inactivity rule, the sequence and the window in the experiment hypothesis template before the split.
Metrics and guardrails
The primary metric is revenue per inactive subscriber over the test window: all revenue attributable to each arm's members, from email and, where the data allows, from any channel, divided by the number assigned to the arm at the split. Counting revenue from all channels matters here, because a suppressed subscriber who still buys through search or the app has not been lost, and an email-only view would overstate the cost of suppression.
Three guardrails complete the picture. The Spam Complaint RateShare of delivered emails that recipients marked as spam. per arm is the benefit side of the hypothesis: Google's email sender guidelines set requirements for bulk senders that include keeping reported spam low, and the deliverability guide explains how complaints shape placement for the whole list. The reactivation rate, the share of the suppression arm that responds to the sequence and becomes active again, shows how many people the sequence rescues. The third guardrail sits outside the test: the engagement of active subscribers, who are not randomised, is watched for sender-level effects, because a reputation change caused by either arm reaches them too. Read unsubscribesShare of delivered emails that led the recipient to opt out. per arm as a diagnostic; the sequence will usually prompt some, which is part of the point.
Design
Define inactivity first, on behaviour that mail clients cannot fake: no clicks, purchases, replies or visits from email within a chosen window. Take everyone who meets the rule on the start date and split them at random into two arms. The suppression arm receives a short re-engagement sequence asking for a positive signal, and everyone who does not give one is moved to the suppression list at the sequence's end. The control arm keeps receiving the normal cadence. Active subscribers are not part of either arm and their mail does not change.
| Design element | Choice | Reason |
|---|---|---|
| Unit of randomisation | Subscriber within the inactive segment | The decision applies to this segment only |
| Arms | Re-engage then suppress, against keep mailing | Compares the policy with the status quo |
| Inactivity rule | Clicks, purchases, replies or visits, not opens alone | Opens are unreliable signals |
| Held constant | Content and cadence for active subscribers; transactional mail for everyone | Isolates the policy |
| Primary metric | Revenue per assigned inactive subscriber, all channels where possible | Counts revenue lost and revenue that moves channel |
| Shared effect | Active-subscriber engagement watched, not randomised | Reputation is domain-wide |
SegmentationDividing a market or customer list into groups that need different offers or messages. decides what the test can say afterwards. Name in advance any sub-segments you expect to behave differently, such as former buyers against people who never bought, so that the decision rule can keep mailing one of them without hunting through the data for a winner.
Sample size reasoning
Inactive subscribers buy rarely, so revenue per subscriber is a small, skewed number and the test needs a large inactive segment or a long window. Most members of either arm will spend nothing, a few will buy, and the difference between arms will be hard to separate from chance unless the segment is big. The sample size and significance guide explains the trade-off between segment size, spread and window length. A test that cannot be powered should be replaced by a judgement call with the complaint data: if the segment is small, its revenue is small too, and the deliverability case is usually enough on its own.
Duration
Run the re-engagement sequence plus a window long enough for occasional buyers to purchase, typically several purchase cycles. Inactive subscribers who still buy do so rarely, and a short window will show the suppression arm losing nothing simply because nobody in either arm bought yet. Fix the window before launch, avoid starting just before the business's seasonal peak unless the window spans it fully, and keep the inactivity rule frozen for the whole period so that people do not drift between definitions.
Reading the results and the decision rule
Read once at the planned end. Compare revenue per inactive subscriber between arms with its interval, then read complaints and reactivation, then check active-subscriber engagement for any sender-level change. Read the pre-named sub-segments last. The rule, written before launch:
| Result | Decision |
|---|---|
| Suppression arm gives up little revenue and complaints fall | Adopt the sunset policy for the whole inactive segment |
| Suppression arm loses revenue concentrated in one pre-named sub-segment | Suppress the rest; keep mailing that sub-segment at a lower cadence |
| Suppression arm loses substantial revenue across the segment | Keep mailing, but tighten the inactivity rule and re-test |
| Test underpowered, no reliable difference | Decide on the complaint data and the deliverability guidance; suppression is the default |
Pitfalls
Sender reputation is shared by both arms. Mailbox providers judge the domain, not the experiment, so fewer complaints from the suppression arm can improve placement for the control arm and for active subscribers alike. The comparison therefore understates the benefit of suppression, and a result showing no difference in complaints may still hide a domain-wide gain. M3AAWG's sender best practices treat list hygiene as part of running a responsible sending operation, and the troubleshooting guide for emails going to spam explains how placement problems spread across a list.
Defining inactivity by opens alone is the second pitfall. Mail clients that fetch images automatically register opens nobody made, so an opens-only rule keeps people who never read and can suppress people who click from a client that blocks images. The third is consent records not updated after suppression. A suppressed subscriber must not be re-imported into a campaign by a later list upload, and their status should be recorded alongside their consent record, as the list building and consent guide describes. The ICO's direct marketing guidance is the reference for how UK consent and opt-outs are handled; suppression is a sending decision, not a withdrawal of the person's consent, and should be recorded that way.
A bookshop tests a sunset policy
Pebblewhistle Books, an invented online bookshop, has 18,000 subscribers who have not clicked, bought or replied in nine months. It splits them into two arms of 9,000. The suppression arm receives two re-engagement emails; 540 respond and stay, and 8,460 are suppressed. Over the following 16 weeks, revenue per assigned subscriber is £0.38 in the suppression arm and £0.44 in the control arm, with an interval that includes no difference. Spam complaints from the control arm run at several times the suppression arm's level, and the open and click rates of Pebblewhistle's active subscribers improve slightly over the period. The loss is concentrated among former buyers. Pebblewhistle adopts the sunset policy and moves former buyers in the inactive tier to one email a month instead.
Common mistakes
Sunset tests fail on their definitions and their windows. The email frequency test changes cadence for the whole list rather than deciding the fate of one segment, and the experiment library lists every design.
- Defining inactivity by opens. Automatic image fetching makes opens unreliable.
- Counting only email revenue. Suppressed subscribers may still buy through other channels.
- Ending before occasional buyers buy. A short window shows no loss by accident.
- Reading reputation effects as arm differences. The domain's reputation is shared.
- Deleting instead of suppressing. The consent record and the way back are lost.
- Running an underpowered test anyway. Use the complaint data and decide.
