Hypothesis
If a random group of subscribers receives campaign emails more often than the current cadence, revenue per subscriber over the test window will rise, because more sends create more buying occasions, unless extra unsubscribes and complaints erode the list faster than the sends earn. The decision behind it is concrete: whether to change the number of marketing emails per subscriber per week, and for whom.
The hypothesis names its own failure mode on purpose. Frequency is the one email decision where the benefit arrives quickly and the cost arrives slowly. Extra sends produce visible orders within days, while the subscribers who quietly stop reading, unsubscribe or report the mail as spam reduce what every later send can earn. A test that only looks at the first fortnight will nearly always favour sending more. Teams also run the hypothesis in reverse: if the list shows signs of fatigue, a lower-cadence cell tests whether fewer, better-timed sends keep revenue while easing complaints. Record the direction, the cells and the expected behaviour of each segment in the experiment hypothesis template before anything is scheduled.
Metrics and guardrails
The primary metric is revenue per subscriber over the whole test window: all revenue attributed to campaign emails from each cell's members, divided by the number of subscribers assigned to that cell at the start. The denominator is fixed at assignment, so people who unsubscribe still count, and their lost future value shows up as lower revenue per subscriber rather than disappearing from the arithmetic. Where revenue is not the outcome, use conversions per subscriber on the same basis. The email metrics and testing guide explains why Revenue per EmailEmail-attributed revenue divided by the number of emails delivered. and open rates mislead here: per-send figures fall as frequency rises even when the programme earns more, and opens are distorted by privacy features.
Three guardrails protect the list. The unsubscribe rate per cell, read per subscriber over the window rather than per send, shows how many people the cadence drives away. The Spam Complaint RateShare of delivered emails that recipients marked as spam. matters more than its size suggests, because mailbox providers use complaints to judge the sender: Google's email sender guidelines set requirements for bulk senders that include keeping reported spam low and making unsubscribing easy, and the deliverability guide explains how complaints affect inbox placement for the whole list. The third guardrail is the share of each cell still mailable at the end of the window, which combines unsubscribes, complaints and hard bounces into one view of what the cadence leaves behind.
Design
Randomise at subscriber level. Take the mailable list, or a defined segment of it, and split it at random into cells: the current cadence as control and one higher cadence, with an optional lower-cadence cell if fatigue is part of the question. Store the cell on each contact record so that every scheduled send respects it. All cells draw on one content calendar: the higher cadence receives every email the control receives plus additional sends from the same calendar, so the test changes how often people hear from you, not what they hear.
| Design element | Choice | Reason |
|---|---|---|
| Unit of randomisation | Subscriber, assigned once | Frequency effects build over weeks for the same person |
| Cells | Current cadence, a higher cadence, optionally a lower one | Each extra cell divides the list, so add only those you would act on |
| Content | One shared calendar; extra sends come from it | Isolates frequency from content quality |
| Automated flows | Identical in every cell, with a shared per-subscriber cap | Measures campaign frequency alone |
| Primary metric | Revenue per assigned subscriber over the window | Counts both extra sales and lost subscribers |
| Segments | Defined before launch, read after the pooled result | Prevents searching for a winning segment afterwards |
Decide the segments you expect to behave differently, typically recent buyers, engaged non-buyers and the least engaged, using the definitions in the segmentation guide. SegmentationDividing a market or customer list into groups that need different offers or messages. matters because the right answer is often different cadences for different groups, and the test should be able to show that without inventing segments after the result arrives.
Sample size reasoning
The metric is per subscriber, so the sample is people, not sends. Two features of revenue per subscriber make the test demanding. Most subscribers buy nothing in any given window, and a few buyers carry most of the revenue, so the metric has a wide spread and a modest difference between cells is hard to separate from chance. And every extra cell takes subscribers away from the others. The inputs are the control cell's expected revenue per subscriber, how widely it varies between people, the smallest difference that would justify changing the cadence and the confidence the team requires; the sample size and significance guide explains how they combine and why a skewed metric needs more people or a longer window than a conversion rate.
If the list is too small to detect a difference worth acting on, change the question rather than running the test anyway. Use conversions per subscriber, which is less skewed than revenue, test a larger change in cadence whose effect would be bigger, or run the test on the largest segment only.
Duration
Run for several weeks covering full purchase cycles, and decide the end date before launch. The early weeks of a higher cadence tend to show pulled-forward purchases: people who would have bought anyway buy sooner because an email arrived. The cost of fatigue appears later, as engagement falls and unsubscribes accumulate. A window that ends early captures the first effect and misses the second. If the business has a seasonal peak, either run the whole test outside it or make sure it spans the peak and the weeks after it, so that both cells experience the same calendar.
Reading the results and the decision rule
Read the test once, at the planned end. Compare revenue per assigned subscriber between cells, with the confidence intervalA range of values for an effect, such as a lift in conversion rate, that is consistent with the data at a chosen confidence level. rather than the point estimate, then check each guardrail. Only after the pooled result is settled, look at the segments named before launch. Write the decision rule into the plan in advance:
| Result | Decision |
|---|---|
| Higher cadence earns more per subscriber and guardrails stay within agreed limits | Move to the higher cadence, for all or for the segments where it won |
| Higher cadence earns more but complaints or unsubscribes break a guardrail | Keep the current cadence, or raise it only for engaged segments |
| No reliable difference in revenue per subscriber | Keep the current cadence; extra sends cost effort and list health for nothing |
| Lower cadence earns about the same with fewer complaints | Reduce cadence and reinvest the effort in content quality |
The lifecycle email programme playbook shows where the result goes next: into the programme's cadence rules, reviewed monthly, so that seasonal pushes stay inside a limit the test has justified.
An online tea retailer tests a second weekly email
Larchfield Teas, an invented online tea retailer, sends one campaign a week to 60,000 subscribers. It splits the list at random into two cells of 30,000: control keeps one email a week, the variant gets two from the same calendar. After ten weeks, the control earned £0.62 per assigned subscriber and the variant £0.71, a difference with a 95% interval of roughly £0.02 to £0.16. Unsubscribes over the window were 1.1% in control and 1.9% in the variant, and spam complaints stayed well inside the team's limit in both. The variant's lead came almost entirely from recent buyers. Larchfield moves recent buyers to two emails a week and keeps everyone else on one.
Pitfalls
Reading revenue per send is the most common way to get the answer wrong, because it falls with frequency whatever happens to total revenue. Pull-forwardPurchases a promotion moves earlier in time rather than creates, as customers who would have bought later at full price buy during the offer. is the second: a short window rewards the higher cadence for sales that would have happened anyway. The third is uneven automation. If one cell triggers more flows, perhaps because more emails mean more clicks into browse-abandonment journeys, the test measures campaign and flow volume together; a shared per-subscriber cap keeps the comparison honest.
Watch the guardrails while the test runs, but act only on harm, not on the primary metric. A complaint rate that climbs sharply in the higher cell is a reason to stop that cell, because the damage to sender reputation affects every subscriber, including control. The guide to an unsubscribe rate spike helps separate a frequency effect from a content or audience problem that happens to coincide with the test. Whatever cadence wins, every email must still carry a working unsubscribe link that is honoured promptly, as the FTC's CAN-SPAM compliance guide sets out for commercial email in the United States.
Common mistakes
Frequency tests fail more often on their window and metric than on their statistics. The email send time test holds the number of sends fixed and moves the time, which is the natural next test once cadence is settled, and the experiment library lists every design.
- Judging on revenue per email. It falls with frequency by construction; use revenue per subscriber.
- Dropping unsubscribers from the denominator. Their lost value is the cost being tested.
- Ending at the first good fortnight. Pull-forward flatters the higher cadence early.
- Changing content along with frequency. Extra sends must come from the same calendar.
- Letting flows differ between cells. Automation volume then contaminates the comparison.
- Hunting for a winning segment afterwards. Name the segments before launch and read them last.
