Hypothesis
If a random group of customers receives coded offers only at planned events instead of the current rhythm, gross profit per customer over several promotion cycles will hold or rise, because fewer orders are discounted and full-price buying recovers between events, unless the orders the missing offers would have won outweigh the margin kept. The decision behind it is concrete: keep sending personal coded offers at the current rhythm, or send them only at a few planned events and protect full-price trading in between.
The guide to planning promotions and discounts explains why frequent promotions teach customers that the full price is a starting point, and its remedy is fewer promotions, each with a job, with stretches of full-price trading deliberately protected. The page on customers only buying in the sale measures the habit from order data and names the routes that taught it. Neither compares a reduced rhythm with the current one on matched customers, and that is the gap this design fills. A team that has read the signs still faces a fair objection: the offers bring in orders, and nobody knows how many of them would arrive without a code. The test answers that objection with the business's own customers.
Write the hypothesis, the planned events, the codes in scope and the limit on each guardrail into the experiment hypothesis template before the first send. The experimentation loop puts this step before any build work, and its application to retention tests explains why a change whose effect builds over months needs a window fixed in advance rather than a quick read.
Metrics and guardrails
The primary metric is gross profit per customer over a window spanning several promotion cycles. For each cell, take all orders from its customers across every channel the store can match to them: revenue net of every discount, less cost of goods, less the value of returns once their window has closed. Divide by the number of customers assigned to the cell at the start, whether they bought or not. The fixed denominator matters twice. A customer who stops buying in the reduced cell still counts, so lost customers show up as lower profit rather than vanishing from the arithmetic. And the metric counts every order, coded or not, so the cell that discounts less gets credit for the full-price orders it protects. The difference between revenue and Gross MarginShare of revenue left after the direct cost of the goods sold. is the point of the test; a reading on revenue alone would favour whichever cell discounts more.
Three guardrails catch the costs the primary metric could hide:
- Customers still active at the end of the window. Use the store's own definition of an active customer, set from how often its customers normally buy. Fewer offers could keep profit level for a while as loyal customers keep buying, while occasional buyers drift away; this guardrail, read by CohortA group of customers who share a starting event and period, tracked together over time. of first purchase, shows whether that is happening.
- Unsubscribes from marketing messages. Some customers subscribed for the codes. If the reduced cell loses them faster, the business loses a channel for future messages even if profit holds.
- Orders through public sitewide sales, by cell. Customers who stop receiving personal codes may wait for the public events instead. A rising share of the reduced cell's orders in public sales says the habit has moved rather than eased.
Report the share of orders placed with a code and the Repeat Purchase RateShare of customers who placed more than one order in the period. at full price as diagnostics; they explain the result but do not decide it.
Design
Randomise at customer level, once. Take the customers reachable on owned channels, such as email, text messages and the app, and assign each at random to a cell before the window opens. Store the cell on the customer record, so that every campaign, flow and loyalty mechanic that issues a personal code checks it. The control cell keeps the current offer rhythm. The reduced cell receives coded offers only at the planned events written into the plan. On the dates when the control receives a coded offer, the reduced cell receives nothing in its place.
| Design element | Choice | Reason |
|---|---|---|
| Unit of randomisation | Customer reachable on owned channels, assigned once | The habit belongs to the person and builds over cycles |
| Cells | Current offer rhythm; coded offers at planned events only | Two cells keep the base large enough for a skewed metric |
| What differs | Personal coded offers only | Isolates the offers from everything else the store does |
| Held constant | Public sales, prices, and every message without an offer | Public prices cannot be withheld from anyone, and other messages must not change frequency |
| Codes in scope | Listed before launch: campaign codes, and any flow or loyalty codes the plan includes | A code route left out keeps teaching the reduced cell to wait |
| Primary metric | Gross profit per assigned customer over the window | Counts protected full-price orders and lost customers alike |
The reduced cell is not a full Holdout GroupA randomly chosen part of an audience kept from exposure to measure what exposure caused.. It still receives the planned events, which is the rhythm the business would adopt, so the test compares two policies the business could run rather than asking whether offers work at all. Choose the planned events from the campaign calendar, with a job for each, and give the control cell the same events inside its usual rhythm, so that both cells meet every event the plan names.
List every route by which a personal code reaches a customer before launch. Campaign offers are the obvious one. The troubleshooting page also names codes in abandoned-cart emails and loyalty rewards that work as standing discounts. The narrowest version of this test changes campaign offers only and leaves automated codes alone; a broader version removes them from the reduced cell too. Either is valid, but the plan must say which, because a test that changes campaign codes while the cart email keeps handing out discounts measures less than the team thinks.
Every offer that does go out, in either cell, follows the same price rules. The planning guide records that the Competition and Markets Authority enforces consumer protection law in the UK covering reference prices that were never genuinely charged and urgency claims that are not true, and its working rules follow: a "was" price must have been a real price, and a deadline must be real. This is general direction rather than legal advice, and the same checks apply in both cells.
Sample size reasoning
Gross profit per customer is skewed. Most customers buy little or nothing in a given cycle, and a few buy a great deal, so the spread is wide and a modest difference between cells is hard to separate from chance. The effect also builds slowly, because a habit learned over many cycles eases over several. Together these mean the test needs a large customer base and patience. Name the smallest difference in gross profit per customer that would justify changing the promotion calendar, and use the sample size and significance guide, whose advice on skewed metrics applies directly, to judge whether the base can detect it.
If the base is too small, narrow the question rather than running an underpowered test. Test only the segment that receives the most offers, such as customers who redeem codes on most of their orders, where the effect should be largest and the decision most valuable. The ROAS calculator is useful alongside: treat the margin given away in codes as the spend and the gross profit it brought as the return, and compare that return between cells at the end of the window.
Duration
Run for several promotion cycles, with a start and an end fixed in advance so that both cells pass through the same sale events. A window that ends just before a large public sale, or just after it, reads a different moment of the habit in each cell. After the last cycle, hold a short cool-down before the final reading, because the weeks after a promotion carry Demand Pull-ForwardPurchases a promotion moves earlier in time rather than creates, as customers who would have bought later at full price buy during the offer. and the weeks before an expected one carry held-back demand. The guide to measuring a sale or promotion explains both effects; this test needs the cool-down so that neither is left inside one cell's window and outside the other's.
Reading the results and the decision rule
Read the test once, at the end of the cool-down. Compare gross profit per customer between cells, with the interval around the difference rather than the point estimate alone. Then check the guardrails by cell against the limits set before launch: active customers, unsubscribes and the share of orders through public sales. Only after that, read any segment named before launch. Write the decision rule into the plan in advance:
| Result | Decision |
|---|---|
| Gross profit per customer is the same or higher in the reduced cell, and active customers and unsubscribes stay within their limits | Move to fewer coded offers |
| Gross profit per customer falls reliably in the reduced cell | Keep the current rhythm |
| Results differ by segment, and the segments were named before launch | Extend the reduced rhythm only to the segments that support it |
| Profit holds but a guardrail breaks its limit | Keep the current rhythm for the affected group and retest a narrower change |
A result in favour of the reduced rhythm is a result for this business's customers and this calendar. It does not prove that any particular number of offers is right, and the next question, how deep the remaining offers should be, is the discount depth test. The academy checkpoint on peaks, promotion close-downs and full-price protection tests how a fewer-offers test is randomised, measured and read, beside decision rights in a peak and the order of a promotion close-down.
Pitfalls
Public sales and shared codes reaching the reduced cell. Public sales stay open to everyone by design, so read them as a guardrail rather than trying to hide them. Shared codes are a different matter: a code posted on a deal site reaches customers in the reduced cell and quietly gives them the control's rhythm. Issue unique codes tied to the customer wherever possible, and record any redemption of a code outside the cell it was issued to.
Reading after one cycle. The first cycle without an offer will look worse for the reduced cell, because customers who expected a code have not yet adjusted. The habit took many cycles to learn and takes more than one to ease; read only at the end of the planned window.
Offers swapped for more emails. If the reduced cell receives an extra message on every date the control gets an offer, the test now changes email frequency as well, which is the question of the email frequency test. Send nothing in place of a withheld offer, or keep any replacement message in both cells.
Segment fishing. With many segments and a skewed metric, one segment will look like a winner by chance. Name the segments before launch, read the pooled result first, and treat any other segment as a lead for the next test.
The general traps, such as checking that each cell received the share of customers the plan intended, are covered in the guide to common testing pitfalls.
Example: an invented footwear shop tests fewer coded offers (illustrative)
Frunkasse Footwear tests sending coded offers only at planned events
Frunkasse Footwear, an invented online footwear shop, sends a personal coded offer to its customers every three weeks, alongside two public sales a year. It takes 40,000 customers reachable by email and app and splits them at random into two cells of 20,000. The control keeps the three-week rhythm. The reduced cell receives coded offers only at two planned events: a spring launch and an autumn customer evening. Abandoned-cart codes are removed from the reduced cell too, and the public sales run for everyone. The window covers six offer cycles, with a four-week cool-down before the reading, and the plan sets the active-customer limit at no more than two percentage points below the control.
| Cell | Coded orders as share of orders | Gross profit per customer | Active at end of window | Unsubscribes |
|---|---|---|---|---|
| Current rhythm | 54% | £41.20 | 63% | 3.1% |
| Planned events only | 29% | £44.60 | 62% | 2.7% |
The reduced cell earns £3.40 more gross profit per customer, with an interval of roughly £0.90 to £5.90, while active customers sit one point below the control, inside the limit. Orders in the public sales rise slightly in the reduced cell. Frunkasse moves every customer to coded offers at planned events only and plans a later test of how deep the event offers should be.
Common mistakes
Offer-rhythm tests fail on their window and their scope more often than on their statistics. The experiment library lists the designs that sit beside this one.
- Measuring revenue instead of gross profit. The cell that discounts more will nearly always win on revenue.
- Dividing by buyers instead of assigned customers. Customers who stop buying are part of the cost of the change.
- Leaving code routes out of scope by accident. A cart code or loyalty reward keeps the habit alive in the reduced cell.
- Stopping at the first slow cycle. The habit eases over several cycles, not one.
- Replacing every withheld offer with another message. The test then measures frequency, not offers.
- Applying the result to acquisition offers. This design reads existing customers; first-order offers have a different job and a different test.
