Hypothesis
If recipients are randomly offered a deeper or a shallower discount, the deeper offer will lift orders, but gross profit per recipient will be higher for the shallower one unless the extra orders more than cover the margin given away. The decision is the depth of a planned promotion: a deeper discount, a shallower one, or none.
Depth is usually chosen by precedent: last year's sale, a competitor's banner, a round number that feels generous. The hypothesis names the trade that precedent hides. Every extra point of discount is paid on every order in the arm, including the orders that a shallower discount, or no discount, would have produced. A deeper offer that wins on orders can lose on profit, and the only way to know is to run the depths side by side on comparable customers. Record the depths, the margin assumptions, the audience and the follow-up window in the experiment hypothesis template before the promotion is built. The test assumes the promotion has already earned its place; the guide to planning promotions and discounts decides its job, mechanic and eligibility, and works out the break-even uplift each depth would need.
Metrics and guardrails
The primary metric is gross profit per recipient: revenue from each arm's recipients during the promotion, net of the discount, minus cost of goods and, once the return window has closed, the value of returns, divided by the number of recipients assigned to the arm. Every recipient counts, whether they bought or not, so an arm cannot look good by converting a few large orders. Where fulfilment and payment costs differ materially by order size, move to Contribution MarginRevenue left after every cost that varies with each sale. per recipient; the contribution margin and break-even guide explains which costs belong in each and how Gross MarginShare of revenue left after the direct cost of the goods sold. differs from contribution.
Three guardrails catch the costs that land outside the promotion. Full-price sales in the weeks after the promotion, per arm, show pull-forwardPurchases a promotion moves earlier in time rather than creates, as customers who would have bought later at full price buy during the offer.: customers who bought early at a discount instead of later at full price. Return rate per arm shows whether a deeper discount brought in purchases that come back. Orders from customers who usually pay full price, identified from purchase history before the test, show how much of each discount was paid to people who did not need it. Report AOVAverage revenue per order. and order counts as diagnostics that explain the result; the average order value levers guide explains why order value alone misleads when discounts are involved.
Design
Randomise at the level of the customer or subscriber who will receive the offer. Take the promotion's audience and split it at random into three arms: a deeper discount, a shallower discount, and a no-offer Holdout GroupA randomly chosen part of an audience kept from exposure to measure what exposure caused. that receives the same communication without a price cut, or no promotional message at all if that is the realistic alternative. Each recipient in a discount arm gets a unique, single-use code tied to their record. Everything else is identical: the products in scope, the creative, the send time, the channel and the promotion period.
| Design element | Choice | Reason |
|---|---|---|
| Unit of randomisation | Customer or subscriber receiving a unique offer | Offers are delivered to known people |
| Arms | Deeper discount, shallower discount, no offer | Compares depths and anchors both to the base |
| Codes | Unique and single-use, tied to the recipient | Stops codes leaking between arms |
| Held constant | Products, creative, channel, timing, promotion period | Only the depth differs |
| Primary metric | Gross profit per assigned recipient after returns | Counts extra orders and margin given away |
| Follow-up window | Full-price sales and returns after the promotion | Captures pull-forward and returns |
The ROAS calculator walkthrough for discount promotions shows how a discount changes the margin that a promotion's return on spend should be judged against, which is the arithmetic the team should agree before choosing which depths to test. Choose depths far enough apart that the difference matters to the business; two depths that differ trivially need an enormous sample to tell apart and would not change the plan either way.
Sample size reasoning
Profit per recipient is skewed and the no-offer arm anchors the comparison, so size on the profit difference that would change the promotion plan. Most recipients buy nothing during a promotion window, a few buy a lot, and the spread of profit per recipient is wide, so a modest difference between arms needs many recipients to separate from chance. Each extra arm divides the list further. The sample size and significance guide covers multi-arm tests and skewed metrics, including why comparing several arms needs a stricter standard for each comparison. If the audience is too small for three arms, drop to the shallower depth against no offer, which answers whether discounting this audience pays at all.
Duration
Run the promotion period plus a follow-up window long enough to see pull-forward in full-price sales and returns. Fix both before launch. The promotion period itself is set by the business calendar; the follow-up window should cover at least the store's return window and the usual gap between a customer's purchases, so that displaced full-price orders have time to fail to appear.
Reading the results and the decision rule
Read gross profit per recipient once the follow-up window has closed and returns are known. Compare each discount arm with the no-offer arm and with each other, with intervals, then read the guardrails. Read segments named before launch, such as habitual full-price buyers and lapsed customers, last. The rule, written before launch:
| Result | Decision |
|---|---|
| One depth earns the highest gross profit per recipient after returns and pull-forward | Run that depth for this audience |
| Both discounts earn less than the no-offer arm | Stop discounting this audience |
| A discount wins only among lapsed or new customers | Target the offer at that segment; send full-price customers the no-offer message |
| Deeper depth wins in the promotion but full-price sales fall afterwards | Treat as a loss; run the shallower depth or none |
Pitfalls
Codes leaking between arms is the first pitfall. A deep code posted on a deals site reaches the shallow and no-offer arms, and the test then compares a public promotion with itself. Unique single-use codes tied to the recipient prevent it, and any public promotion during the test should be postponed. A public promotion that runs without a split can still be judged afterwards against a modelled baseline, as the guide to measuring a sale or promotion explains, but with far more uncertainty than this test leaves.
Reading revenue rather than profit is the second. Revenue rises with depth almost by construction; only profit per recipient, after the discount, cost of goods and returns, shows whether the extra orders were worth what was given away. The third is reference prices that misstate the saving. A discount presented against a "was" price that was never genuinely charged misleads customers whatever the test shows. The pricing and packaging guide notes that the CMA enforces consumer protection law covering misleading pricing practices in the UK, and that the ASA's advertising codes cover savings claims and price comparisons; treat those as general direction and have promotions with reference pricing reviewed before they run. The free shipping threshold test faces the same presentation duty for delivery offers.
A pet supplies shop sets its spring promotion depth
Ravelstone Pet Supplies, an invented online pet supplies shop, plans a spring email promotion to 90,000 customers. It splits them into three arms of 30,000: 25% off, 10% off, and the same email with no discount. Over the promotion week and a six-week follow-up, gross profit per recipient after returns is £1.48 for 25% off, £1.71 for 10% off and £1.52 for no offer. The deeper arm placed the most orders, but full-price sales among its recipients fell in the following weeks, and a large share of its orders came from customers who usually pay full price. Among lapsed customers, 10% off beat no offer by a margin the team had agreed was worth acting on. Ravelstone runs 10% off for lapsed customers and sends everyone else the no-discount email.
Common mistakes
Depth tests fail on their metric and their window. The abandoned cart discount test asks the incentive question inside a flow rather than a promotion, and the experiment library lists every design.
- Judging on orders or revenue. Both rise with depth; profit per recipient decides.
- No no-offer arm. The test then cannot say whether discounting pays at all.
- Shared codes. They leak between arms and to deal sites.
- Stopping on the promotion's last day. Pull-forward and returns arrive later.
- Depths too close together. The difference is undetectable and would not change the plan.
- Inflated reference prices. A saving must be real, whatever the test shows.
