Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Experiment Library

Discount Depth Test

An experiment design for choosing a promotion's discount level: randomised offer codes, gross profit per recipient, and pull-forward and return guardrails.
Experimentadvanced7 min readUpdated

Hypothesis

If recipients are randomly offered a deeper or a shallower discount, the deeper offer will lift orders, but gross profit per recipient will be higher for the shallower one unless the extra orders more than cover the margin given away. The decision is the depth of a planned promotion: a deeper discount, a shallower one, or none.

Depth is usually chosen by precedent: last year's sale, a competitor's banner, a round number that feels generous. The hypothesis names the trade that precedent hides. Every extra point of discount is paid on every order in the arm, including the orders that a shallower discount, or no discount, would have produced. A deeper offer that wins on orders can lose on profit, and the only way to know is to run the depths side by side on comparable customers. Record the depths, the margin assumptions, the audience and the follow-up window in the experiment hypothesis template before the promotion is built. The test assumes the promotion has already earned its place; the guide to planning promotions and discounts decides its job, mechanic and eligibility, and works out the break-even uplift each depth would need.

Metrics and guardrails

The primary metric is gross profit per recipient: revenue from each arm's recipients during the promotion, net of the discount, minus cost of goods and, once the return window has closed, the value of returns, divided by the number of recipients assigned to the arm. Every recipient counts, whether they bought or not, so an arm cannot look good by converting a few large orders. Where fulfilment and payment costs differ materially by order size, move to Contribution Margin per recipient; the contribution margin and break-even guide explains which costs belong in each and how Gross Margin differs from contribution.

Three guardrails catch the costs that land outside the promotion. Full-price sales in the weeks after the promotion, per arm, show pull-forward: customers who bought early at a discount instead of later at full price. Return rate per arm shows whether a deeper discount brought in purchases that come back. Orders from customers who usually pay full price, identified from purchase history before the test, show how much of each discount was paid to people who did not need it. Report AOV and order counts as diagnostics that explain the result; the average order value levers guide explains why order value alone misleads when discounts are involved.

Design

Randomise at the level of the customer or subscriber who will receive the offer. Take the promotion's audience and split it at random into three arms: a deeper discount, a shallower discount, and a no-offer Holdout Group that receives the same communication without a price cut, or no promotional message at all if that is the realistic alternative. Each recipient in a discount arm gets a unique, single-use code tied to their record. Everything else is identical: the products in scope, the creative, the send time, the channel and the promotion period.

Design elementChoiceReason
Unit of randomisationCustomer or subscriber receiving a unique offerOffers are delivered to known people
ArmsDeeper discount, shallower discount, no offerCompares depths and anchors both to the base
CodesUnique and single-use, tied to the recipientStops codes leaking between arms
Held constantProducts, creative, channel, timing, promotion periodOnly the depth differs
Primary metricGross profit per assigned recipient after returnsCounts extra orders and margin given away
Follow-up windowFull-price sales and returns after the promotionCaptures pull-forward and returns

The ROAS calculator walkthrough for discount promotions shows how a discount changes the margin that a promotion's return on spend should be judged against, which is the arithmetic the team should agree before choosing which depths to test. Choose depths far enough apart that the difference matters to the business; two depths that differ trivially need an enormous sample to tell apart and would not change the plan either way.

Sample size reasoning

Profit per recipient is skewed and the no-offer arm anchors the comparison, so size on the profit difference that would change the promotion plan. Most recipients buy nothing during a promotion window, a few buy a lot, and the spread of profit per recipient is wide, so a modest difference between arms needs many recipients to separate from chance. Each extra arm divides the list further. The sample size and significance guide covers multi-arm tests and skewed metrics, including why comparing several arms needs a stricter standard for each comparison. If the audience is too small for three arms, drop to the shallower depth against no offer, which answers whether discounting this audience pays at all.

Duration

Run the promotion period plus a follow-up window long enough to see pull-forward in full-price sales and returns. Fix both before launch. The promotion period itself is set by the business calendar; the follow-up window should cover at least the store's return window and the usual gap between a customer's purchases, so that displaced full-price orders have time to fail to appear.

Reading the results and the decision rule

Read gross profit per recipient once the follow-up window has closed and returns are known. Compare each discount arm with the no-offer arm and with each other, with intervals, then read the guardrails. Read segments named before launch, such as habitual full-price buyers and lapsed customers, last. The rule, written before launch:

ResultDecision
One depth earns the highest gross profit per recipient after returns and pull-forwardRun that depth for this audience
Both discounts earn less than the no-offer armStop discounting this audience
A discount wins only among lapsed or new customersTarget the offer at that segment; send full-price customers the no-offer message
Deeper depth wins in the promotion but full-price sales fall afterwardsTreat as a loss; run the shallower depth or none

Pitfalls

Codes leaking between arms is the first pitfall. A deep code posted on a deals site reaches the shallow and no-offer arms, and the test then compares a public promotion with itself. Unique single-use codes tied to the recipient prevent it, and any public promotion during the test should be postponed. A public promotion that runs without a split can still be judged afterwards against a modelled baseline, as the guide to measuring a sale or promotion explains, but with far more uncertainty than this test leaves.

Reading revenue rather than profit is the second. Revenue rises with depth almost by construction; only profit per recipient, after the discount, cost of goods and returns, shows whether the extra orders were worth what was given away. The third is reference prices that misstate the saving. A discount presented against a "was" price that was never genuinely charged misleads customers whatever the test shows. The pricing and packaging guide notes that the CMA enforces consumer protection law covering misleading pricing practices in the UK, and that the ASA's advertising codes cover savings claims and price comparisons; treat those as general direction and have promotions with reference pricing reviewed before they run. The free shipping threshold test faces the same presentation duty for delivery offers.

Illustrative example — not a real result

A pet supplies shop sets its spring promotion depth

Ravelstone Pet Supplies, an invented online pet supplies shop, plans a spring email promotion to 90,000 customers. It splits them into three arms of 30,000: 25% off, 10% off, and the same email with no discount. Over the promotion week and a six-week follow-up, gross profit per recipient after returns is £1.48 for 25% off, £1.71 for 10% off and £1.52 for no offer. The deeper arm placed the most orders, but full-price sales among its recipients fell in the following weeks, and a large share of its orders came from customers who usually pay full price. Among lapsed customers, 10% off beat no offer by a margin the team had agreed was worth acting on. Ravelstone runs 10% off for lapsed customers and sends everyone else the no-discount email.

Common mistakes

Depth tests fail on their metric and their window. The abandoned cart discount test asks the incentive question inside a flow rather than a promotion, and the experiment library lists every design.

  • Judging on orders or revenue. Both rise with depth; profit per recipient decides.
  • No no-offer arm. The test then cannot say whether discounting pays at all.
  • Shared codes. They leak between arms and to deal sites.
  • Stopping on the promotion's last day. Pull-forward and returns arrive later.
  • Depths too close together. The difference is undetectable and would not change the plan.
  • Inflated reference prices. A saving must be real, whatever the test shows.

Frequently asked questions

Why include a no-offer arm?

It anchors the comparison. Without it, the test can only say which discount earned more, not whether either earned more than sending nothing, or than a message with no price cut. The no-offer arm shows what recipients would have bought anyway, which is the base every discount is paying to improve on.

Is it fair to offer different customers different discounts?

Targeted offers are a routine part of marketing, but each recipient should see one clear, honest offer, and any saving claimed must be real. Consumer protection and advertising bodies such as the CMA and the ASA expect price claims to be accurate; take advice for your market on how the promotion is presented, since this page is not legal advice.

Why read profit after the promotion has ended?

Because a discount moves purchases in time. Customers who would have bought at full price next month buy now at a lower price, and that cost appears only in the weeks after the promotion, as weaker full-price sales. A reading that stops on the promotion's last day counts the extra orders and misses what they displaced.

Terms used on this page

Tools to try

Frameworks

Where this page is used

Also related

Sources

  1. Competition and Markets Authority — GOV.UK
  2. Advertising Standards Authority — ASA

Related