Hypothesis
If the abandoned cart email offers a discount instead of a plain reminder, recovered margin per abandoner will rise only if the extra orders outweigh the discount given to people who would have bought after the reminder alone. The decision is whether to include a discount in the abandoned cart email or send a plain reminder, and for which shoppers.
The hypothesis is written to make the cost visible. A discount is paid to every buyer in its arm, not only to the shoppers it persuaded. Some abandoners were comparing prices and needed a nudge; others were interrupted and would have returned once reminded. A discount recovers more orders from the first group and gives margin away to the second. The abandoned cart flows guide sets out the incentive policies a shop can choose and why a standing discount compounds; this test chooses between them with evidence. Record the arms, the discount, the margin assumptions and the segments you expect to differ in the experiment hypothesis template before the flow changes.
Metrics and guardrails
The primary metric is contribution margin per cart abandoner: revenue from orders placed by each arm's abandoners within the recovery window, minus cost of goods, the discount, payment fees, fulfilment and, where the window allows, returns, divided by the number of abandoners assigned to the arm. Every assigned abandoner counts, whether they opened anything or not. The contribution margin and break-even guide explains which costs belong in the margin, and the ROAS calculator walkthrough for discount promotions shows how a discount changes the margin that a return figure should be judged against.
Three guardrails catch the costs the primary metric cannot see in one window. Repeat abandonment among discount recipients, read over a further window after the test, shows whether shoppers are learning that leaving a basket produces a code. Discount cost per recovered order shows how much margin each extra order cost once the discount paid to people who would have bought anyway is counted. The unsubscribe rate per arm, read per assigned abandoner, catches an incentive message that annoys more than it persuades. Report the recovery rate and the Cart Abandonment RateShare of shopping carts created that did not become an order. of returning shoppers as diagnostics that explain the result, never as the result.
Design
Randomise at the level of the cart abandoner, assigned once when they first enter the flow and kept in the same arm for every later abandonment during the test. Three arms run side by side: the plain reminder, the same reminder with a discount, and a small Holdout GroupA randomly chosen part of an audience kept from exposure to measure what exposure caused. of eligible abandoners who receive no cart email at all. The emails are identical apart from the offer block: same timing, same subject line logic, same product content, same exit checks.
| Design element | Choice | Reason |
|---|---|---|
| Unit of randomisation | Cart abandoner, assigned once at flow entry | Shoppers abandon more than once; one arm per person |
| Arms | Plain reminder, reminder with discount, no-email holdout | Incentive against no incentive, with a reference for both |
| Held constant | Timing, subject logic, content, exits, browse flow | Isolates the offer |
| Discount codes | Unique and single-use per abandoner | Stops codes spreading to other arms |
| Primary metric | Contribution margin per assigned abandoner | Counts extra orders and margin given away |
| Holdout size | Smaller than the email arms | It is a reference, not the main comparison |
The holdout is not there to prove the flow works; the email metrics and testing guide covers flow holdouts and their sizing. Here it tells the reader how much of each arm's recovery was incrementalConversions caused by marketing. to email at all, which keeps a discount from taking credit for shoppers who were returning anyway. Leave the rest of the programme alone during the test: the guide to e-commerce email and SMS flows maps the welcome, browse and post-purchase flows that should behave identically in every arm. Every cart email remains marketing in most readings, so the entry rule still reads each shopper's consent; the ICO's direct marketing guidance and the FTC's CAN-SPAM compliance guide set out the general direction in the UK and the United States, and neither changes because a test is running.
Sample size reasoning
The metric is margin per abandoner, which mixes conversion and discount, and the holdout is the smallest arm, so size on the difference in margin that would change the decision and accept a longer run. Margin per abandoner is skewed: most abandoners buy nothing in the window and buyers differ widely in basket value, so the spread is wide and a modest gap between arms needs many abandoners to separate from chance. The unequal split adds to the demand, because the holdout's estimate is noisier than the email arms'. The sample size and significance guide explains how the baseline, the spread, the minimum effect and an unequal allocation combine. If the shop's flow volume cannot carry three arms, run the discount against the reminder with no holdout and say plainly that the reading cannot separate email's effect from shoppers returning on their own.
Duration
Run long enough for the planned entrants plus each entrant's full recovery window; repeat abandonment is read after a further window. Fix both windows before launch. Entrants who join on the last day of enrolment still need their whole recovery window before the primary metric is read, and the repeat-abandonment guardrail needs a second period after that. Keep the test away from a site-wide sale, which changes the value of a cart discount in every arm, or pause enrolment while the sale runs.
Reading the results and the decision rule
Read the primary metric once every entrant's recovery window has closed. Compare margin per abandoner between the discount and reminder arms with its interval, then read each against the holdout, then check the guardrails. Look at segments named before launch, such as first-time and returning shoppers, only after the pooled result. The rule, written before launch:
| Result | Decision |
|---|---|
| Discount arm's margin per abandoner beats the reminder and repeat abandonment does not rise | Keep the discount, for all or for the segments where it won |
| Discount arm recovers more orders but earns less margin per abandoner | Send the reminder; the extra orders cost more than they earn |
| Discount wins only among first-time shoppers | Reserve the offer for that segment and send everyone else the reminder |
| Discount wins in the window but repeat abandonment rises afterwards | Send the reminder; the shop is training abandonment |
Pitfalls
Customers learning to abandon for a code is the slowest and most expensive pitfall. A regular shopper who receives a discount once may leave the next basket on purpose, and that cost lands after the test window closes. Reading the repeat-abandonment guardrail later is the only defence. Codes shared beyond the arm are the second: a reusable code appears on coupon sites and is used by shoppers in the reminder arm and the holdout, so the test measures a public promotion. Unique single-use codes tied to the customer record prevent it. The third is reading recovered revenue instead of margin. Recovered revenue rises with the discount almost by definition; only margin per abandoner counts the discount paid to people who needed no discount. The free shipping threshold test faces the same trap with a site-wide offer and answers it the same way.
An outdoor shop tests a cart discount
Tarnbeck Outdoor, an invented outdoor gear shop, sends one cart reminder three hours after abandonment. It assigns 24,000 abandoners over eight weeks: 10,000 to the plain reminder, 10,000 to the reminder with a 10% code, and 4,000 to a no-email holdout. Within a 14-day window, 6.0% of the holdout, 8.5% of the reminder arm and 10.0% of the discount arm place an order. Contribution margin per assigned abandoner is £2.90 in the holdout, £4.10 in the reminder arm and £4.00 in the discount arm, whose extra orders do not cover the discount paid on the orders the reminder would have recovered. Among first-time shoppers, the discount arm leads by a margin the team agreed in advance was worth acting on. Tarnbeck sends the plain reminder to returning customers and the code to first-time shoppers only.
Common mistakes
Cart discount tests fail on their metric and their codes more often than on their statistics. The experiment library lists the related designs, including site-wide offers and email cadence.
- Judging on recovered revenue. It rises with any discount; use margin per abandoner.
- Counting only openers. Every assigned abandoner belongs in the denominator.
- Using one shared code. It leaks to other arms and to coupon sites.
- Reading before the recovery window closes. Late entrants have not finished deciding.
- Ignoring repeat abandonment. The training effect arrives after the window.
- Changing the flow's timing or content along with the offer. The test then measures two changes.
