Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Experiment Library

Abandoned Cart Discount Test

A test plan for whether an abandoned cart email should carry a discount, judged on margin per abandoner with a no-email holdout as the reference.
Experimentintermediate7 min readUpdated

Hypothesis

If the abandoned cart email offers a discount instead of a plain reminder, recovered margin per abandoner will rise only if the extra orders outweigh the discount given to people who would have bought after the reminder alone. The decision is whether to include a discount in the abandoned cart email or send a plain reminder, and for which shoppers.

The hypothesis is written to make the cost visible. A discount is paid to every buyer in its arm, not only to the shoppers it persuaded. Some abandoners were comparing prices and needed a nudge; others were interrupted and would have returned once reminded. A discount recovers more orders from the first group and gives margin away to the second. The abandoned cart flows guide sets out the incentive policies a shop can choose and why a standing discount compounds; this test chooses between them with evidence. Record the arms, the discount, the margin assumptions and the segments you expect to differ in the experiment hypothesis template before the flow changes.

Metrics and guardrails

The primary metric is contribution margin per cart abandoner: revenue from orders placed by each arm's abandoners within the recovery window, minus cost of goods, the discount, payment fees, fulfilment and, where the window allows, returns, divided by the number of abandoners assigned to the arm. Every assigned abandoner counts, whether they opened anything or not. The contribution margin and break-even guide explains which costs belong in the margin, and the ROAS calculator walkthrough for discount promotions shows how a discount changes the margin that a return figure should be judged against.

Three guardrails catch the costs the primary metric cannot see in one window. Repeat abandonment among discount recipients, read over a further window after the test, shows whether shoppers are learning that leaving a basket produces a code. Discount cost per recovered order shows how much margin each extra order cost once the discount paid to people who would have bought anyway is counted. The unsubscribe rate per arm, read per assigned abandoner, catches an incentive message that annoys more than it persuades. Report the recovery rate and the Cart Abandonment Rate of returning shoppers as diagnostics that explain the result, never as the result.

Design

Randomise at the level of the cart abandoner, assigned once when they first enter the flow and kept in the same arm for every later abandonment during the test. Three arms run side by side: the plain reminder, the same reminder with a discount, and a small Holdout Group of eligible abandoners who receive no cart email at all. The emails are identical apart from the offer block: same timing, same subject line logic, same product content, same exit checks.

Design elementChoiceReason
Unit of randomisationCart abandoner, assigned once at flow entryShoppers abandon more than once; one arm per person
ArmsPlain reminder, reminder with discount, no-email holdoutIncentive against no incentive, with a reference for both
Held constantTiming, subject logic, content, exits, browse flowIsolates the offer
Discount codesUnique and single-use per abandonerStops codes spreading to other arms
Primary metricContribution margin per assigned abandonerCounts extra orders and margin given away
Holdout sizeSmaller than the email armsIt is a reference, not the main comparison

The holdout is not there to prove the flow works; the email metrics and testing guide covers flow holdouts and their sizing. Here it tells the reader how much of each arm's recovery was incremental to email at all, which keeps a discount from taking credit for shoppers who were returning anyway. Leave the rest of the programme alone during the test: the guide to e-commerce email and SMS flows maps the welcome, browse and post-purchase flows that should behave identically in every arm. Every cart email remains marketing in most readings, so the entry rule still reads each shopper's consent; the ICO's direct marketing guidance and the FTC's CAN-SPAM compliance guide set out the general direction in the UK and the United States, and neither changes because a test is running.

Sample size reasoning

The metric is margin per abandoner, which mixes conversion and discount, and the holdout is the smallest arm, so size on the difference in margin that would change the decision and accept a longer run. Margin per abandoner is skewed: most abandoners buy nothing in the window and buyers differ widely in basket value, so the spread is wide and a modest gap between arms needs many abandoners to separate from chance. The unequal split adds to the demand, because the holdout's estimate is noisier than the email arms'. The sample size and significance guide explains how the baseline, the spread, the minimum effect and an unequal allocation combine. If the shop's flow volume cannot carry three arms, run the discount against the reminder with no holdout and say plainly that the reading cannot separate email's effect from shoppers returning on their own.

Duration

Run long enough for the planned entrants plus each entrant's full recovery window; repeat abandonment is read after a further window. Fix both windows before launch. Entrants who join on the last day of enrolment still need their whole recovery window before the primary metric is read, and the repeat-abandonment guardrail needs a second period after that. Keep the test away from a site-wide sale, which changes the value of a cart discount in every arm, or pause enrolment while the sale runs.

Reading the results and the decision rule

Read the primary metric once every entrant's recovery window has closed. Compare margin per abandoner between the discount and reminder arms with its interval, then read each against the holdout, then check the guardrails. Look at segments named before launch, such as first-time and returning shoppers, only after the pooled result. The rule, written before launch:

ResultDecision
Discount arm's margin per abandoner beats the reminder and repeat abandonment does not riseKeep the discount, for all or for the segments where it won
Discount arm recovers more orders but earns less margin per abandonerSend the reminder; the extra orders cost more than they earn
Discount wins only among first-time shoppersReserve the offer for that segment and send everyone else the reminder
Discount wins in the window but repeat abandonment rises afterwardsSend the reminder; the shop is training abandonment

Pitfalls

Customers learning to abandon for a code is the slowest and most expensive pitfall. A regular shopper who receives a discount once may leave the next basket on purpose, and that cost lands after the test window closes. Reading the repeat-abandonment guardrail later is the only defence. Codes shared beyond the arm are the second: a reusable code appears on coupon sites and is used by shoppers in the reminder arm and the holdout, so the test measures a public promotion. Unique single-use codes tied to the customer record prevent it. The third is reading recovered revenue instead of margin. Recovered revenue rises with the discount almost by definition; only margin per abandoner counts the discount paid to people who needed no discount. The free shipping threshold test faces the same trap with a site-wide offer and answers it the same way.

Illustrative example — not a real result

An outdoor shop tests a cart discount

Tarnbeck Outdoor, an invented outdoor gear shop, sends one cart reminder three hours after abandonment. It assigns 24,000 abandoners over eight weeks: 10,000 to the plain reminder, 10,000 to the reminder with a 10% code, and 4,000 to a no-email holdout. Within a 14-day window, 6.0% of the holdout, 8.5% of the reminder arm and 10.0% of the discount arm place an order. Contribution margin per assigned abandoner is £2.90 in the holdout, £4.10 in the reminder arm and £4.00 in the discount arm, whose extra orders do not cover the discount paid on the orders the reminder would have recovered. Among first-time shoppers, the discount arm leads by a margin the team agreed in advance was worth acting on. Tarnbeck sends the plain reminder to returning customers and the code to first-time shoppers only.

Common mistakes

Cart discount tests fail on their metric and their codes more often than on their statistics. The experiment library lists the related designs, including site-wide offers and email cadence.

  • Judging on recovered revenue. It rises with any discount; use margin per abandoner.
  • Counting only openers. Every assigned abandoner belongs in the denominator.
  • Using one shared code. It leaks to other arms and to coupon sites.
  • Reading before the recovery window closes. Late entrants have not finished deciding.
  • Ignoring repeat abandonment. The training effect arrives after the window.
  • Changing the flow's timing or content along with the offer. The test then measures two changes.

Frequently asked questions

Why judge the test on margin rather than recovered revenue?

A discount almost always recovers more orders, so recovered revenue flatters it by construction. Margin per abandoner subtracts the discount given to every buyer in the arm, including the many who would have bought after a plain reminder, which is the cost the decision turns on.

Why include a no-email holdout if the question is discount or no discount?

The holdout is the reference point. It shows how many abandoners come back with no email at all, so the reading can say how much each email arm adds over nothing. Without it, a discount that mostly rewards people already on their way back looks like a recovery engine.

Can the discount arm use one shared code?

It should not. A shared code travels to coupon sites and to shoppers in the other arms, and the test then measures a site-wide offer. Issue a unique, single-use code to each abandoner in the discount arm and tie it to the customer record.

Terms used on this page

Tools to try

Frameworks

Where this page is used

Also related

Sources

  1. CAN-SPAM Act: A Compliance Guide for Business — Federal Trade Commission
  2. Direct marketing guidance — Information Commissioner's Office (UK)

Related