Hypothesis
Setting the free shipping threshold just above the typical basket will raise contribution margin per visitor, because enough shoppers add items to qualify to outweigh the shipping cost absorbed. The mechanism is familiar: a shopper close to a threshold adds something to avoid paying for delivery. The uncertainty is in the balance. Some shoppers add an item; some abandon because the threshold is out of reach; some who would have paid for shipping now receive it free. Only a test reveals the net effect.
Ground the new threshold in your own order data. Look at the distribution of basket values and choose a level that a meaningful share of baskets sit just below. The average order value levers guide explains how thresholds interact with other levers such as bundles and recommendations, which should be left unchanged during the test. Write the current and proposed thresholds, the reasoning and the margin assumptions into the experiment hypothesis template.
Metrics and guardrails
The primary metric is contribution margin per visitor: revenue from each arm, minus cost of goods, shipping costs paid by the business, payment fees and, where the data allows, the cost of returns, all divided by visitors in that arm. It rewards a threshold that makes the business more money per visitor, whichever route it takes. The contribution margin and break-even guide explains how to calculate it, and the break-even calculator helps check the cost assumptions behind it.
Checkout conversion rate is the first guardrail. A threshold set too high can make shipping costs feel like a penalty and push shoppers to abandon; track each arm's CVRShare of visits or clicks that convert. with the conversion rate calculator. Return rateShare of orders or units sold that customers sent back. is the second, because basket padding that ends in returns raises AOVAverage revenue per order. while lowering real margin. Report average order value as a diagnostic that explains the result, never as the result.
Design
Randomise at visitor level and hold the assignment across sessions, so a shopper who browses on Monday and buys on Wednesday sees the same threshold both times. The control keeps the current threshold; the challenger uses the new one. Everything else stays identical: product prices, paid shipping charges below the threshold, promotions and the checkout flow. The checkout optimisation guide covers how shipping costs are best presented at checkout; use the same presentation in both arms.
| Design element | Choice | Reason |
|---|---|---|
| Unit of randomisation | Visitor, held across sessions | Shoppers return before buying |
| Variants | Current threshold versus threshold just above the typical basket | One offer change |
| Messaging | Each arm's threshold shown consistently on site, basket and checkout | Fair, clear offer |
| Held constant | Prices, paid shipping rates, promotions | Isolates the threshold |
| Duration | Whole weekly cycles until the planned sample is reached | Covers weekday and weekend shopping |
| Margin calculation | After returns where the window allows | Catches basket padding |
Messaging matters as much as the threshold. If a site-wide banner shows one threshold while the checkout applies another, the test is measuring confusion, and it may breach consumer expectations about clear pricing.
Sample size reasoning
Margin per visitor varies a lot from visitor to visitor: most visitors contribute nothing, and buyers contribute widely different amounts. That spread means a margin-based test needs more visitors than a test on conversion rate alone to detect the same relative change. The inputs are the current margin per visitor, how much it varies, the smallest improvement worth acting on and the confidence you need. The sample size and significance guide explains how variability enters the calculation.
If the required sample is beyond your traffic, test a larger change in threshold to increase the expected effect, or extend the test across more weeks. Do not swap the primary metric for average order value because it reaches significance sooner; it would answer an easier question than the one the business is asking.
Reading the results
Read once the planned sample is reached and, ideally, after the return window for orders in the test period has closed. Compare contribution margin per visitor between arms first. Then check checkout conversion rate and return rate.
Interpret the result through its components. A rise in margin per visitor driven by higher order values with steady conversion is the outcome the hypothesis predicted. A rise in order value with a fall in conversion can still be a win if margin per visitor rose, but it means some shoppers were priced out; check which segments abandoned. A fall in margin per visitor with higher order value means the threshold is costing more in absorbed shipping and lost buyers than it gains in bigger baskets. Segment by new and returning customers, who often respond differently to shipping offers.
A homeware shop raises its free delivery threshold
Linen and Loom, an invented homeware shop, offers free delivery over £40. Its typical basket is about £48. It tests a £55 threshold for five weeks with roughly 30,000 visitors per arm. Average order value rises from £52 to £58 in the challenger arm, and checkout conversion slips slightly. After cost of goods, absorbed shipping and returns, contribution margin per visitor is £1.62 in the challenger arm against £1.48 in the control. Return rate is unchanged. The shop adopts the new threshold and shows it clearly in the site header.
Pitfalls
Inconsistent messaging is the first pitfall. Banners, product pages, emails and checkout must all show the arm's threshold. A shopper told one thing on a product page and charged differently at checkout will abandon, and the business may face complaints about unclear pricing. General guidance from bodies such as the CMA expects delivery charges to be presented clearly; treat that as direction and seek advice for your market.
Basket padding with returns is the second. Some shoppers add a cheap item to qualify and return it later, which lifts order value but erodes margin. Calculate margin after returns and keep return rate as a guardrail. Promo overlap is the third: a site-wide discount during the test changes basket values in both arms and can push many baskets over either threshold, masking the difference. Schedule the test away from major promotions, or pause it while they run.
Common mistakes
Shipping offers touch every order, so a careless test affects the whole shop rather than one page. For subscription or software businesses the equivalent question is plan presentation, covered by the pricing page layout test, and the experiment library lists the rest.
- Judging on average order value. It rises almost by definition; margin per visitor is the real test.
- Mixed messages across the site. Every touchpoint must show the arm's threshold.
- Ignoring returns. Padded baskets that come back cost twice.
- Running through a sale. Promotions swamp the threshold effect.
- Choosing the threshold by guesswork. Use your own basket distribution.
- Forgetting absorbed shipping costs. Free shipping is a cost, and it belongs in the margin.
