Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Experiment Library

Free Shipping Threshold Test

A test design for a free shipping threshold judged on contribution margin per visitor, with conversion and returns guardrails, design and reading results.
Experimentintermediate5 min readUpdated

Hypothesis

Setting the free shipping threshold just above the typical basket will raise contribution margin per visitor, because enough shoppers add items to qualify to outweigh the shipping cost absorbed. The mechanism is familiar: a shopper close to a threshold adds something to avoid paying for delivery. The uncertainty is in the balance. Some shoppers add an item; some abandon because the threshold is out of reach; some who would have paid for shipping now receive it free. Only a test reveals the net effect.

Ground the new threshold in your own order data. Look at the distribution of basket values and choose a level that a meaningful share of baskets sit just below. The average order value levers guide explains how thresholds interact with other levers such as bundles and recommendations, which should be left unchanged during the test. Write the current and proposed thresholds, the reasoning and the margin assumptions into the experiment hypothesis template.

Metrics and guardrails

The primary metric is contribution margin per visitor: revenue from each arm, minus cost of goods, shipping costs paid by the business, payment fees and, where the data allows, the cost of returns, all divided by visitors in that arm. It rewards a threshold that makes the business more money per visitor, whichever route it takes. The contribution margin and break-even guide explains how to calculate it, and the break-even calculator helps check the cost assumptions behind it.

Checkout conversion rate is the first guardrail. A threshold set too high can make shipping costs feel like a penalty and push shoppers to abandon; track each arm's CVR with the conversion rate calculator. Return rate is the second, because basket padding that ends in returns raises AOV while lowering real margin. Report average order value as a diagnostic that explains the result, never as the result.

Design

Randomise at visitor level and hold the assignment across sessions, so a shopper who browses on Monday and buys on Wednesday sees the same threshold both times. The control keeps the current threshold; the challenger uses the new one. Everything else stays identical: product prices, paid shipping charges below the threshold, promotions and the checkout flow. The checkout optimisation guide covers how shipping costs are best presented at checkout; use the same presentation in both arms.

Design elementChoiceReason
Unit of randomisationVisitor, held across sessionsShoppers return before buying
VariantsCurrent threshold versus threshold just above the typical basketOne offer change
MessagingEach arm's threshold shown consistently on site, basket and checkoutFair, clear offer
Held constantPrices, paid shipping rates, promotionsIsolates the threshold
DurationWhole weekly cycles until the planned sample is reachedCovers weekday and weekend shopping
Margin calculationAfter returns where the window allowsCatches basket padding

Messaging matters as much as the threshold. If a site-wide banner shows one threshold while the checkout applies another, the test is measuring confusion, and it may breach consumer expectations about clear pricing.

Sample size reasoning

Margin per visitor varies a lot from visitor to visitor: most visitors contribute nothing, and buyers contribute widely different amounts. That spread means a margin-based test needs more visitors than a test on conversion rate alone to detect the same relative change. The inputs are the current margin per visitor, how much it varies, the smallest improvement worth acting on and the confidence you need. The sample size and significance guide explains how variability enters the calculation.

If the required sample is beyond your traffic, test a larger change in threshold to increase the expected effect, or extend the test across more weeks. Do not swap the primary metric for average order value because it reaches significance sooner; it would answer an easier question than the one the business is asking.

Reading the results

Read once the planned sample is reached and, ideally, after the return window for orders in the test period has closed. Compare contribution margin per visitor between arms first. Then check checkout conversion rate and return rate.

Interpret the result through its components. A rise in margin per visitor driven by higher order values with steady conversion is the outcome the hypothesis predicted. A rise in order value with a fall in conversion can still be a win if margin per visitor rose, but it means some shoppers were priced out; check which segments abandoned. A fall in margin per visitor with higher order value means the threshold is costing more in absorbed shipping and lost buyers than it gains in bigger baskets. Segment by new and returning customers, who often respond differently to shipping offers.

Illustrative example — not a real result

A homeware shop raises its free delivery threshold

Linen and Loom, an invented homeware shop, offers free delivery over £40. Its typical basket is about £48. It tests a £55 threshold for five weeks with roughly 30,000 visitors per arm. Average order value rises from £52 to £58 in the challenger arm, and checkout conversion slips slightly. After cost of goods, absorbed shipping and returns, contribution margin per visitor is £1.62 in the challenger arm against £1.48 in the control. Return rate is unchanged. The shop adopts the new threshold and shows it clearly in the site header.

Pitfalls

Inconsistent messaging is the first pitfall. Banners, product pages, emails and checkout must all show the arm's threshold. A shopper told one thing on a product page and charged differently at checkout will abandon, and the business may face complaints about unclear pricing. General guidance from bodies such as the CMA expects delivery charges to be presented clearly; treat that as direction and seek advice for your market.

Basket padding with returns is the second. Some shoppers add a cheap item to qualify and return it later, which lifts order value but erodes margin. Calculate margin after returns and keep return rate as a guardrail. Promo overlap is the third: a site-wide discount during the test changes basket values in both arms and can push many baskets over either threshold, masking the difference. Schedule the test away from major promotions, or pause it while they run.

Common mistakes

Shipping offers touch every order, so a careless test affects the whole shop rather than one page. For subscription or software businesses the equivalent question is plan presentation, covered by the pricing page layout test, and the experiment library lists the rest.

  • Judging on average order value. It rises almost by definition; margin per visitor is the real test.
  • Mixed messages across the site. Every touchpoint must show the arm's threshold.
  • Ignoring returns. Padded baskets that come back cost twice.
  • Running through a sale. Promotions swamp the threshold effect.
  • Choosing the threshold by guesswork. Use your own basket distribution.
  • Forgetting absorbed shipping costs. Free shipping is a cost, and it belongs in the margin.

Frequently asked questions

Why not judge the test on average order value?

A higher threshold almost always raises average order value among people who buy, but it can also reduce how many people buy and increase the shipping cost absorbed. Contribution margin per visitor captures all three effects in one number.

Is it fair to show different shipping offers to different visitors?

Tests of this kind are common, but each visitor should see one clear, consistent offer throughout their visit and at checkout. Consumer protection bodies such as the CMA expect prices and delivery charges to be presented clearly, so take advice for your market rather than relying on this page.

What about shoppers who pad baskets and then return items?

That is why return rate is a guardrail and why margin should be calculated after returns where possible. A threshold that encourages adding an item only to send it back costs the business twice.

Terms used on this page

Tools to try

Frameworks

Where this page is used

Part of these learning paths

Also related

Sources

  1. Competition and Markets Authority — GOV.UK

Related