Hypothesis
If first-time visitors can buy a low-priced trial size beside the full size, contribution margin per first-time visitor over a fixed window will rise, because a small first order wins buyers who would not risk the full price and some go on to full-size orders, unless the trial size mostly replaces full-size first orders. The decision behind it: offer first-time visitors a paid trial size or sample set beside the full size, or sell full sizes only.
The guide to samples and trial sizes covers the kinds of sample, which products suit them, pricing a trial so it finds buyers rather than collectors, and the path from sample to full-size order. It judges a running programme by comparing sample buyers with a group of similar shoppers, as similar as the store can make it. That comparison carries a doubt the guide itself names: sample buyers chose the sample, and the choice says something about them. This design replaces the matched comparison with a random one, assigning visitors before anyone chooses.
The risk the hypothesis names last is the one that decides the test. A trial size can win first orders from people who would never have bought, and it can also take first orders from people who would have bought the full size anyway, at a fraction of the margin. Only a comparison against visitors who never saw the trial separates the two. Write the trial's price, its credit against a first full-size order, its cost and the limits on each guardrail into the plan before launch, as the experimentation loop asks of every test.
Metrics and guardrails
The primary metric is contribution margin per first-time visitor over a fixed window. Take every visitor assigned to an arm and follow them to the same age from their first visit. Add the contribution margin on every order they place in that window, trial and full size alike, after product cost, packing, delivery and the cost of making each trial size, and after any credit redeemed against a full-size order. Divide by every visitor assigned to the arm, including those who bought nothing. The window is the same for every visitor, so a visitor who arrived late in the entry period has had as long to reorder as one who arrived first.
How the trial sizes themselves are counted is a finance decision, and the samples guide sends it there. The marketing P&L guide lists free samples and promotional stock as an item whose treatment varies and says to agree it with finance first. For this test, agree before launch whether trial stock sits in cost of sales or in marketing, and count it once, in the margin.
Three guardrails catch a trial size that wins the primary metric for the wrong reasons, or loses it for reasons worth knowing:
- Full-size first orders per first-time visitor. The cannibalisation check. If the treatment arm's full-size first orders fall, some visitors who would have risked the full size took the trial instead. That is acceptable only if enough trial buyers go on to the full size within the window.
- Cost per full-size customer. The samples guide's measure of a programme's real CACCost to acquire one new customer.: everything the trial cost, including credits and the margin given up on diverted orders, divided by the full-size customers the arm produced. Compare it with a ceiling set before launch.
- Repeat purchase rate of each arm's buyers at fixed ages. A trial that wins customers who buy one full size and stop may be worth less than it looks. Read the Repeat Purchase RateShare of customers who placed more than one order in the period. of each arm's buyers at the same ages, as the samples guide reads them against lifetime value (LTVTotal value a customer brings over time.).
Design
Randomise at first-time visitor level, on the first visit, and hold each visitor in the arm on later visits. The trial size is a separate product. In the treatment arm it is listed beside the full size on the product page, linked from the relevant category pages and promoted wherever the shop promotes new-visitor offers. In the control arm it does not exist: no listing, no link, no search result.
| Design element | Choice | Reason |
|---|---|---|
| Unit of randomisation | First-time visitor, assigned on the first visit and held in the arm | The trial is an offer to new shoppers; returning customers already know the product |
| Treatment arm | Full sizes plus a paid trial size, listed, linked and promoted | The offer the samples guide designs, shown to a random half |
| Control arm | Full sizes only | What the shop sells today |
| Held constant | Full-size prices, delivery terms, welcome offers, pages and emails that reach both arms | Only the trial size differs |
| Trial terms | One price, one credit against a full-size order, one limit per customer, fixed for the run | The guide's pricing mechanics, set once so the test is not of pricing |
| Primary metric | Contribution margin per first-time visitor at a fixed age | Counts trial orders, diverted full-size orders and non-buyers together |
Keep the trial size out of every channel that reaches both arms during the run. Product feeds, ads, marketplace listings and emails to the whole list would all let control visitors find it, and the test would then measure a trial that half the shop could see by another door. Where the trial must appear in an email, send that email only to treatment-arm subscribers.
The trial's price and credit come from the samples guide, not from the test. Work out what each trial size costs to make, pack and post, and how many trial buyers must go on to a full-size order for the trial to pay; the break-even calculator makes that arithmetic explicit and gives the target the result is read against. Change the trial's price during the run and the test becomes a pricing test with a contaminated first half.
The free shipping threshold test and the discount depth test are the neighbouring designs for lowering the risk of a first order. They change delivery terms and discounts on the full range; this design adds a product. If the shop is weighing all three, run them one at a time rather than together, so each answer is about one change.
Sample size reasoning
The metric is read per visitor over a long window, and most first-time visitors buy nothing, so it is mostly zeros with a few large values. That makes it skewed and slow. The inputs are the current contribution margin per first-time visitor, how widely it varies, new-visitor traffic per week and the smallest change worth stocking a trial size for. The sample size and significance guide shows how they combine, and why a skewed margin metric needs more visitors than a conversion rate on the same traffic.
If the shop cannot reach that number in a reasonable entry period, move the test to a smaller audience where the effect is larger: offer the trial size in the welcome email to new subscribers, assigning subscribers at random to a version with the trial and one without, and read margin per subscriber over the same window. Say in the plan that the answer then applies to subscribers, not to every visitor.
Duration
Open entries for whole weekly cycles, then follow every visitor to the same fixed age. The age must be long enough for a trial buyer to receive the trial, use it and reorder at the product's natural cycle: a product used up in weeks needs a shorter window than one that lasts months. Set it from the shop's own reorder history for the full size, and add the time a trial takes to arrive and be used. Read each week's entrants as a CohortA group of customers who share a starting event and period, tracked together over time., as the cohort analysis tutorial shows, so a late entrant is never compared with an early one at a different age. Keep a seasonal peak wholly inside the entry period or wholly outside it.
Reading the results and the decision rule
Read once, when the last cohort reaches the fixed age. Compare contribution margin per first-time visitor between arms, with the interval around the difference. Then check the guardrails: full-size first orders per visitor, cost per full-size customer against its ceiling, and each arm's repeat purchase rate at the same ages. Write the rule into the plan before launch:
| Result | Decision |
|---|---|
| Margin per first-time visitor rises reliably and cost per full-size customer stays within its limit | Keep the trial size for first-time visitors |
| Full-size first orders fall by more than the trial orders bring back in margin | Drop the trial size |
| Margin rises but trial buyers repeat less than full-size buyers at the same age | Reprice the trial or turn it into a set, as the samples guide suggests, and retest |
| No reliable difference in margin | Keep full sizes only; the trial adds stock and work without a measurable return |
Pitfalls
The trial size leaking through feeds and emails is the commonest flaw. A control visitor who meets the trial in a shopping ad, a marketplace listing or a newsletter is no longer a control visitor. List every place a product can appear before launch and close each one to the control arm.
Reading before the reorder cycle flatters the control arm. Full-size first orders arrive at once; trial buyers' full-size orders arrive only after the trial has been used. A reading taken early sees the diverted margin and none of the recovery. Hold every visitor to the same fixed age.
Trial buyers counted as returning customers distort every guardrail. When a trial buyer places a first full-size order, analytics may record them as a returning customer, because they ordered before. The guide to new and returning customers explains how to record customer status so that first full-size order counts as a new customer's order.
Sample cost booked twice is the quiet error. If the trial's cost sits in marketing and is also deducted in the margin, the treatment arm pays for it twice. Agree the treatment with finance before launch and apply it once.
Example: an invented haircare brand tests a trial set (illustrative)
Jestavon Haircare tests a trial set for first-time visitors
Jestavon Haircare, an invented haircare brand, sells shampoo and conditioner in full-size bottles that last about eight weeks. It lists a trial set of travel-size shampoo and conditioner at £7, credited against a first full-size order within 60 days, beside its full-size bottles for half of its first-time visitors. The set costs £4.20 to make, pack and post. Over six weeks, 36,000 first-time visitors are assigned, and each is followed for 16 weeks from the first visit, long enough for a trial buyer to use the set and reorder.
| Arm | Visitors | Full-size first orders | Trial sets | Full-size customers at 16 weeks | Margin per visitor |
|---|---|---|---|---|---|
| Full sizes only | 18,050 | 451 | 0 | 451 | £0.98 |
| With trial set | 17,950 | 412 | 720 | 594 | £1.11 |
The trial arm loses 39 full-size first orders to the set, but 182 trial buyers go on to a full size within the window, and margin per visitor is £0.13 higher, with an interval of roughly £0.02 to £0.24. Cost per full-size customer stays inside the ceiling the team set. At 16 weeks, trial-arm customers have reordered at a similar rate to full-size-first customers. Jestavon keeps the trial set for first-time visitors and plans to read repeat purchase again at a later age.
Common mistakes
Trial size tests fail on their window and their boundaries more often than on their statistics. The experiment library holds the other offer and pricing designs.
- Comparing trial buyers with full-size buyers. They chose differently; compare arms, not buyers.
- Dividing by buyers instead of visitors. The trial changes who buys; only a per-visitor metric counts that.
- Reading at the first order. The trial's return arrives after the reorder cycle.
- Letting the trial appear in shared channels. Feeds, ads and list-wide emails reach the control arm.
- Changing the trial's price mid-run. That turns the test into a pricing test with half its data spoiled.
- Ignoring cannibalisation. Full-size first orders lost to the trial are part of its cost.
