Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Experiment Library

Trial Size Product Test

An experiment design for offering a paid trial size to first-time visitors, read on contribution margin per visitor once full-size orders have arrived.
Experimentintermediate10 min readUpdated

Hypothesis

If first-time visitors can buy a low-priced trial size beside the full size, contribution margin per first-time visitor over a fixed window will rise, because a small first order wins buyers who would not risk the full price and some go on to full-size orders, unless the trial size mostly replaces full-size first orders. The decision behind it: offer first-time visitors a paid trial size or sample set beside the full size, or sell full sizes only.

The guide to samples and trial sizes covers the kinds of sample, which products suit them, pricing a trial so it finds buyers rather than collectors, and the path from sample to full-size order. It judges a running programme by comparing sample buyers with a group of similar shoppers, as similar as the store can make it. That comparison carries a doubt the guide itself names: sample buyers chose the sample, and the choice says something about them. This design replaces the matched comparison with a random one, assigning visitors before anyone chooses.

The risk the hypothesis names last is the one that decides the test. A trial size can win first orders from people who would never have bought, and it can also take first orders from people who would have bought the full size anyway, at a fraction of the margin. Only a comparison against visitors who never saw the trial separates the two. Write the trial's price, its credit against a first full-size order, its cost and the limits on each guardrail into the plan before launch, as the experimentation loop asks of every test.

Metrics and guardrails

The primary metric is contribution margin per first-time visitor over a fixed window. Take every visitor assigned to an arm and follow them to the same age from their first visit. Add the contribution margin on every order they place in that window, trial and full size alike, after product cost, packing, delivery and the cost of making each trial size, and after any credit redeemed against a full-size order. Divide by every visitor assigned to the arm, including those who bought nothing. The window is the same for every visitor, so a visitor who arrived late in the entry period has had as long to reorder as one who arrived first.

How the trial sizes themselves are counted is a finance decision, and the samples guide sends it there. The marketing P&L guide lists free samples and promotional stock as an item whose treatment varies and says to agree it with finance first. For this test, agree before launch whether trial stock sits in cost of sales or in marketing, and count it once, in the margin.

Three guardrails catch a trial size that wins the primary metric for the wrong reasons, or loses it for reasons worth knowing:

  • Full-size first orders per first-time visitor. The cannibalisation check. If the treatment arm's full-size first orders fall, some visitors who would have risked the full size took the trial instead. That is acceptable only if enough trial buyers go on to the full size within the window.
  • Cost per full-size customer. The samples guide's measure of a programme's real CAC: everything the trial cost, including credits and the margin given up on diverted orders, divided by the full-size customers the arm produced. Compare it with a ceiling set before launch.
  • Repeat purchase rate of each arm's buyers at fixed ages. A trial that wins customers who buy one full size and stop may be worth less than it looks. Read the Repeat Purchase Rate of each arm's buyers at the same ages, as the samples guide reads them against lifetime value (LTV).

Design

Randomise at first-time visitor level, on the first visit, and hold each visitor in the arm on later visits. The trial size is a separate product. In the treatment arm it is listed beside the full size on the product page, linked from the relevant category pages and promoted wherever the shop promotes new-visitor offers. In the control arm it does not exist: no listing, no link, no search result.

Design elementChoiceReason
Unit of randomisationFirst-time visitor, assigned on the first visit and held in the armThe trial is an offer to new shoppers; returning customers already know the product
Treatment armFull sizes plus a paid trial size, listed, linked and promotedThe offer the samples guide designs, shown to a random half
Control armFull sizes onlyWhat the shop sells today
Held constantFull-size prices, delivery terms, welcome offers, pages and emails that reach both armsOnly the trial size differs
Trial termsOne price, one credit against a full-size order, one limit per customer, fixed for the runThe guide's pricing mechanics, set once so the test is not of pricing
Primary metricContribution margin per first-time visitor at a fixed ageCounts trial orders, diverted full-size orders and non-buyers together

Keep the trial size out of every channel that reaches both arms during the run. Product feeds, ads, marketplace listings and emails to the whole list would all let control visitors find it, and the test would then measure a trial that half the shop could see by another door. Where the trial must appear in an email, send that email only to treatment-arm subscribers.

The trial's price and credit come from the samples guide, not from the test. Work out what each trial size costs to make, pack and post, and how many trial buyers must go on to a full-size order for the trial to pay; the break-even calculator makes that arithmetic explicit and gives the target the result is read against. Change the trial's price during the run and the test becomes a pricing test with a contaminated first half.

The free shipping threshold test and the discount depth test are the neighbouring designs for lowering the risk of a first order. They change delivery terms and discounts on the full range; this design adds a product. If the shop is weighing all three, run them one at a time rather than together, so each answer is about one change.

Sample size reasoning

The metric is read per visitor over a long window, and most first-time visitors buy nothing, so it is mostly zeros with a few large values. That makes it skewed and slow. The inputs are the current contribution margin per first-time visitor, how widely it varies, new-visitor traffic per week and the smallest change worth stocking a trial size for. The sample size and significance guide shows how they combine, and why a skewed margin metric needs more visitors than a conversion rate on the same traffic.

If the shop cannot reach that number in a reasonable entry period, move the test to a smaller audience where the effect is larger: offer the trial size in the welcome email to new subscribers, assigning subscribers at random to a version with the trial and one without, and read margin per subscriber over the same window. Say in the plan that the answer then applies to subscribers, not to every visitor.

Duration

Open entries for whole weekly cycles, then follow every visitor to the same fixed age. The age must be long enough for a trial buyer to receive the trial, use it and reorder at the product's natural cycle: a product used up in weeks needs a shorter window than one that lasts months. Set it from the shop's own reorder history for the full size, and add the time a trial takes to arrive and be used. Read each week's entrants as a Cohort, as the cohort analysis tutorial shows, so a late entrant is never compared with an early one at a different age. Keep a seasonal peak wholly inside the entry period or wholly outside it.

Reading the results and the decision rule

Read once, when the last cohort reaches the fixed age. Compare contribution margin per first-time visitor between arms, with the interval around the difference. Then check the guardrails: full-size first orders per visitor, cost per full-size customer against its ceiling, and each arm's repeat purchase rate at the same ages. Write the rule into the plan before launch:

ResultDecision
Margin per first-time visitor rises reliably and cost per full-size customer stays within its limitKeep the trial size for first-time visitors
Full-size first orders fall by more than the trial orders bring back in marginDrop the trial size
Margin rises but trial buyers repeat less than full-size buyers at the same ageReprice the trial or turn it into a set, as the samples guide suggests, and retest
No reliable difference in marginKeep full sizes only; the trial adds stock and work without a measurable return

Pitfalls

The trial size leaking through feeds and emails is the commonest flaw. A control visitor who meets the trial in a shopping ad, a marketplace listing or a newsletter is no longer a control visitor. List every place a product can appear before launch and close each one to the control arm.

Reading before the reorder cycle flatters the control arm. Full-size first orders arrive at once; trial buyers' full-size orders arrive only after the trial has been used. A reading taken early sees the diverted margin and none of the recovery. Hold every visitor to the same fixed age.

Trial buyers counted as returning customers distort every guardrail. When a trial buyer places a first full-size order, analytics may record them as a returning customer, because they ordered before. The guide to new and returning customers explains how to record customer status so that first full-size order counts as a new customer's order.

Sample cost booked twice is the quiet error. If the trial's cost sits in marketing and is also deducted in the margin, the treatment arm pays for it twice. Agree the treatment with finance before launch and apply it once.

Example: an invented haircare brand tests a trial set (illustrative)

Illustrative example — not a real result

Jestavon Haircare tests a trial set for first-time visitors

Jestavon Haircare, an invented haircare brand, sells shampoo and conditioner in full-size bottles that last about eight weeks. It lists a trial set of travel-size shampoo and conditioner at £7, credited against a first full-size order within 60 days, beside its full-size bottles for half of its first-time visitors. The set costs £4.20 to make, pack and post. Over six weeks, 36,000 first-time visitors are assigned, and each is followed for 16 weeks from the first visit, long enough for a trial buyer to use the set and reorder.

ArmVisitorsFull-size first ordersTrial setsFull-size customers at 16 weeksMargin per visitor
Full sizes only18,0504510451£0.98
With trial set17,950412720594£1.11

The trial arm loses 39 full-size first orders to the set, but 182 trial buyers go on to a full size within the window, and margin per visitor is £0.13 higher, with an interval of roughly £0.02 to £0.24. Cost per full-size customer stays inside the ceiling the team set. At 16 weeks, trial-arm customers have reordered at a similar rate to full-size-first customers. Jestavon keeps the trial set for first-time visitors and plans to read repeat purchase again at a later age.

Common mistakes

Trial size tests fail on their window and their boundaries more often than on their statistics. The experiment library holds the other offer and pricing designs.

  • Comparing trial buyers with full-size buyers. They chose differently; compare arms, not buyers.
  • Dividing by buyers instead of visitors. The trial changes who buys; only a per-visitor metric counts that.
  • Reading at the first order. The trial's return arrives after the reorder cycle.
  • Letting the trial appear in shared channels. Feeds, ads and list-wide emails reach the control arm.
  • Changing the trial's price mid-run. That turns the test into a pricing test with half its data spoiled.
  • Ignoring cannibalisation. Full-size first orders lost to the trial are part of its cost.

Frequently asked questions

Why not just compare trial buyers with shoppers who bought the full size first?

Because the two groups chose differently, and the choice says something about them. Shoppers who take a trial are often less sure of the product or more price-conscious, so they would behave differently even with no trial on offer. The samples guide's matched comparison is the best a running programme can do; assigning visitors at random before anyone chooses removes the difference, because both arms contain the same mix of shoppers.

Why is the metric per visitor rather than per buyer?

A trial size changes who buys, not only what they buy. Dividing by buyers would hide a trial that draws in many small orders while losing full-size buyers, or one that wins buyers who never go on to a full size. Dividing by every first-time visitor in the arm counts the people the trial won, the people it diverted and the people who bought nothing, all in one number.

What if the shop's traffic is too small for a visitor-level test?

Move the test to a smaller, more engaged audience. A shop can offer the trial size in the welcome email to new subscribers, assigning subscribers at random to a version with the trial and one without, and read margin per subscriber over the same window. The answer applies to subscribers rather than to every visitor, and the plan should say so.

Terms used on this page

Tools to try

Frameworks

Also related

Related