Hypothesis
If invited customers are split at random between a reward for the referrer, a welcome for the friend and a reward on both sides, the two-sided arm will bring the most referral margin per invited customer, because a welcome for the friend makes the recommendation easier to pass on, unless the second reward costs more than the referrals it adds. The decision behind it is concrete: choose who the programme rewards, and keep the structure that brings the most margin per invited customer.
The tutorial on designing a referral programme frames the choice. A welcome for the friend lets the customer offer something rather than ask a favour; a reward for the referrer gives them a reason to remember. Which one holds referrals back is a property of the business and its customers, not a general rule, which is why it is worth testing rather than copying. The same tutorial asks for one element of the programme to change at a time, so this design moves only the structure of the reward. The ask, its timing, the sharing flow, the terms and the fraud checks stay identical in every arm. Write the arms, the expected behaviour of each and the limits on each guardrail into the experiment hypothesis template before anyone is invited.
Metrics and guardrails
The primary metric is referral margin per invited customer. Take every referred first order from customers in an arm, keep only those that have passed the return or cancellation period, and add up their Contribution MarginRevenue left after every cost that varies with each sale.. Subtract every reward paid to referrers in that arm, every welcome discount given to their friends and any platform fee per referral. Divide the result by the number of customers invited in the arm, whether or not they ever shared. Two features matter. The denominator is fixed at invitation, so an arm whose reward nobody cares about shows up as low margin rather than disappearing from the arithmetic. And the numerator is net of reward cost, so a generous arm cannot win simply by paying for more referrals than they are worth.
Three guardrails protect against a structure that wins the primary metric for the wrong reasons:
- Share of referrals flagged by the fraud checks. Self-referral through a second account and codes published on deal sites are the patterns the affiliate partner and fraud guide lists, and the referral tutorial applies them to referrals. A structure that pays the referrer is the more attractive one to game, so watch this flag by arm.
- Repeat purchase rate of referred customers at a fixed age. A referral is worth more only if the customer it brings is worth at least as much as other new customers. Read the Repeat Purchase RateShare of customers who placed more than one order in the period. of each arm's referred customers at the same age, so a welcome that attracts one-order bargain hunters is caught.
- Cost per referred customer against the allowable cost. The referral tutorial sets the allowable cost per new customer from the margin on the first order plus the repeat value the business can rely on (LTVTotal value a customer brings over time.), less the profit it wants to keep. The CPA calculator turns each arm's total reward and discount cost and its referred customers into a cost per referred customer to compare with that ceiling and with CACCost to acquire one new customer. from paid channels.
Design
Randomise at customer level, before the first invitation. Take the customers who will be invited during the test window, usually those reaching the success point the programme asks at, and assign each one at random to an arm when they first qualify. Store the arm on the customer record. The personal link or code issued to that customer carries the arm, so every friend who arrives through it is read back to the referrer's arm, and the friend sees the welcome that arm offers, or none.
| Design element | Choice | Reason |
|---|---|---|
| Unit of randomisation | Existing customer, assigned once at first invitation | The reward acts on whether the customer shares; friends follow from that |
| Arms | Referrer reward; friend welcome; both sides | Two arms if the customer base is small: the current structure against the one most likely to beat it |
| Reward value | Each arm's total cost set from the same allowable cost | Compares structures, not amounts; a richer arm would win on generosity alone |
| Held constant | The ask, its moment, the sharing flow, terms, fraud checks and disclosure wording | The referral tutorial's rule: change one element at a time |
| Attribution | The personal link or code, with the code also accepted at checkout | Friends who type the address are still counted to the right arm |
| Primary metric | Referral margin per invited customer, after returns and all rewards | Counts customers who never shared and the full cost of each structure |
Hold the total cost per successful referral roughly level across arms, so the comparison is between structures rather than between a cheap reward and a generous one. A two-sided arm that splits one budget between referrer and friend is the fair counterpart to a one-sided arm that gives the whole budget to one person. If the business wants to test the amount as well, run that as a second test once the structure is settled.
Two rules about people apply in every arm, and they are general practice rather than legal advice. Customers share from their own email, messaging and social accounts; the business does not collect friends' details to message them itself, which is the question the UK Information Commissioner's Office's direct marketing guidance addresses, as the referral tutorial explains. And when a rewarded customer posts publicly, their audience should be able to tell: the FTC's endorsement guidance, as the influencer contracts and disclosure guide sets out, expects a material connection to be disclosed clearly. Give the same suggested wording in every arm, adjusted only for what that arm's reward is.
Sample size reasoning
The sample is invited customers, not referrals. Most invited customers refer nobody, so the primary metric is mostly zeros with a few large values, and a modest difference between arms is hard to separate from chance. Three arms divide the customer base three ways. The inputs are the expected referral margin per invited customer in the current structure, how widely it varies between customers, the smallest difference that would change the reward and the confidence the team requires; the sample size and significance guide explains how they combine and why a skewed metric needs far more people than a conversion rate on the same base.
If the base cannot detect a difference worth acting on, change the question before launch rather than running the test anyway. Drop to two arms, the current structure against the challenger. Read a less skewed metric alongside, such as the share of invited customers who brought at least one completed referral. Or widen the window so that more customers are invited, accepting the longer wait.
Duration
The test runs in two stages, both fixed before launch. The invitation stage lasts long enough for every customer assigned to reach the success point and receive the same ask. The reading stage stays open until friends invited during the first stage have had time to order, and until every referred order has passed its return or cancellation period, because the referral tutorial pays rewards only after that point and the metric should count what the programme would actually pay. If the business has a gifting peak, keep it wholly inside or wholly outside the invitation stage, so that every arm meets the same season.
Reading the results and the decision rule
Read the test once, at the end of the reading stage. Compare referral margin per invited customer between arms, with the interval around each difference rather than the point estimate. Then check the guardrails by arm: fraud flags, the referred customers' repeat purchase at the fixed age and the cost per referred customer against the allowable cost. Only after that, look at any customer segment named before launch, such as long-standing customers against recent ones. Write the decision rule into the plan in advance:
| Result | Decision |
|---|---|
| One arm earns reliably more margin per invited customer and its guardrails hold | Adopt that structure for every customer |
| The best arm wins but its fraud flags or the repeat purchase of its referred customers break a limit | Keep the current structure and tighten the checks or the reward timing before retesting |
| No reliable difference between arms | Keep the cheapest structure to run and explain |
| Every arm costs more per referred customer than the allowable cost | Redesign the ask or its moment, or retire the programme |
Remember what the test cannot say. It compares structures, and every arm contains friends who would have arrived anyway, so a winning arm is not proof that the programme adds customers overall. The referral tutorial's measurement step, comparing how new customers arrive with and without the programme, answers that. The growth loops framework explains why a referral programme is often a small loop presented as a large one, which is the reason to check both.
Pitfalls
Friends talk. A friend invited by a customer in one arm may know a customer in another and ask for the better offer, or a code may travel beyond the referrer's own contacts. Accept only the code a friend actually used, honour it exactly, and treat a code that appears on a deal site as a fraud flag in its arm. Rewards counted before returns are the second trap: a structure that pays quickly looks strong until cancellations arrive, so the metric waits for the return period. The third is uneven asking. If one arm's invitation goes out by email and another's in the app, or one arm gets an extra reminder, the test measures the ask rather than the reward.
The general testing traps apply too. Check that the arms received the shares of customers the plan intended, because a broken assignment rule produces a sample ratio mismatch, and resist reading the arms each week. The guide to common testing pitfalls covers both. Once a structure is chosen, the discount depth test is the design for setting how large a welcome discount should be, and the win-back offer test is the design for customers who have stopped buying rather than recommending.
Example: an invented flower subscription tests its referral reward (illustrative)
Kluvarne Flowers tests who its referral reward should go to
Kluvarne Flowers, an invented flower subscription, invites customers to refer a friend after a delivery they rated well. Its allowable cost per new subscriber is £30, and it sets every arm's cost per completed referral at £20: £20 credit for the referrer, a £20 welcome off the friend's first box, or £10 for each. Over eight weeks, 6,000 customers reach the invitation and are split at random into three arms of 2,000. Rewards are released only once the friend's second box is delivered, and the reading stage closes twelve weeks after the last invitation.
| Arm | Completed referrals | Referral margin after rewards | Margin per invited customer | Flagged referrals |
|---|---|---|---|---|
| Referrer reward | 96 | £1,250 | £0.63 | 9 |
| Friend welcome | 118 | £1,480 | £0.74 | 3 |
| Both sides | 141 | £1,860 | £0.93 | 4 |
The two-sided arm leads the referrer-only arm by £0.30 per invited customer, with an interval of roughly £0.06 to £0.54, and leads the friend-only arm by a smaller margin whose interval includes zero. At ninety days, the referred subscribers in all three arms renew at similar rates, and the referrer-only arm carries most of the flagged self-referrals. Kluvarne adopts the two-sided structure, keeps the cost per referral at £20, and plans a later test of the amount.
Common mistakes
Reward tests fail more often on their metric and window than on their statistics. The experiment library lists the other designs, including the offer tests that sit beside this one.
- Counting referrals instead of margin. A generous arm wins on volume while losing money on each referral.
- Dividing by customers who shared. The cost of a reward nobody acts on is part of what is being tested; divide by everyone invited.
- Changing the amount along with the structure. Hold the cost per referral level, or the test measures generosity.
- Reading before the return period ends. Rewards on orders that are later cancelled flatter the fastest arm.
- Asking differently in each arm. The moment, the channel and the reminders must match.
- Treating the winner as proof the programme works. A structure test says which reward is better, not whether the programme adds customers at all.
