Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Experiment Library

Loyalty Programme Pilot Test

An experiment design for a loyalty programme pilot: eligible customers invited at random or held out, read on incremental margin after every reward cost.
Experimentintermediate11 min readUpdated

Hypothesis

If eligible customers are split at random between an invitation to the planned loyalty programme and a held-out group that is not invited, the invited group will earn more contribution margin per eligible customer after every reward and running cost, because the rewards target behaviour the business would not otherwise get, unless most rewards go to purchases that would have happened anyway. The decision behind it is concrete: launch the programme to every eligible customer, redesign its rewards around the behaviour that moved, or keep the money for other retention work.

The guide to designing a loyalty programme sets out the models, what is worth rewarding and the economics of liability and breakage, and it says plainly that measuring what the programme changes requires a comparison group, which is why a pilot beats a full launch. Its own pilot compares members with similar non-members. It also reports work associated with the Ehrenberg-Bass Institute arguing that programmes mostly attract a brand's existing heavy buyers, which is exactly why that comparison flatters: members were different before they joined. The repeat purchase and loyalty guide calls a holdout the only clean measure of incrementality. This page turns that paragraph into a plan, with the randomisation placed on the invitation rather than on membership. Write the groups, the expected result and the limit on each guardrail into the experiment hypothesis template before the first invitation goes out.

Metrics and guardrails

The primary metric is incremental contribution margin per eligible customer. For each group, take the Contribution Margin on every order its customers placed during the reading period, from the business's own order records and by every route, and divide by the number of customers assigned to the group. Subtract the held-out group's figure from the invited group's. Then subtract, per invited customer, three costs: the rewards redeemed, the value of rewards earned but still owed at the reading date, and the running costs of the programme, from platform fees to the staff time and messages it needs.

Two choices keep the metric honest. The denominator is everyone assigned, so customers who ignored the invitation count against it, as they would after a real launch. And rewards still owed are valued in full, not discounted for the Breakage the business hopes for. The loyalty guide warns that a programme which only breaks even when customers fail to redeem is designed to disappoint them; a test that assumes breakage would repeat that mistake.

Three guardrails, each with a limit written before launch:

  • Value of rewards still owed per member at the reading date. A programme that looks cheap because its rewards have not been claimed yet is carrying a debt to its members. Read the balance per member beside the primary metric.
  • Programme contacts and complaints per member. Confusing rules or rewards that are hard to redeem show up as questions and complaints before they show up in spending.
  • Purchases per customer in a follow-on period. Read for both groups after the reading period closes. If invited customers bought earlier to reach a reward and then paused, the gap narrows here.

Read the Purchase Frequency and Repeat Purchase Rate of both groups as diagnostics: they show which behaviour moved, which is what a redesign would be built around.

Design

Randomise at the level of the identified customer who is eligible to join. Apply the eligibility rules first, then assign each eligible customer at random, once, before the first invitation, and flag the group on the customer record in the CRM so every later message and order can be read against it. A customer's group never changes during the test.

Design elementChoiceReason
Unit of randomisationIdentified eligible customer, assigned once before the first invitationThe invitation is the treatment; joining is the customer's own choice
GroupsInvited to the programme; held out and not invitedThe held-out group shows what eligible customers buy without the programme
Held constantPrices, offers, emails, advertising and serviceOnly the invitation differs, so only the programme can explain the difference
Programme termsOne set of rewards, rules and wording for everyone invitedA result that mixes several schemes cannot say which one to launch
CountingEveryone assigned, joined or notMembers alone are the customers who were already buying most
Primary metricIncremental contribution margin per eligible customer, after all reward and running costsIt is the margin a launch would add, net of what it gives away

Invite through owned channels only, and keep the programme off public pages and advertising during the test, so held-out customers are not invited by accident. If a held-out customer finds the programme and joins anyway, let them and leave them in the held-out group for the reading. Held-out customers keep every existing email, offer and price.

The invitation and the terms are part of the treatment. The loyalty guide records that UK consumer protection law, which the Competition and Markets Authority enforces, expects terms to be fair and transparent and marketing not to mislead, and that this applies to how rewards, expiry and changes to the scheme are described; the comparison of loyalty programmes and subscribe-and-save states the same direction. For a pilot that means plain terms that say the programme is being trialled, every reward earned during the pilot honoured in full, and, if the programme does not launch, a closure explained plainly with balances honoured. Held-out customers are invited if it does launch. This is general direction, not legal advice.

Sample size reasoning

Spend per customer varies widely, and much of what the invited group buys would have been bought anyway, so the programme's effect is small against the noise. Counting everyone assigned dilutes it further, because only part of the invited group joins. The held-out group therefore has to be large enough to read a small difference, and is often a larger share of the base than a team expects. The inputs are spend per eligible customer, how much it varies, the share expected to join and the smallest incremental margin that would cover the programme's costs. The section on cells, size and duration in the incrementality testing guide explains how those combine for a holdout, the sample size and significance guide covers intervals and power, and the break-even calculator turns the programme's costs into the incremental margin it has to produce.

If the base cannot support a reading of the whole programme, change the question before launch: pilot one reward on one behaviour, such as a first purchase in a second category, and read that behaviour against the held-out group.

Duration

The test has three periods, all fixed before the first invitation. An invitation period, long enough to reach every invited customer through the channels that would carry a real launch. A reading period long enough for several purchase cycles in the category and for rewards to be earned and redeemed, since a programme read before anyone can claim a reward measures anticipation, not the scheme. Then a follow-on period, set by the same purchase cycle, for the guardrail that catches spending pulled forward. Avoid starting the reading period in the middle of the business's largest sale, which changes buying in both groups and the costs of rewards with it.

Reading the results and the decision rule

Read once, at the end of the reading period, and again for the guardrail at the end of the follow-on period. Compare incremental contribution margin per eligible customer with the interval around the difference, not a single figure. Then read the guardrails, then the diagnostics: which behaviours moved, and under which reward. Name the reward-level readings before launch, so a redesign rests on a planned comparison rather than a search through the results. Write the rule into the plan:

ResultDecision
Incremental margin per eligible customer after all reward and running costs is reliably above zero, and the guardrails stay within their limitsLaunch the programme to every eligible customer and invite the held-out group
Overall margin is negative, but one reward named before launch shows a reliable change in behaviourRedesign the rewards around that behaviour and test the redesign the same way
The invited group behaves no differently from the held-out groupDo not launch; keep the money for other retention work
Margin looks positive but follow-on purchases fall below the held-out group'sTreat the gain as spending pulled forward and do not launch as designed

The flywheel framework frames the result: a programme belongs in the plan if it adds momentum to a loop that already turns. The flywheel applied to customer loyalty treats rewards that only pay for existing purchases as friction, and the double jeopardy pattern is a reminder that loyalty figures largely follow a brand's size, so a modest effect is the normal outcome, not a failure of the test.

Pitfalls

Comparing members with non-members is the first trap and the one this design exists to avoid. Members were heavier buyers before they joined; any reading built on members alone credits the programme with that difference.

Ignoring rewards still owed is the second. A reading taken before members claim their rewards shows the sales and leaves out much of the cost. Value the balance in full at the reading date.

The programme advertised to the held-out group is the third. A banner on the website, a mention in a general newsletter or a cashier offering sign-up to everyone turns the held-out group into a partly invited one and shrinks the difference. Keep the invitation on owned, flagged channels.

Spending pulled into the reward period is the fourth. Customers who bring purchases forward to reach a reward look like converts until they stop. The follow-on period catches it.

Reading before rewards are redeemed is the fifth, and the general traps apply too: check that each group received the share of the base the plan intended, and resist reading early. The guide to common testing pitfalls covers both. Two designs sit beside this one. For lapsed customers who might need an offer rather than a programme, the win-back offer test is the design; for a business that suspects its customers have learned to wait for codes, the full-price protection test is. Recognition messages to members once a programme runs belong to the guide to email for loyalty and advocacy.

Example: an invented online pet supplies shop pilots a points programme (illustrative)

Illustrative example — not a real result

Druvampel Pet Supplies pilots a points programme with a held-out group

Druvampel Pet Supplies, an invented online pet supplies shop, plans a points programme: one point per £1, a £5 voucher at 250 points, and 100 bonus points for a first order in a second category such as grooming or toys. It has 30,000 identified customers who ordered in the past year. Before any invitation, 24,000 are assigned at random to be invited and 6,000 are held out. The invitation period is two weeks, the reading period six months and the follow-on period three months. Before launch the team caps rewards still owed at £4 per member and names the second-category bonus as the one reward to read on its own. Running costs are £14,400 for the six months, or £0.60 per invited customer.

Measure, per eligible customerInvited (24,000)Held out (6,000)
Contribution margin in the reading period£46.80£44.20
Rewards redeemed£1.40—
Rewards still owed, valued in full£0.95—
Running costs£0.60—
Margin after programme costs£43.85£44.20

7,200 invited customers join. The invited group earns £2.60 more contribution margin per eligible customer, but the programme costs £2.95, leaving −£0.35, with an interval of roughly −£1.70 to £1.00. Rewards still owed come to about £3.17 per member, inside the cap. In the follow-on period invited customers place 1.12 orders each against 1.15 for the held-out group, a sign that some food orders were brought forward. The named reward tells a different story: 8.9% of the invited group made a first order in a second category, against 6.1% of the held-out group, an interval of roughly 2.1 to 3.5 points. Druvampel does not launch the points scheme. It redesigns the programme around the second-category bonus, drops points on routine food orders and plans a second pilot of that reward with a fresh held-out group.

Before the first invitation, the loyalty programme walk-through of the break-even calculator turns the scheme's full cost into the extra orders per eligible customer this pilot must detect.

Common mistakes

Loyalty pilots usually fail on whom they compare and when they read. The experiment library lists the offer and retention designs that sit beside this one.

  • Comparing members with non-members. Members were already the heaviest buyers; randomise the invitation instead.
  • Counting only those who joined. The decision is whether to offer the programme, so count everyone invited.
  • Leaving rewards still owed out of the cost. They are a debt to members, not a saving.
  • Counting on breakage. A test that assumes rewards will go unclaimed approves a programme that disappoints.
  • Promoting the programme publicly during the pilot. The held-out group stops being held out.
  • Reading before rewards can be claimed. The sales arrive before the costs; wait for both.

Frequently asked questions

Why count customers who were invited but never joined?

Because the decision is about offering the programme, not about the customers who chose to take it up. Those who join are usually the ones who were already buying most, so reading members alone credits the programme with loyalty it did not create. Counting everyone invited against everyone held out keeps the two groups alike, so the only systematic difference between them is the invitation. The effect per eligible customer is smaller than the effect per member, but it is the one that tells the business what a launch would add.

Is it fair to leave some customers out of a loyalty programme?

Holding a random part of the base out of a new programme for a fixed period is a normal way to learn whether it works, and the held-out customers keep every existing message, offer and price. What matters is how the invited customers are treated: plain terms, every reward earned honoured in full, and no claim about rewards or expiry that the business cannot keep. Held-out customers are invited if the programme launches. This is general direction, not legal advice; a scheme with expiry, paid membership or changes to balances deserves review by someone qualified.

Can the same design test a programme the business already runs?

Not as written. Once every customer has been offered the programme, there is nobody left to hold out from the invitation. For an existing programme, hold a random part of eligible customers out of one reward or one tier benefit and compare, which answers whether that reward earns its cost. Use this design when the question is whether to launch at all, and plan the held-out group before anyone is told the programme exists.

Terms used on this page

Tools to try

Frameworks

Where this page is used

Also related

Sources

  1. Competition and Markets Authority — GOV.UK
  2. Ehrenberg-Bass Institute for Marketing Science — Ehrenberg-Bass Institute

Related