Hypothesis
If the pricing page shows the annual billing period first, with the monthly price visible and each labelled with how it is billed, contracted revenue per pricing-page visitor at a fixed horizon will rise because buyers ready to commit see that option first, unless plan starts fall or more annual buyers ask for refunds. The decision behind it is narrow: show the annual or the monthly billing period first on the pricing page, with both prices always visible and each labelled with how it is billed.
The guide to designing pricing pages frames the choice. Either period can lead, provided both are shown clearly; which one appears first, or whether a switch sets the default, is a presentation choice that can be tested, while the honesty of the labelling is not. The test is worth running because the order works in both directions. Leading with the annual period may help buyers who were ready to commit, and it may also push a hesitant buyer towards a commitment they later regret, or away from the page altogether. Which effect wins depends on the product and its buyers. Write the arms, the expected direction and the limit on each guardrail into the experiment hypothesis template before the first visitor is assigned.
Metrics and guardrails
The primary metric is contracted revenue per pricing-page visitor at a fixed horizon. Take every plan started by visitors in an arm, add the first-term value of each, meaning the monthly payment for a monthly plan and the annual payment for an annual plan, subtract every refund issued within the horizon, and divide by the number of visitors assigned to the arm. Take the figures from the billing system, not the analytics tool: the subscription measurement guide names billing as the source of truth for amounts actually charged after discounts, refunds and credits. The denominator is fixed at assignment, so an arm that loses buyers shows up as lower revenue rather than vanishing from the sum.
A plain conversion rateShare of visits or clicks that convert. is the wrong primary metric here, because the treatment changes what each start is worth as well as how many there are. The guardrails keep each effect visible:
- Plan starts per pricing-page visitor. A first period that scares buyers off can still win on revenue if the remaining buyers pay annually. The conversion rate calculator turns each arm's starts and visitors into a rate to compare against the limit written before launch.
- Refunds and early cancellations by billing period. Annual buyers who did not mean to commit ask for their money back; monthly buyers who were nudged in the wrong direction leave at the first renewal. Read both by arm and by period, as an early sign of churnShare of customers at the start of a period who left during it..
- Support contacts about price or billing. A rise in questions such as "why was I charged for a year?" is a warning that the labelling is not working for some buyers, even when it is accurate.
Each of these is a Guardrail MetricA metric an experiment must not worsen beyond an agreed bound, watched beside the primary metric that decides the test. with its limit agreed in advance. Renewals beyond the horizon and long-run LTVTotal value a customer brings over time. matter too, but they arrive months later; record them as a follow-up read rather than waiting for them before deciding.
Design
This is an A/B testControlled comparison of two variants. at visitor level. Assign each visitor at random at their first pricing-page view and hold them in that arm on every return, across sessions and across devices where sign-in allows, because buyers often visit a pricing page several times before choosing. One arm shows the monthly period first, the other the annual period first; whichever is the current page is the control.
| Design element | Choice | Reason |
|---|---|---|
| Unit of randomisation | Visitor, from the first pricing-page view, held across sessions and devices where possible | A returning buyer who sees the order flip is confused and contaminates both arms |
| Arms | Monthly period shown first; annual period shown first | The only decision on the table is the order |
| Labelling | Both prices visible in both arms, each stating how it is billed, total and commitment beside any per-month figure | Honest labelling is a condition, not a variable |
| Held constant | Prices, discounts, plan names, contents, tiers, copy, layout and checkout | Choosing prices and building tiers belongs to product marketing |
| Excluded | Existing customers upgrading or renewing | Their choice is shaped by the plan they already hold |
| Primary metric | Contracted revenue per pricing-page visitor, from billing, net of refunds | Captures both how many buyers start and what each start is worth |
Everything outside the order stays fixed. The pricing and packaging guide owns price levels and annual discounts, and the good-better-best framework owns how tiers are built and which one is the honest default; neither changes during the test. If the page uses a switch, the switch sets the starting position and the buyer can still change it in both arms. The pricing page layout test is the design for changing how plans are presented while billing periods stay put; this design is its mirror, moving only the period and leaving the layout alone. Run them one after the other, not at once on the same visitors.
The labelling rules apply in both arms and are general direction, not legal advice. The site's guide to honest persuasion records, from the UK Competition and Markets Authority's and the Advertising Standards Authority's remits, that misleading price presentation attracts regulatory attention, and it asks that wherever a price is framed per month, the total payable and the commitment period are stated beside it. A per-month figure that is only available on an annual commitment says so.
Sample size reasoning
The outcome is a purchase on a page with modest traffic, and the metric mixes fewer large annual payments with more small monthly ones, so its spread is wide. Revenue per visitor is mostly zeros, then two clusters of values, and a modest difference between arms is hard to separate from chance. The inputs are the expected contracted revenue per visitor in the control, how much it varies between visitors, the smallest difference worth acting on and the confidence the team requires. The sample size and significance guide explains how they combine and why a skewed revenue metric needs far more visitors than a conversion rate on the same page.
Name the smallest difference in contracted revenue per visitor worth acting on and check it with that guide. Where the page cannot detect it in a period the business would accept, decide on judgement and record that, as the pricing page guide advises, rather than running a test that can only produce noise.
Duration
Run whole weekly cycles from launch, with entries closing on a date fixed in advance, because business buyers often compare plans on particular days of the week. Then hold the reading open until every start has passed the refund or cooling-off period and the first monthly renewal date, so early cancellations in both arms are counted. Avoid launching across the business's own price change or a renewal campaign, both of which move the same buyers for reasons unrelated to the order.
Reading the results and the decision rule
Read the test once, after the horizon has passed. Compare contracted revenue per visitor with the interval around the difference rather than the point estimate. Then read the guardrails by arm: plan starts per visitor, refunds and early cancellations split by billing period, and billing contacts. Only then look at the plan mix, as an explanation of the revenue result rather than as a result in itself. Write the rule into the plan before launch:
| Result | Decision |
|---|---|
| Annual-first earns reliably more contracted revenue per visitor and every guardrail stays inside its limit | Lead with the annual period |
| The difference is unclear | Keep the current order |
| Refunds or billing complaints pass their limit in either arm | Switch that arm back at once and review the labelling |
| Monthly-first earns reliably more and its guardrails hold | Lead with the monthly period |
Remember what the test cannot say. It does not tell the business whether its annual discount is the right size, and it does not measure full lifetime revenue. The card-required trial test is the design for the separate question of asking for payment details at sign-up.
Pitfalls
A per-month figure that hides an annual commitment is the first trap. If the annual arm shows a monthly-looking number without the total and the commitment beside it, the test measures confusion rather than preference, and the refunds that follow are the business's problem as well as the experiment's. Check both arms against the labelling rules before launch.
Reading before refund windows close is the second. Annual starts arrive quickly and look impressive; the refunds that follow arrive later. A test read in its first weeks almost always flatters the annual arm. Hold the reading to the horizon fixed in the plan.
Plan mix mistaken for revenue is the third. A higher share of annual plans is not the goal; contracted revenue per visitor is. An arm can shift the mix towards annual and still lose, because fewer people start or more of them ask for refunds. The general testing traps apply as well: check that the arms received the visitor shares the plan intended, and resist reading them weekly.
Example: an invented payroll software firm tests its billing order (illustrative)
Quarvenn Payroll tests whether to show annual billing first
Quarvenn Payroll, an invented payroll software firm for small employers, sells one main plan at £40 a month billed monthly or £432 a year billed annually, shown as "£36 a month, billed £432 annually". Its pricing page has always led with monthly. Before launch the team writes two limits: plan starts per visitor may fall by no more than a tenth, and refunds from annual buyers may not exceed 8% of annual starts. Entries run for six weeks, 14,000 new visitors split evenly, and the reading closes after the 30-day refund period and the first monthly renewal have passed.
| Arm | Plan starts | Annual / monthly | Annual refunds | Net contracted revenue | Revenue per visitor |
|---|---|---|---|---|---|
| Monthly first | 210 | 42 / 168 | 2 | £24,000 | £3.43 |
| Annual first | 203 | 71 / 132 | 11 | £31,200 | £4.46 |
Annual-first earns £1.03 more per visitor, with an interval of roughly £0.05 to £2.01, and plan starts fall by only about 3%. But 11 of its 71 annual buyers, about 15%, ask for refunds, well past the 8% limit, and billing contacts rise by a few. Under the rule written before launch, Quarvenn keeps monthly first, reads the refund reasons, and plans a retest only once it understands why annual buyers changed their minds.
Common mistakes
Billing-period tests usually fail on their metric or their horizon rather than their statistics. The experiment library lists the other pricing and offer designs, and the experimentation loop puts this test in sequence with them.
- Judging on plan starts alone. Starts can stay flat while what each is worth changes completely.
- Changing the annual discount at the same time. The test then measures the price, not the order.
- Letting the labelling drift between arms. Both arms show both prices, each labelled with how it is billed.
- Reading before the refund window closes. Early annual payments flatter the annual arm.
- Counting revenue from analytics. Refunds and credits live in billing; read the money there.
- Treating a higher annual share as the win. The mix explains the result; it is not the result.
