Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Experiment Library

Billing Period Default Test

An experiment design for whether a pricing page should show annual or monthly billing first, read on contracted revenue per visitor with refund guardrails.
Experimentintermediate9 min readUpdated

Hypothesis

If the pricing page shows the annual billing period first, with the monthly price visible and each labelled with how it is billed, contracted revenue per pricing-page visitor at a fixed horizon will rise because buyers ready to commit see that option first, unless plan starts fall or more annual buyers ask for refunds. The decision behind it is narrow: show the annual or the monthly billing period first on the pricing page, with both prices always visible and each labelled with how it is billed.

The guide to designing pricing pages frames the choice. Either period can lead, provided both are shown clearly; which one appears first, or whether a switch sets the default, is a presentation choice that can be tested, while the honesty of the labelling is not. The test is worth running because the order works in both directions. Leading with the annual period may help buyers who were ready to commit, and it may also push a hesitant buyer towards a commitment they later regret, or away from the page altogether. Which effect wins depends on the product and its buyers. Write the arms, the expected direction and the limit on each guardrail into the experiment hypothesis template before the first visitor is assigned.

Metrics and guardrails

The primary metric is contracted revenue per pricing-page visitor at a fixed horizon. Take every plan started by visitors in an arm, add the first-term value of each, meaning the monthly payment for a monthly plan and the annual payment for an annual plan, subtract every refund issued within the horizon, and divide by the number of visitors assigned to the arm. Take the figures from the billing system, not the analytics tool: the subscription measurement guide names billing as the source of truth for amounts actually charged after discounts, refunds and credits. The denominator is fixed at assignment, so an arm that loses buyers shows up as lower revenue rather than vanishing from the sum.

A plain conversion rate is the wrong primary metric here, because the treatment changes what each start is worth as well as how many there are. The guardrails keep each effect visible:

  • Plan starts per pricing-page visitor. A first period that scares buyers off can still win on revenue if the remaining buyers pay annually. The conversion rate calculator turns each arm's starts and visitors into a rate to compare against the limit written before launch.
  • Refunds and early cancellations by billing period. Annual buyers who did not mean to commit ask for their money back; monthly buyers who were nudged in the wrong direction leave at the first renewal. Read both by arm and by period, as an early sign of churn.
  • Support contacts about price or billing. A rise in questions such as "why was I charged for a year?" is a warning that the labelling is not working for some buyers, even when it is accurate.

Each of these is a Guardrail Metric with its limit agreed in advance. Renewals beyond the horizon and long-run LTV matter too, but they arrive months later; record them as a follow-up read rather than waiting for them before deciding.

Design

This is an A/B test at visitor level. Assign each visitor at random at their first pricing-page view and hold them in that arm on every return, across sessions and across devices where sign-in allows, because buyers often visit a pricing page several times before choosing. One arm shows the monthly period first, the other the annual period first; whichever is the current page is the control.

Design elementChoiceReason
Unit of randomisationVisitor, from the first pricing-page view, held across sessions and devices where possibleA returning buyer who sees the order flip is confused and contaminates both arms
ArmsMonthly period shown first; annual period shown firstThe only decision on the table is the order
LabellingBoth prices visible in both arms, each stating how it is billed, total and commitment beside any per-month figureHonest labelling is a condition, not a variable
Held constantPrices, discounts, plan names, contents, tiers, copy, layout and checkoutChoosing prices and building tiers belongs to product marketing
ExcludedExisting customers upgrading or renewingTheir choice is shaped by the plan they already hold
Primary metricContracted revenue per pricing-page visitor, from billing, net of refundsCaptures both how many buyers start and what each start is worth

Everything outside the order stays fixed. The pricing and packaging guide owns price levels and annual discounts, and the good-better-best framework owns how tiers are built and which one is the honest default; neither changes during the test. If the page uses a switch, the switch sets the starting position and the buyer can still change it in both arms. The pricing page layout test is the design for changing how plans are presented while billing periods stay put; this design is its mirror, moving only the period and leaving the layout alone. Run them one after the other, not at once on the same visitors.

The labelling rules apply in both arms and are general direction, not legal advice. The site's guide to honest persuasion records, from the UK Competition and Markets Authority's and the Advertising Standards Authority's remits, that misleading price presentation attracts regulatory attention, and it asks that wherever a price is framed per month, the total payable and the commitment period are stated beside it. A per-month figure that is only available on an annual commitment says so.

Sample size reasoning

The outcome is a purchase on a page with modest traffic, and the metric mixes fewer large annual payments with more small monthly ones, so its spread is wide. Revenue per visitor is mostly zeros, then two clusters of values, and a modest difference between arms is hard to separate from chance. The inputs are the expected contracted revenue per visitor in the control, how much it varies between visitors, the smallest difference worth acting on and the confidence the team requires. The sample size and significance guide explains how they combine and why a skewed revenue metric needs far more visitors than a conversion rate on the same page.

Name the smallest difference in contracted revenue per visitor worth acting on and check it with that guide. Where the page cannot detect it in a period the business would accept, decide on judgement and record that, as the pricing page guide advises, rather than running a test that can only produce noise.

Duration

Run whole weekly cycles from launch, with entries closing on a date fixed in advance, because business buyers often compare plans on particular days of the week. Then hold the reading open until every start has passed the refund or cooling-off period and the first monthly renewal date, so early cancellations in both arms are counted. Avoid launching across the business's own price change or a renewal campaign, both of which move the same buyers for reasons unrelated to the order.

Reading the results and the decision rule

Read the test once, after the horizon has passed. Compare contracted revenue per visitor with the interval around the difference rather than the point estimate. Then read the guardrails by arm: plan starts per visitor, refunds and early cancellations split by billing period, and billing contacts. Only then look at the plan mix, as an explanation of the revenue result rather than as a result in itself. Write the rule into the plan before launch:

ResultDecision
Annual-first earns reliably more contracted revenue per visitor and every guardrail stays inside its limitLead with the annual period
The difference is unclearKeep the current order
Refunds or billing complaints pass their limit in either armSwitch that arm back at once and review the labelling
Monthly-first earns reliably more and its guardrails holdLead with the monthly period

Remember what the test cannot say. It does not tell the business whether its annual discount is the right size, and it does not measure full lifetime revenue. The card-required trial test is the design for the separate question of asking for payment details at sign-up.

Pitfalls

A per-month figure that hides an annual commitment is the first trap. If the annual arm shows a monthly-looking number without the total and the commitment beside it, the test measures confusion rather than preference, and the refunds that follow are the business's problem as well as the experiment's. Check both arms against the labelling rules before launch.

Reading before refund windows close is the second. Annual starts arrive quickly and look impressive; the refunds that follow arrive later. A test read in its first weeks almost always flatters the annual arm. Hold the reading to the horizon fixed in the plan.

Plan mix mistaken for revenue is the third. A higher share of annual plans is not the goal; contracted revenue per visitor is. An arm can shift the mix towards annual and still lose, because fewer people start or more of them ask for refunds. The general testing traps apply as well: check that the arms received the visitor shares the plan intended, and resist reading them weekly.

Example: an invented payroll software firm tests its billing order (illustrative)

Illustrative example — not a real result

Quarvenn Payroll tests whether to show annual billing first

Quarvenn Payroll, an invented payroll software firm for small employers, sells one main plan at £40 a month billed monthly or £432 a year billed annually, shown as "£36 a month, billed £432 annually". Its pricing page has always led with monthly. Before launch the team writes two limits: plan starts per visitor may fall by no more than a tenth, and refunds from annual buyers may not exceed 8% of annual starts. Entries run for six weeks, 14,000 new visitors split evenly, and the reading closes after the 30-day refund period and the first monthly renewal have passed.

ArmPlan startsAnnual / monthlyAnnual refundsNet contracted revenueRevenue per visitor
Monthly first21042 / 1682£24,000£3.43
Annual first20371 / 13211£31,200£4.46

Annual-first earns £1.03 more per visitor, with an interval of roughly £0.05 to £2.01, and plan starts fall by only about 3%. But 11 of its 71 annual buyers, about 15%, ask for refunds, well past the 8% limit, and billing contacts rise by a few. Under the rule written before launch, Quarvenn keeps monthly first, reads the refund reasons, and plans a retest only once it understands why annual buyers changed their minds.

Common mistakes

Billing-period tests usually fail on their metric or their horizon rather than their statistics. The experiment library lists the other pricing and offer designs, and the experimentation loop puts this test in sequence with them.

  • Judging on plan starts alone. Starts can stay flat while what each is worth changes completely.
  • Changing the annual discount at the same time. The test then measures the price, not the order.
  • Letting the labelling drift between arms. Both arms show both prices, each labelled with how it is billed.
  • Reading before the refund window closes. Early annual payments flatter the annual arm.
  • Counting revenue from analytics. Refunds and credits live in billing; read the money there.
  • Treating a higher annual share as the win. The mix explains the result; it is not the result.

Frequently asked questions

Why not judge the test on plan starts, like a normal pricing page test?

Because the order of billing periods changes what each start is worth. Showing the annual period first can move buyers from small monthly payments to one large annual payment while the number of starts barely moves, or falls slightly. Contracted revenue per visitor, taken from the billing system and net of refunds, captures both effects at once; plan starts stay on the page as a guardrail so a fall in buyers is never hidden by larger payments.

Can the annual arm show only the per-month equivalent to make it look cheaper?

No. In both arms both billing periods are shown clearly and each price says how it is billed, with the total payable and the commitment beside any per-month figure that needs an annual contract. The site's pricing guides record that UK regulators treat misleading price presentation as a consumer protection matter. The honesty of the labelling is a fixed condition of the test, never a variable. This is general direction, not legal advice.

What if our pricing page has too little traffic for this test?

Then decide on judgement and record it as such, which is what the pricing page guide advises for low-traffic pages. Research with buyers, such as interviews or a usability study, can show whether they understand both periods. A test that cannot detect the difference worth acting on in a period the business would accept is not worth running.

Terms used on this page

Tools to try

Frameworks

Where this page is used

Sources

  1. Competition and Markets Authority — GOV.UK
  2. Advertising Standards Authority — ASA

Related