Hypothesis
If existing accounts that reach a plan limit are split at random between the current block and a prompt that names the action the customer attempted, the plan that permits it, that plan's recurring price and today's charge, with a way to carry on within the current plan, more of them will upgrade within the fixed window, because the questions that sent them to support or away are answered at the moment of need, unless the clearer prompt moves customers into a plan that does not fit and downgrades, refunds or cancellations rise. The decision is concrete: keep the current block, or replace it with the explained prompt for that limit.
The upgrade path guide designs that moment and everything after it. It says to test prompt changes but leaves the plan to this page, and its example reads a redrawn prompt quarter on quarter with no controlled comparison behind it. The pricing page design guide treats upgraders as a journey of their own, upgrading from inside the account. The pricing page layout test leaves upgraders out because they follow a different path, and the billing period default test because the plan they already hold shapes their choice. Record the arms, the window and each guardrail's limit in the experiment hypothesis template before assignment begins.
Metrics and guardrails
The primary metric is upgrades per account that reached the limit. For each arm, count the accounts that moved to a higher plan within the fixed window after first reaching the limit, and divide by every account assigned to that arm, including those that never saw the prompt again; a rate over clicks would let each prompt set its own denominator. Count each upgrade once, from billing: the guide to measurement for subscription businesses makes billing the record of every plan change, joined to analytics and the CRM by one customer identifier, so a later downgrade or cancellation can be traced to the upgrade.
Three guardrails, each a Guardrail MetricA metric an experiment must not worsen beyond an agreed bound, watched beside the primary metric that decides the test. with a limit written before launch, catch a prompt that wins for the wrong reason:
- Downgrades and refunds among each arm's upgraders. Counted from each upgrade to the end of a follow-on window that starts when the account's fixed window closes. A prompt that rushes customers into a plan they do not need shows up here.
- Cancellations per assigned account. Counted from assignment to the end of the same follow-on window, across everyone assigned, so that a prompt that drives customers away is caught as well as one that rushes them. Read it by CohortA group of customers who share a starting event and period, tracked together over time. of the week each account reached the limit, beside the usual Churn RateShare of customers at the start of a period who left during it..
- Support contacts about billing or plans per assigned account. A sign that the prompt, the upgrade screen and the invoice disagree.
Design
Randomise at account level, at the moment of need. The first time any user of an eligible account reaches the limit under test, assign the account at random to the current block or the explained prompt, store the assignment on the account and serve the same version to every user on every later visit. The A/B testing guide names the account as a possible Unit of RandomisationWhat a test assigns at random to control or variant, such as a visitor, session, account or region; it should match the level at which the metric is counted. for account-based products, so that colleagues see the same thing. Free and paid plans can both enter if the limit is the same; trials stay out, because a trial's first payment is a different moment with its own path.
The explained prompt carries the elements the upgrade path guide asks of a prompt (the action attempted, the plan that permits it, what that plan costs, how to move to it and, where the product allows, a way to carry on within the current plan) and adds today's charge, which that guide shows on the upgrade screen. Where the plans follow good-better-best tiering, the upgrade path guide treats the next tier, not the top one, as usually the natural step, so the prompt offers that. Both arms lead to the same upgrade screen inside the account, which shows the full cost before the customer confirms. If no such screen exists yet, build it first: an existing customer sent to the public Pricing PageThe page where buyers compare plans and prices, treated as a conversion step that shows who each plan is for, the complete cost and the next step. is a fault to fix, not an arm to test.
| Design element | Choice | Reason |
|---|---|---|
| Unit of randomisation | Account, assigned the first time any user reaches the limit | The plan and the bill belong to the account |
| Arms | Current block; explained prompt | One change at the moment of need |
| Held constant | Prices, plans, limits, billing rules, upgrade screen, way down, upgrade emails | Only the prompt can explain the difference |
| Delivery | Served from the server, assignment stored on the account | The prompt shows a price and a charge; no flash of the other version |
| Primary metric | Upgrades per account that reached the limit, within a fixed window, from billing | Counts every assigned account, once |
| Guardrails | Downgrades and refunds; cancellations; billing contacts | Catches revenue moved forward and trust moved back |
The guide to client-side and server-side testing keeps tests that show a price, a charge or a total away from browser-side page editing, as general practice, because the words on screen and the charge billing makes can then disagree.
The way down sits beside the way up in both arms: the guide to designing the cancellation experience records that regulators such as the UK's Competition and Markets Authority (CMA) treat easy cancellation of subscriptions as consumer protection, and the pricing page guide notes that the CMA acts on misleading pricing, fees that surface late among them. The guide to honest persuasion rules out deadlines that are not real and a higher plan pre-selected. Upgrade emails continue unchanged in both arms; the upgrade path guide treats them as marketing under the ICO's direct marketing guidance, sent only where marketing is allowed. None of this is legal advice.
Sample size reasoning
Only accounts that reach the limit can enter, so the sample builds at the rate accounts reach it, not at the rate of traffic. The inputs are how many accounts reach the limit in a typical period, the share that upgrade within the window under the current block, and the smallest rise worth building. The sample size and significance guide sets out how those inputs combine for an A/B TestControlled comparison of two variants. read on a rate, and the conversion rate calculator turns each arm's counts into comparable rates. Size the guardrails too, so that a meaningful rise in downgrades or refunds would also show; a test large enough to see more upgrades may still be too small for that, and if so the plan should say so before launch.
If the limit is reached too rarely, test a bolder prompt, or ship the change and read it against a prediction written in advance, as the guide to conversion work without A/B tests describes. If most accounts reach it soon after joining, the upgrade path guide puts the fix in the packaging, not the prompt, and the pricing and packaging guide separates fences that follow value from fences that read as a penalty.
Duration
Enrol accounts for whole weeks until the planned number in each arm have reached the limit. Read the primary metric once every assigned account's fixed window has closed, and the guardrails once the follow-on window, covering at least a full billing period, has closed too, so that the first full charge on the new plan has landed. Set both windows before launch from the business's own billing history: how long accounts that reached the limit in the past took to upgrade, and how long downgrades and refunds took to arrive. The sample size guide cites Evan Miller's essay against peeking to explain why a result checked repeatedly and stopped when it looks good reports noise; the reading happens once, on the dates written down.
Reading the results and the decision rule
Check the split first: each arm should hold the share of accounts the plan intended. Then read upgrades per account that reached the limit, with the range around the difference, not a single figure; the guide to reading small numbers shows how far rates built on modest counts can move by chance. Read the guardrails against the limits written before launch, then the step rates to see where each arm lost customers. Fix the decision rule in the plan before launch:
| Result | Decision |
|---|---|
| Upgrades per account are reliably higher with the explained prompt, outside the range set before launch, and no guardrail passes its limit | Ship the explained prompt for this limit |
| The difference sits within the range set before launch | Keep the current block, or test a bolder prompt |
| Upgrades rise, but downgrades, refunds, cancellations or billing contacts pass their limits | Treat the result as revenue moved forward; redesign the prompt rather than ship it |
| Upgrades are lower with the explained prompt | Keep the current block and read the step rates to find where the prompt lost customers |
Record the result, the step rates and every guardrail in the experiment results report template, whichever way it falls; the experimentation loop closes there, and the next limit gets its own test.
Pitfalls
Assigning users instead of accounts is the first trap, because the account's single upgrade is then credited to whichever arm its payer was in. The second is a prompt that flashes, swapped in by a browser-side tool after the page draws. The third is counting upgrades from page loads or clicks rather than from billing. The fourth is a difference that is not the prompt: a discount, a different upgrade screen or a different way down in one arm. The fifth is reading before the windows close, which flatters whichever arm moves customers sooner rather than more often, and the sixth is judging on upgrades alone. The guide to common testing pitfalls covers the general traps, starting with assignment and tracking faults. Customers who move to leave rather than upgrade have their own design in the cancellation save offer test, and the guide to product-qualified leads covers accounts at a limit that a seller would serve better than a prompt.
Example: an invented visit-planning app for farriers tests an explained prompt at its saved-address limit (illustrative)
Hupnadrel Rounds tests an explained prompt at its address limit
Hupnadrel Rounds, an invented visit-planning app for farriers, sells a Starter plan with up to 400 saved customer addresses and a Crew plan with no address limit. When a Starter account tries to save its 401st address, the app shows "Address limit reached. Upgrade to continue." with a link to the plan page inside the account. Last quarter about 15% of the accounts that hit that block upgraded within 21 days.
The explained prompt says that the account has reached Starter's 400 addresses, that Crew removes the limit, and shows Crew's monthly price and today's charge for the rest of the billing period, with a link to archive addresses no longer visited. The plan, written before launch, looks for a rise from about 15% to 22%, which the sample size method puts at about 480 accounts per arm; about 120 accounts reach the limit each week, so enrolment runs for 8 weeks. A difference whose range includes zero, or sits mostly below the 7-point rise sought, counts as within the range that keeps the current block. The fixed window is 21 days from first reaching the limit; the follow-on window is the 60 days after it. The guardrail limits allow at most 3 points more downgrades and refunds among upgraders, 2 points more cancellations per assigned account, and no more billing contacts than the current block. The plan notes that so few upgraders can reveal only a large rise in downgrades and refunds, so that guardrail is judged on its estimate and followed monthly after rollout.
| Arm | Accounts | Opened upgrade screen | Upgraded within 21 days | Upgrades per account | Downgrades or refunds among upgraders | Cancellations | Billing contacts |
|---|---|---|---|---|---|---|---|
| Current block | 481 | 181 | 75 | 15.6% | 7 (9.3%) | 32 (6.7%) | 68 |
| Explained prompt | 479 | 232 | 115 | 24.0% | 11 (9.6%) | 26 (5.4%) | 24 |
The split matches the plan. The explained prompt raises upgrades per account by about 8.4 points, with an interval of roughly 3.4 to 13.4 points. Downgrades and refunds among upgraders read 9.3% and 9.6%, an estimate inside the 3-point limit, though the range cannot rule out a rise past it, so the team keeps reading it monthly after rollout; the change in cancellations runs from about 4.2 points lower to 1.8 points higher, inside the limit, and billing contacts fall by about two-thirds. Hupnadrel ships the explained prompt and tests its next limit afresh.
Common mistakes
The experiment library holds the other designs.
- Letting trial accounts in. Their first payment is a separate journey.
- Adding a discount to one arm. The test then measures the money, not the explanation.
- Ending enrolment the week the arms look different. Stop at the planned number.
- Comparing with last quarter instead of the other arm. A quarter-on-quarter rise is directional at best.
- Dropping accounts that never return after the block. They stay in the denominator; a block that drives them away is part of what the test measures.
- Copying one limit's answer to every limit. Each fence meets customers at a different moment; test each one.
