Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Experiment Library

Upgrade Prompt Test

An experiment design for the prompt customers meet at a plan limit: accounts split between the current block and an explained prompt, read on upgrades.
Experimentintermediate10 min readUpdated

Hypothesis

If existing accounts that reach a plan limit are split at random between the current block and a prompt that names the action the customer attempted, the plan that permits it, that plan's recurring price and today's charge, with a way to carry on within the current plan, more of them will upgrade within the fixed window, because the questions that sent them to support or away are answered at the moment of need, unless the clearer prompt moves customers into a plan that does not fit and downgrades, refunds or cancellations rise. The decision is concrete: keep the current block, or replace it with the explained prompt for that limit.

The upgrade path guide designs that moment and everything after it. It says to test prompt changes but leaves the plan to this page, and its example reads a redrawn prompt quarter on quarter with no controlled comparison behind it. The pricing page design guide treats upgraders as a journey of their own, upgrading from inside the account. The pricing page layout test leaves upgraders out because they follow a different path, and the billing period default test because the plan they already hold shapes their choice. Record the arms, the window and each guardrail's limit in the experiment hypothesis template before assignment begins.

Metrics and guardrails

The primary metric is upgrades per account that reached the limit. For each arm, count the accounts that moved to a higher plan within the fixed window after first reaching the limit, and divide by every account assigned to that arm, including those that never saw the prompt again; a rate over clicks would let each prompt set its own denominator. Count each upgrade once, from billing: the guide to measurement for subscription businesses makes billing the record of every plan change, joined to analytics and the CRM by one customer identifier, so a later downgrade or cancellation can be traced to the upgrade.

Three guardrails, each a Guardrail Metric with a limit written before launch, catch a prompt that wins for the wrong reason:

  • Downgrades and refunds among each arm's upgraders. Counted from each upgrade to the end of a follow-on window that starts when the account's fixed window closes. A prompt that rushes customers into a plan they do not need shows up here.
  • Cancellations per assigned account. Counted from assignment to the end of the same follow-on window, across everyone assigned, so that a prompt that drives customers away is caught as well as one that rushes them. Read it by Cohort of the week each account reached the limit, beside the usual Churn Rate.
  • Support contacts about billing or plans per assigned account. A sign that the prompt, the upgrade screen and the invoice disagree.

Design

Randomise at account level, at the moment of need. The first time any user of an eligible account reaches the limit under test, assign the account at random to the current block or the explained prompt, store the assignment on the account and serve the same version to every user on every later visit. The A/B testing guide names the account as a possible Unit of Randomisation for account-based products, so that colleagues see the same thing. Free and paid plans can both enter if the limit is the same; trials stay out, because a trial's first payment is a different moment with its own path.

The explained prompt carries the elements the upgrade path guide asks of a prompt (the action attempted, the plan that permits it, what that plan costs, how to move to it and, where the product allows, a way to carry on within the current plan) and adds today's charge, which that guide shows on the upgrade screen. Where the plans follow good-better-best tiering, the upgrade path guide treats the next tier, not the top one, as usually the natural step, so the prompt offers that. Both arms lead to the same upgrade screen inside the account, which shows the full cost before the customer confirms. If no such screen exists yet, build it first: an existing customer sent to the public Pricing Page is a fault to fix, not an arm to test.

Design elementChoiceReason
Unit of randomisationAccount, assigned the first time any user reaches the limitThe plan and the bill belong to the account
ArmsCurrent block; explained promptOne change at the moment of need
Held constantPrices, plans, limits, billing rules, upgrade screen, way down, upgrade emailsOnly the prompt can explain the difference
DeliveryServed from the server, assignment stored on the accountThe prompt shows a price and a charge; no flash of the other version
Primary metricUpgrades per account that reached the limit, within a fixed window, from billingCounts every assigned account, once
GuardrailsDowngrades and refunds; cancellations; billing contactsCatches revenue moved forward and trust moved back

The guide to client-side and server-side testing keeps tests that show a price, a charge or a total away from browser-side page editing, as general practice, because the words on screen and the charge billing makes can then disagree.

The way down sits beside the way up in both arms: the guide to designing the cancellation experience records that regulators such as the UK's Competition and Markets Authority (CMA) treat easy cancellation of subscriptions as consumer protection, and the pricing page guide notes that the CMA acts on misleading pricing, fees that surface late among them. The guide to honest persuasion rules out deadlines that are not real and a higher plan pre-selected. Upgrade emails continue unchanged in both arms; the upgrade path guide treats them as marketing under the ICO's direct marketing guidance, sent only where marketing is allowed. None of this is legal advice.

Sample size reasoning

Only accounts that reach the limit can enter, so the sample builds at the rate accounts reach it, not at the rate of traffic. The inputs are how many accounts reach the limit in a typical period, the share that upgrade within the window under the current block, and the smallest rise worth building. The sample size and significance guide sets out how those inputs combine for an A/B Test read on a rate, and the conversion rate calculator turns each arm's counts into comparable rates. Size the guardrails too, so that a meaningful rise in downgrades or refunds would also show; a test large enough to see more upgrades may still be too small for that, and if so the plan should say so before launch.

If the limit is reached too rarely, test a bolder prompt, or ship the change and read it against a prediction written in advance, as the guide to conversion work without A/B tests describes. If most accounts reach it soon after joining, the upgrade path guide puts the fix in the packaging, not the prompt, and the pricing and packaging guide separates fences that follow value from fences that read as a penalty.

Duration

Enrol accounts for whole weeks until the planned number in each arm have reached the limit. Read the primary metric once every assigned account's fixed window has closed, and the guardrails once the follow-on window, covering at least a full billing period, has closed too, so that the first full charge on the new plan has landed. Set both windows before launch from the business's own billing history: how long accounts that reached the limit in the past took to upgrade, and how long downgrades and refunds took to arrive. The sample size guide cites Evan Miller's essay against peeking to explain why a result checked repeatedly and stopped when it looks good reports noise; the reading happens once, on the dates written down.

Reading the results and the decision rule

Check the split first: each arm should hold the share of accounts the plan intended. Then read upgrades per account that reached the limit, with the range around the difference, not a single figure; the guide to reading small numbers shows how far rates built on modest counts can move by chance. Read the guardrails against the limits written before launch, then the step rates to see where each arm lost customers. Fix the decision rule in the plan before launch:

ResultDecision
Upgrades per account are reliably higher with the explained prompt, outside the range set before launch, and no guardrail passes its limitShip the explained prompt for this limit
The difference sits within the range set before launchKeep the current block, or test a bolder prompt
Upgrades rise, but downgrades, refunds, cancellations or billing contacts pass their limitsTreat the result as revenue moved forward; redesign the prompt rather than ship it
Upgrades are lower with the explained promptKeep the current block and read the step rates to find where the prompt lost customers

Record the result, the step rates and every guardrail in the experiment results report template, whichever way it falls; the experimentation loop closes there, and the next limit gets its own test.

Pitfalls

Assigning users instead of accounts is the first trap, because the account's single upgrade is then credited to whichever arm its payer was in. The second is a prompt that flashes, swapped in by a browser-side tool after the page draws. The third is counting upgrades from page loads or clicks rather than from billing. The fourth is a difference that is not the prompt: a discount, a different upgrade screen or a different way down in one arm. The fifth is reading before the windows close, which flatters whichever arm moves customers sooner rather than more often, and the sixth is judging on upgrades alone. The guide to common testing pitfalls covers the general traps, starting with assignment and tracking faults. Customers who move to leave rather than upgrade have their own design in the cancellation save offer test, and the guide to product-qualified leads covers accounts at a limit that a seller would serve better than a prompt.

Example: an invented visit-planning app for farriers tests an explained prompt at its saved-address limit (illustrative)

Illustrative example — not a real result

Hupnadrel Rounds tests an explained prompt at its address limit

Hupnadrel Rounds, an invented visit-planning app for farriers, sells a Starter plan with up to 400 saved customer addresses and a Crew plan with no address limit. When a Starter account tries to save its 401st address, the app shows "Address limit reached. Upgrade to continue." with a link to the plan page inside the account. Last quarter about 15% of the accounts that hit that block upgraded within 21 days.

The explained prompt says that the account has reached Starter's 400 addresses, that Crew removes the limit, and shows Crew's monthly price and today's charge for the rest of the billing period, with a link to archive addresses no longer visited. The plan, written before launch, looks for a rise from about 15% to 22%, which the sample size method puts at about 480 accounts per arm; about 120 accounts reach the limit each week, so enrolment runs for 8 weeks. A difference whose range includes zero, or sits mostly below the 7-point rise sought, counts as within the range that keeps the current block. The fixed window is 21 days from first reaching the limit; the follow-on window is the 60 days after it. The guardrail limits allow at most 3 points more downgrades and refunds among upgraders, 2 points more cancellations per assigned account, and no more billing contacts than the current block. The plan notes that so few upgraders can reveal only a large rise in downgrades and refunds, so that guardrail is judged on its estimate and followed monthly after rollout.

ArmAccountsOpened upgrade screenUpgraded within 21 daysUpgrades per accountDowngrades or refunds among upgradersCancellationsBilling contacts
Current block4811817515.6%7 (9.3%)32 (6.7%)68
Explained prompt47923211524.0%11 (9.6%)26 (5.4%)24

The split matches the plan. The explained prompt raises upgrades per account by about 8.4 points, with an interval of roughly 3.4 to 13.4 points. Downgrades and refunds among upgraders read 9.3% and 9.6%, an estimate inside the 3-point limit, though the range cannot rule out a rise past it, so the team keeps reading it monthly after rollout; the change in cancellations runs from about 4.2 points lower to 1.8 points higher, inside the limit, and billing contacts fall by about two-thirds. Hupnadrel ships the explained prompt and tests its next limit afresh.

Common mistakes

The experiment library holds the other designs.

  • Letting trial accounts in. Their first payment is a separate journey.
  • Adding a discount to one arm. The test then measures the money, not the explanation.
  • Ending enrolment the week the arms look different. Stop at the planned number.
  • Comparing with last quarter instead of the other arm. A quarter-on-quarter rise is directional at best.
  • Dropping accounts that never return after the block. They stay in the denominator; a block that drives them away is part of what the test measures.
  • Copying one limit's answer to every limit. Each fence meets customers at a different moment; test each one.

Frequently asked questions

Why assign accounts rather than individual users?

Because the plan, the limit and the bill belong to the account. If two colleagues are assigned separately, one may see the explained prompt and the other the bare block on the same day, and the upgrade decision, which belongs to whoever pays, mixes both versions. Assigning the account once, the first time any of its users reaches the limit, keeps everyone in the account on one version and lets the upgrade be counted where billing records it, once per account.

Should the explained prompt offer a discount for upgrading now?

Not in this test. A discount changes what the customer pays, which turns a test of the prompt into a test of the price and leaves nobody able to say whether the explanation or the money did the work. Showing different plan prices to comparable customers also raises fairness and trust questions that belong to pricing strategy. Keep prices, plans and billing identical in both arms, and settle any promotion as a separate decision with whoever owns pricing.

What if only a handful of accounts reach the limit?

Then the test fills slowly, because only accounts that reach the moment can enter it. Honest options remain: test a bolder change to the prompt, so that a larger effect can be seen, or ship the change and read it against a prediction written beforehand, labelled as directional. If most accounts reach the limit soon after joining, look at where the plans are fenced before testing any prompt.

Terms used on this page

Tools to try

Frameworks

Also related

Sources

  1. Competition and Markets Authority — GOV.UK
  2. Direct marketing guidance — Information Commissioner's Office (UK)
  3. How Not To Run an A/B Test — Evan Miller

Related