Hypothesis
If the trial asks for payment details at sign-up, fewer visitors will start a trial but a larger share will convert to paid, and counting paying customers per visitor rather than per trial shows which effect is larger. The decision behind it: ask for payment details at trial sign-up, or let people start without them.
Both halves of the prediction are close to certain on their own, which is why the test exists. A card step adds effort and asks for trust before the product has earned any, so trial starts fall. The people who still start are more committed, and a trial that rolls into a subscription unless cancelled converts some people who simply did not act, so the per-trial rate rises. The question is the product of the two, measured on one denominator. The pricing and packaging guide explains where trials sit among the other ways to let people try a product; this test changes only whether payment details are taken at the start. Record the arms, the renewal wording both arms will use and the guardrail limits in the experiment hypothesis template before launch.
Metrics and guardrails
The primary metric is paying customers per sign-up page visitor, counted at a fixed horizon that covers the trial and the first paid period, and net of refunds and chargebacks. The denominator is every visitor assigned to the arm, whether or not they started a trial. The Trial-to-Paid Conversion RateShare of free trials that became paying subscriptions. and the trial-start CVRShare of visits or clicks that convert. are both reported, because together they explain the result, but neither decides it.
The guardrails exist because a card-required trial can win the primary metric in ways the business would not want. The refund and chargeback rate shows whether people are being charged who did not mean to pay; a rise here is a direct cost and a warning about the renewal terms. Early churn, read after the first paid period, shows whether the extra customers stay. Cancellation complaints, logged by support with the arm attached, catch confusion that never reaches a refund. Regulators such as the Federal Trade Commission, in its business guidance, and the Competition and Markets Authority treat clear terms and easy cancellation for subscriptions as consumer protection matters; read their guidance as general direction, not legal advice, and set the guardrail limits with that in mind.
Design
Randomise at visitor level, on first view of the sign-up page, and keep the assignment across sessions and devices where possible. The control is the card-free trial. The challenger adds a payment step at sign-up. Both arms state the trial length, the price that follows and how to cancel in the same words and with the same prominence, and both send the same reminder before the first charge. Only the timing of the payment step differs.
| Design element | Choice | Reason |
|---|---|---|
| Unit of randomisation | Visitor to the sign-up page, assigned on first view | The card step changes who starts a trial, so assign before it |
| Arms | Card-free trial versus card required at sign-up | One change: when payment details are taken |
| Renewal terms | Identical wording, prominence and reminder in both arms | Clarity is non-negotiable, not a variable |
| Primary metric | Paying customers per visitor at a fixed horizon, net of refunds | Per-trial rates flatter the card-required arm |
| Payment step design | Standard checkout fields, nothing extra | Tests the step, not a badly built form |
| Arm record | Stored on the account and in billing | Lets refunds and complaints be joined to the arm |
The card step itself is a short form, and its usability affects the result. The forms guide covers field design, error handling and trust cues; build the payment step to that standard before the test, so that the experiment measures asking for the card rather than asking for it badly. The lead form length test is the related design that removes fields from a form; this one adds a payment step and is read on paying customers. The academy exercise on designing a card-at-sign-up test for a podcast hosting service practises writing this design with refunds and early churn as guardrails, and explaining why it should not share sign-ups with a trial length test.
The card step is one screen in a longer flow, and the guide to sign-up and trial-start flows covers the rest of it, from what to ask at sign-up and when to verify the address to the first screen after the account is created.
Sample size reasoning
The metric's numerator is paying customers, which is rare per visitor, so the sample is large and the test long. The per-visitor framing needs more traffic than a per-trial comparison would, because the event is rarer, but it is still the right one: a per-trial comparison would reach significance sooner on the wrong question. The inputs are the current paying customers per visitor, the smallest difference that would change the decision and the confidence required; the sample size and significance guide explains how they set the visitors each arm needs, and the conversion rate calculator helps derive the baseline rate from your own visitor and customer counts.
If traffic cannot support the per-visitor read, do not fall back to per-trial rates. Run longer, test on the highest-traffic entry route only, or accept that the test can detect only a large difference and say so in the plan.
Duration
Enrol to the planned sample across full weekly cycles, then wait for a horizon that covers the trial and the first paid period for the last visitors enrolled, so that refunds and early cancellations have time to appear. Set the end date before launch. Avoid running across a price change or a promotion that changes the offer in one arm's period more than the other's.
Reading the results and the decision rule
Read once, at the planned end. Compare paying customers per visitor, net of refunds, with its confidence interval; then trial starts per visitor and the trial-to-paid rate, to understand the mechanism; then each guardrail. The rule, written before launch:
| Result | Decision |
|---|---|
| Card-required arm has more paying customers per visitor and refunds, chargebacks and complaints stay within limits | Require the card at sign-up |
| Card-required arm has more paying customers per visitor but a guardrail breaks | Keep the card-free trial; fix renewal clarity before any retest |
| No reliable difference per visitor | Keep the card-free trial, which serves more people at the same yield |
| Card-free arm has more paying customers per visitor | Keep the card-free trial |
Carry the result into acquisition planning: a change in paying customers per visitor changes CACCost to acquire one new customer. for every channel that sends traffic, and the LTV and CAC guide shows how to check whether the customers gained are worth what they cost to acquire.
A photo editing app tests asking for the card
Lumenpath, an invented online photo-editing service, splits 80,000 sign-up page visitors evenly between its card-free trial and a card-required version with identical renewal wording. In the card-free arm 12% start a trial and 15% of those pay, giving 720 paying customers, or 1.80% per visitor. In the card-required arm 5% start and 40% pay, giving 800 paying customers, or 2.00% per visitor; the 95% interval for the difference is roughly 0.01 to 0.39 points. But refunds and chargebacks on first charges are 7% in the card-required arm against 2% in the card-free arm, above the 4% limit the team agreed. Under the rule's second row, Lumenpath keeps the card-free trial and plans a clearer pre-charge reminder before any retest.
Pitfalls
Reading the trial-to-paid rate is the first pitfall, and the most persuasive, because the card-required arm nearly always wins it. Unclear renewal terms are the second: if the challenger's price or renewal date is less visible than the control's, the test measures confusion, and the guardrails will eventually show it as refunds and complaints. Keep the wording identical and prominent in both arms. Involuntary conversions counted as wins are the third. Count customers net of refunds and chargebacks, and treat a cluster of cancellations immediately after the first charge as a guardrail signal, not as normal churn.
Two operational traps matter too. Payment failures at the card step can look like reluctance; log declines separately from abandonment. And support may handle refund requests differently once they know a test is running; give them one written policy for both arms. The academy capstone on a growth operating system for a vintage clothing resale app practises planning trial and save offer experiments for a seller plan as one part of a year's growth system.
Common mistakes
Card tests are usually lost in the denominator. The free trial length test changes how long the trial lasts rather than how it starts, the pricing page layout test changes how plans are presented, and the experiment library lists every design.
- Reporting per-trial conversion as the result. The card step changes who starts; count per visitor.
- Making renewal terms quieter in one arm. Clarity is held constant, never tested.
- Counting refunded customers as conversions. Net them out at the horizon.
- Reading before the first paid period ends. Refunds and early churn arrive after the first charge.
- Testing with a clumsy payment form. Build the step well first, or the test measures the form.
- Treating regulator guidance as a checklist. It is general direction; take qualified advice.
