Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Experiment Library

Card-Required Trial Test

An experiment design for whether a free trial should ask for payment details at sign-up, judged on paying customers per visitor with refund guardrails.
Experimentadvanced8 min readUpdated

Hypothesis

If the trial asks for payment details at sign-up, fewer visitors will start a trial but a larger share will convert to paid, and counting paying customers per visitor rather than per trial shows which effect is larger. The decision behind it: ask for payment details at trial sign-up, or let people start without them.

Both halves of the prediction are close to certain on their own, which is why the test exists. A card step adds effort and asks for trust before the product has earned any, so trial starts fall. The people who still start are more committed, and a trial that rolls into a subscription unless cancelled converts some people who simply did not act, so the per-trial rate rises. The question is the product of the two, measured on one denominator. The pricing and packaging guide explains where trials sit among the other ways to let people try a product; this test changes only whether payment details are taken at the start. Record the arms, the renewal wording both arms will use and the guardrail limits in the experiment hypothesis template before launch.

Metrics and guardrails

The primary metric is paying customers per sign-up page visitor, counted at a fixed horizon that covers the trial and the first paid period, and net of refunds and chargebacks. The denominator is every visitor assigned to the arm, whether or not they started a trial. The Trial-to-Paid Conversion Rate and the trial-start CVR are both reported, because together they explain the result, but neither decides it.

The guardrails exist because a card-required trial can win the primary metric in ways the business would not want. The refund and chargeback rate shows whether people are being charged who did not mean to pay; a rise here is a direct cost and a warning about the renewal terms. Early churn, read after the first paid period, shows whether the extra customers stay. Cancellation complaints, logged by support with the arm attached, catch confusion that never reaches a refund. Regulators such as the Federal Trade Commission, in its business guidance, and the Competition and Markets Authority treat clear terms and easy cancellation for subscriptions as consumer protection matters; read their guidance as general direction, not legal advice, and set the guardrail limits with that in mind.

Design

Randomise at visitor level, on first view of the sign-up page, and keep the assignment across sessions and devices where possible. The control is the card-free trial. The challenger adds a payment step at sign-up. Both arms state the trial length, the price that follows and how to cancel in the same words and with the same prominence, and both send the same reminder before the first charge. Only the timing of the payment step differs.

Design elementChoiceReason
Unit of randomisationVisitor to the sign-up page, assigned on first viewThe card step changes who starts a trial, so assign before it
ArmsCard-free trial versus card required at sign-upOne change: when payment details are taken
Renewal termsIdentical wording, prominence and reminder in both armsClarity is non-negotiable, not a variable
Primary metricPaying customers per visitor at a fixed horizon, net of refundsPer-trial rates flatter the card-required arm
Payment step designStandard checkout fields, nothing extraTests the step, not a badly built form
Arm recordStored on the account and in billingLets refunds and complaints be joined to the arm

The card step itself is a short form, and its usability affects the result. The forms guide covers field design, error handling and trust cues; build the payment step to that standard before the test, so that the experiment measures asking for the card rather than asking for it badly. The lead form length test is the related design that removes fields from a form; this one adds a payment step and is read on paying customers. The academy exercise on designing a card-at-sign-up test for a podcast hosting service practises writing this design with refunds and early churn as guardrails, and explaining why it should not share sign-ups with a trial length test.

The card step is one screen in a longer flow, and the guide to sign-up and trial-start flows covers the rest of it, from what to ask at sign-up and when to verify the address to the first screen after the account is created.

Sample size reasoning

The metric's numerator is paying customers, which is rare per visitor, so the sample is large and the test long. The per-visitor framing needs more traffic than a per-trial comparison would, because the event is rarer, but it is still the right one: a per-trial comparison would reach significance sooner on the wrong question. The inputs are the current paying customers per visitor, the smallest difference that would change the decision and the confidence required; the sample size and significance guide explains how they set the visitors each arm needs, and the conversion rate calculator helps derive the baseline rate from your own visitor and customer counts.

If traffic cannot support the per-visitor read, do not fall back to per-trial rates. Run longer, test on the highest-traffic entry route only, or accept that the test can detect only a large difference and say so in the plan.

Duration

Enrol to the planned sample across full weekly cycles, then wait for a horizon that covers the trial and the first paid period for the last visitors enrolled, so that refunds and early cancellations have time to appear. Set the end date before launch. Avoid running across a price change or a promotion that changes the offer in one arm's period more than the other's.

Reading the results and the decision rule

Read once, at the planned end. Compare paying customers per visitor, net of refunds, with its confidence interval; then trial starts per visitor and the trial-to-paid rate, to understand the mechanism; then each guardrail. The rule, written before launch:

ResultDecision
Card-required arm has more paying customers per visitor and refunds, chargebacks and complaints stay within limitsRequire the card at sign-up
Card-required arm has more paying customers per visitor but a guardrail breaksKeep the card-free trial; fix renewal clarity before any retest
No reliable difference per visitorKeep the card-free trial, which serves more people at the same yield
Card-free arm has more paying customers per visitorKeep the card-free trial

Carry the result into acquisition planning: a change in paying customers per visitor changes CAC for every channel that sends traffic, and the LTV and CAC guide shows how to check whether the customers gained are worth what they cost to acquire.

Illustrative example — not a real result

A photo editing app tests asking for the card

Lumenpath, an invented online photo-editing service, splits 80,000 sign-up page visitors evenly between its card-free trial and a card-required version with identical renewal wording. In the card-free arm 12% start a trial and 15% of those pay, giving 720 paying customers, or 1.80% per visitor. In the card-required arm 5% start and 40% pay, giving 800 paying customers, or 2.00% per visitor; the 95% interval for the difference is roughly 0.01 to 0.39 points. But refunds and chargebacks on first charges are 7% in the card-required arm against 2% in the card-free arm, above the 4% limit the team agreed. Under the rule's second row, Lumenpath keeps the card-free trial and plans a clearer pre-charge reminder before any retest.

Pitfalls

Reading the trial-to-paid rate is the first pitfall, and the most persuasive, because the card-required arm nearly always wins it. Unclear renewal terms are the second: if the challenger's price or renewal date is less visible than the control's, the test measures confusion, and the guardrails will eventually show it as refunds and complaints. Keep the wording identical and prominent in both arms. Involuntary conversions counted as wins are the third. Count customers net of refunds and chargebacks, and treat a cluster of cancellations immediately after the first charge as a guardrail signal, not as normal churn.

Two operational traps matter too. Payment failures at the card step can look like reluctance; log declines separately from abandonment. And support may handle refund requests differently once they know a test is running; give them one written policy for both arms. The academy capstone on a growth operating system for a vintage clothing resale app practises planning trial and save offer experiments for a seller plan as one part of a year's growth system.

Common mistakes

Card tests are usually lost in the denominator. The free trial length test changes how long the trial lasts rather than how it starts, the pricing page layout test changes how plans are presented, and the experiment library lists every design.

  • Reporting per-trial conversion as the result. The card step changes who starts; count per visitor.
  • Making renewal terms quieter in one arm. Clarity is held constant, never tested.
  • Counting refunded customers as conversions. Net them out at the horizon.
  • Reading before the first paid period ends. Refunds and early churn arrive after the first charge.
  • Testing with a clumsy payment form. Build the step well first, or the test measures the form.
  • Treating regulator guidance as a checklist. It is general direction; take qualified advice.

Frequently asked questions

Why is the trial-to-paid rate the wrong metric here?

Because the card step changes who starts a trial. Asking for payment details filters out casual visitors before the trial begins, so the people left convert at a higher rate almost by definition. Counting paying customers per visitor to the sign-up page puts both arms on the same denominator and shows whether the filter adds customers or only removes trials.

Is a card-required trial allowed?

Trials that turn into paid subscriptions are a consumer protection matter in many markets. Bodies such as the Federal Trade Commission and the Competition and Markets Authority publish guidance on clear terms, consent to charges and easy cancellation, which is useful general direction. It is not legal advice; agree the renewal wording, reminders and cancellation route with someone qualified to advise before testing.

Should customers who forgot to cancel count as wins?

Not without scrutiny. A charge that is refunded, disputed or followed by an immediate cancellation is not a customer the business has won. Count paying customers at the horizon net of refunds and chargebacks, and track early churn, so involuntary conversions cannot carry the result.

Terms used on this page

Tools to try

Frameworks

Where this page is used

Part of these learning paths

Also related

Sources

  1. FTC business guidance — Federal Trade Commission
  2. Competition and Markets Authority — GOV.UK

Related