Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Growth Marketing › Expert

Designing Trial Experiments for a Podcast Hosting Service

Two founders of an invented podcast hosting service each want a different change to the free trial: a shorter trial, or payment details at sign-up. You turn both into test plans with a hypothesis, a primary metric read at a fixed horizon, guardrails with limits, a unit of randomisation, sample and calendar reasoning, pitfalls closed before launch and a decision rule, then decide which test runs first and why the two must not share sign-ups.
ExerciseExpert levelModule 2: Trials and cancellations as experiments8 minUpdated

Why this matters

The advanced growth level audited experiment rigour across a backlog: power, peeking, interacting tests. Trial experiments add a clock. Every sign-up takes weeks to resolve into a paying customer or not, two trial lengths finish at different moments, and a card step changes who enters the trial at all. The free trial length test and the card-required trial test are the designs. Leading them means writing the rules before either test can flatter the founder who proposed it.

The hosting service and its two proposals

Pashkevro Audio, an invented service, hosts shows for independent podcasters: it stores each episode, sends the feed to listening apps and reports listener statistics; after a card-free trial the podcaster pays monthly. The founders disagree about the trial and have asked the growth lead for tests both will accept.

Illustrative example — not a real result

Pashkevro Audio: the trial, the funnel and the two proposals

Last six monthsMonthly average
Visitors to the sign-up page24,000
Trials started, all card-free, 30 days2,400
Paying by 45 days from sign-up288 (12% of sign-ups, 1.2% of visitors)
Paying at the end of the 30-day trial252 (10.5% of sign-ups)
New payers who cancel after their first paid month14%
Refunds and chargebacks on first charges1.2%
Trials extended by support on request5%
  • Of sign-ups who went on to pay, 81% published a first episode within 7 days and 93% within 14; 62% paid in the last 3 days of the trial or after it ended.
  • Onboarding emails go out on days 1, 3, 7, 21, 27 and 29, the last two being trial-ending reminders. A plan code sets the trial length, which is printed on the home and pricing pages.
  • A trial account costs about £0.80 a month in storage and bandwidth; a paying customer contributes about £95 in their first year.
  • The product founder wants a 14-day trial. The commercial founder wants a card at sign-up and expects a jump in trial-to-paid.
  • The team's calculator: 4,400 sign-ups per arm to detect a 2-point change from 12% paid per sign-up; about 33,000 visitors per arm to detect a 0.25-point change from 1.2% paying per visitor.

Paying customers at a fixed horizon

Both designs rest on one discipline. Each test has a single denominator chosen before launch, sign-ups or visitors, and one clock for both arms, long enough for late payers, followed by the first paid period for early churn and refunds. The Trial-to-Paid Conversion Rate is context only: the length test would read it at different moments, and a card step filters who reaches a trial at all. Each assignment is made once and stored on the account and in billing, so refunds, support contacts and churn join to the arm.

Task

  1. Write each founder's proposal as a hypothesis with the decision it serves.
  2. Choose each test's primary metric and the horizon at which it is read.
  3. Set the guardrails and their limits, including refunds, chargebacks and early churn.
  4. Choose the unit of randomisation for each test and what stays identical between arms.
  5. Reason through the sample each test needs and the calendar it implies.
  6. Close the pitfalls specific to Pashkevro before launch.
  7. Write the decision rules, choose which test runs first and say why the two must not share sign-ups.

Worked answer

Hypotheses. A shorter trial will hold or raise paid conversions per sign-up at a fixed horizon, because the deadline arrives after most future payers have published, provided the first episode comes within it. The materials support the proviso: almost every payer publishes early and then waits for the deadline. A card at sign-up will cut trial starts but raise the share who pay; paying customers per visitor shows which effect is larger. Both are written up in the experiment hypothesis template and signed before the first account is assigned.

Metrics, guardrails and units. Reading each arm when its trial ends misses payers who arrive after the ending email and flatters the shorter arm, which resolves first. The card founder's jump will appear almost by definition, so the card test counts per visitor, net of refunds. Each Guardrail Metric gets its limit before launch; the card guide reports regulators treating subscription terms and the ease of cancelling as consumer protection questions, so the limits on refunds, chargebacks and complaints reflect that and the renewal wording gets qualified advice. The length test's Unit of Randomisation is the account at sign-up, but the pages before sign-up print the length, so Pashkevro removes it from them for both arms during the test and states it on the account-creation screen. The card test assigns visitors before the step that changes who signs up.

Illustrative example — not a real result

Pashkevro Audio: the two plans and their calendar

ElementTrial length testCard-required test
Arms30 days (control) against 14 daysCard-free (control) against card at sign-up, identical renewal wording and reminder
UnitAccount, once at sign-upVisitor, first view of the sign-up page, kept across sessions
Primary metricPaid conversions per sign-up at 45 daysPaying customers per visitor, net of refunds and chargebacks, at 75 days (60 with a 14-day trial)
Guardrail limitsActivation within the trial: fall of 3 points at most; early churn: rise of 1.5 points at most; support contacts per trial: rise of 0.05 at mostRefunds and chargebacks: 3% of first charges at most; early churn: 2 points above card-free at most; complaints about charges: 15 at most
Sample4,400 sign-ups per arm33,000 visitors per arm
Calendar16 weeks of enrolment, read 75 days after the last sign-up: about 6 months12 weeks of enrolment, then the horizon: about 5 months
  • Break-even: 16 fewer days at £0.80 a month saves about £0.43 per non-paying sign-up, £0.38 per sign-up overall, the value of 0.4 points of paid conversion at £95. Detecting 0.4 points would need about 100,000 sign-ups per arm.
  • In sequence: about 11 months. Side by side on half the traffic each: the length test reads after about 10 months, the card test after about 8.

Sample, duration and pitfalls. How small a change matters is decided by money, using the break-even calculator: the hosting saved is worth a fall too small for Pashkevro ever to detect, so the length test is sized for a larger change and its rule says what a smaller one means. Both tests enrol in whole weeks, avoid price changes and seasonal peaks, and fix their end dates at the start. The shorter arm would lose every email after its end, both trial-ending reminders among them, so emails are rescaled or triggered by behaviour. Plan changes and reactivations must keep each account in its arm, support follows one extension and refund policy, the payment step is built properly first, and card declines are logged apart from abandonment.

Decision rules. Written into each plan and read once:

TestResultDecision
LengthShorter trial converts more per sign-up, guardrails holdAdopt the shorter trial
LengthConverts more, but early churn or activation breaks its limitKeep the longer trial; the extra payers would leave early
LengthNo reliable difference, or converts fewerKeep the current trial, unless with no difference cost and churn both clearly favour the shorter one
CardMore paying customers per visitor, every guardrail within its limitRequire the card
CardAny other resultKeep card-free; if a guardrail broke, fix renewal clarity before any retest

Which runs first, and why apart. Shared sign-ups would cross two changes that interact, the case the section on interacting tests warns about: tests on the same or consecutive steps expose each other's arms unevenly, so neither result is clean. Here the card step changes who enters the length test, the card arm's terms state the length, and accounts would carry assignments made at different units. Crossing the arms in one combined test would need more traffic still. Splitting the traffic saves about a month and runs the card test on a length that may be about to change. The length test goes first: it takes no payment details, the payment step and wording need building and advice anyway, and its answer fixes the length the card test states and shortens its horizon. If the card later wins, the length is rechecked with a card-required trial rather than assumed to carry over.

The cancellation save offer test can run alongside either: it changes a different step, the exit, and assigns subscribers independently when they start to cancel, which the same section treats as usually negligible. Its arm is recorded on the account so early churn in the trial tests can be read with it in view. Each test closes with a report from the experiment results report template, the rule quoted, before the next starts.

Common mistakes

  • Reading each arm when its trial ends. Read both arms on one clock that starts at sign-up.
  • Judging the card on trial-to-paid. Fewer, keener people start, so divide by visitors, net of refunds.
  • Leaving day-numbered emails alone. The shorter arm silently loses its reminders.
  • Assigning after a page has promised a length. Remove the promise or randomise the visitor.
  • Sizing for the change the founders hope for. Size for the change worth acting on, and say what a smaller one means.
  • Running both tests on the same sign-ups. Each test's arms meet the other's unevenly and neither result is clean.

The module checkpoint moves from design to reading results at a photo editing app, and brings in the save offer.

Pages this lesson applies

Terms used on this page

Also related

Related