Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Growth Marketing › Expert

Checkpoint: Trial and Save Offer Experiments

Scenario questions set at an invented photo editing app test the second expert growth module: reading a trial length test on one clock, judging a card at sign-up on the right denominator, guarding against refunds and early churn, valuing a save offer after its discount and keeping the exit easy. Each answer gives the reasoning and the experiment design behind it, so a miss leads straight back to the reading.
CheckpointExpert levelModule 2: Trials and cancellations as experiments4 minUpdated

What this checkpoint covers

Ezquarvel Photo is an invented photo editing app sold as a monthly subscription after a free trial. It ran three tests from the library's designs: the free trial length test, the card-required trial test and the cancellation save offer test. The questions assume the testing judgement of the advanced growth level.

Illustrative example — not a real result

Ezquarvel Photo: the figures used in the questions

Trial test, 6,400 sign-ups per arm14-day trial (current)7-day trial
Paying at 30 days from sign-up576 (9.0%)666 (10.4%)
Still paying a month later484480

Interval for the difference: about +0.4 to +2.4 points. The plan allowed early churn to rise by 2 points at most.

Card test, 30,000 visitors per armCard-freeCard at sign-up
Trials started4,2001,800
Paying at the horizon, net of refunds504594
Refunds and chargebacks (limit 4%)2.0%3.5%
Complaints about unexpected charges (limit 10)427
Still paying a month later413410
Save offer test, 2,400 starters per arm, at 150 daysDirect cancellationHalf price for three months
Revenue per starter before the discount£2.40£5.10
Discount per starter£0.00£2.30
Net retained revenue per starter£2.40£2.80

Interval for the net difference: about −£0.50 to +£1.30. 38% of offer-arm starters accepted; 71% of those cancelled within a month of the offer ending. About 800 subscribers start cancelling each month.

Task

  • Write a decision and a reason for every question before reading on; an answer counts only if the reason matches.
  • Reread the design linked from each miss.

Questions

  1. Halfway through enrolment, the product manager compares the shorter trial's trial-to-paid rate, complete for its early cohorts, with the longer trial's, and wants to stop: the short trial is ahead.
  2. Marketing calls the card test a landslide on trial-to-paid rates. What is the right comparison?
  3. Refunds in the card arm stayed under the agreed limit. Does the card at sign-up go live?
  4. The retention lead reports the save offer's acceptance rate as its save rate and calls the offer a success. What does the test show?
  5. For a retest, the product lead proposes a large "Keep my discount" button, a small cancel link and a screen listing the edits a subscriber will lose. Approve it?
  6. Finance wants a pause of up to two months tested against direct cancellation, on the same number of starters per arm as the discount test, and read within a month of launch. What do you tell them?
  7. At the final read, the growth lead wants the shorter trial: it won the primary metric with an interval above zero.

Answers explained

  1. Keep going, and read both arms on one clock. The shorter arm resolves first, so it always looks good first. The Trial-to-Paid Conversion Rate at each trial's end is not the metric; paid conversions per sign-up at one fixed horizon, longer than the longer trial, is. See why both arms share a fixed horizon.
  2. Paying customers per visitor. The card step filters who starts, so the card arm wins per trial almost by definition; per visitor the gap is far smaller. See the per-visitor primary metric.
  3. No. A Guardrail Metric breaks although refunds held: complaints rose sharply and a month later the arms are level, so the extra customers did not mean to pay. Keep the card-free trial and fix renewal clarity before any retest. See refunds, early churn and complaints as guardrails.
  4. No reliable gain. Acceptance is not a save. A Save Offer is judged on net retained revenue per starter at a horizon past the offer's end, after its cost. The interval spans zero and most accepters left when the discount ended: remove it and let people cancel directly. See the decision rule for save offers.
  5. No. Each change inflates the save rate by obstruction. Declining stays a single action as prominent as accepting, and steps, time to cancel and complaints are hard guardrails. Regulators treat easy cancellation as a consumer protection matter; take qualified advice on the flow. See keeping the exit easy.
  6. It cannot be read in a month. Only starters enter, revenue per starter is skewed, and the horizon runs a full billing period past the longest pause. Write the end date down before launch, and never read early. See sample size and duration for save offers.
  7. Keep the current trial. The rule written before launch covers this row: more paid conversions but early churn beyond its limit, so the gain is borrowed from retention and lost in LTV. No segment hunt rescues it. See the trial length decision rule.
Illustrative example — not a real result

The arithmetic behind answers two, six and seven

  • Two: per trial, 12% against 33%; per visitor, 1.68% against 1.98%.
  • Six: 4,800 starters at about 800 a month take six months to enrol; a pause of up to two months plus a month's billing puts the read about nine months after launch.
  • Seven: early churn of 92 in 576 (16%) against 186 in 666 (28%), a rise of 12 points against a limit of 2.

Where to go next

Misses on questions one and seven send you back to the podcast hosting trial exercise; when every answer holds, move on to the dog walking marketplace exercise.

Pages this lesson applies

Terms used on this page

Also related

Related