What this checkpoint covers
Ezquarvel Photo is an invented photo editing app sold as a monthly subscription after a free trial. It ran three tests from the library's designs: the free trial length test, the card-required trial test and the cancellation save offer test. The questions assume the testing judgement of the advanced growth level.
Ezquarvel Photo: the figures used in the questions
| Trial test, 6,400 sign-ups per arm | 14-day trial (current) | 7-day trial |
|---|---|---|
| Paying at 30 days from sign-up | 576 (9.0%) | 666 (10.4%) |
| Still paying a month later | 484 | 480 |
Interval for the difference: about +0.4 to +2.4 points. The plan allowed early churn to rise by 2 points at most.
| Card test, 30,000 visitors per arm | Card-free | Card at sign-up |
|---|---|---|
| Trials started | 4,200 | 1,800 |
| Paying at the horizon, net of refunds | 504 | 594 |
| Refunds and chargebacks (limit 4%) | 2.0% | 3.5% |
| Complaints about unexpected charges (limit 10) | 4 | 27 |
| Still paying a month later | 413 | 410 |
| Save offer test, 2,400 starters per arm, at 150 days | Direct cancellation | Half price for three months |
|---|---|---|
| Revenue per starter before the discount | £2.40 | £5.10 |
| Discount per starter | £0.00 | £2.30 |
| Net retained revenue per starter | £2.40 | £2.80 |
Interval for the net difference: about −£0.50 to +£1.30. 38% of offer-arm starters accepted; 71% of those cancelled within a month of the offer ending. About 800 subscribers start cancelling each month.
Task
- Write a decision and a reason for every question before reading on; an answer counts only if the reason matches.
- Reread the design linked from each miss.
Questions
- Halfway through enrolment, the product manager compares the shorter trial's trial-to-paid rate, complete for its early cohorts, with the longer trial's, and wants to stop: the short trial is ahead.
- Marketing calls the card test a landslide on trial-to-paid rates. What is the right comparison?
- Refunds in the card arm stayed under the agreed limit. Does the card at sign-up go live?
- The retention lead reports the save offer's acceptance rate as its save rate and calls the offer a success. What does the test show?
- For a retest, the product lead proposes a large "Keep my discount" button, a small cancel link and a screen listing the edits a subscriber will lose. Approve it?
- Finance wants a pause of up to two months tested against direct cancellation, on the same number of starters per arm as the discount test, and read within a month of launch. What do you tell them?
- At the final read, the growth lead wants the shorter trial: it won the primary metric with an interval above zero.
Answers explained
- Keep going, and read both arms on one clock. The shorter arm resolves first, so it always looks good first. The Trial-to-Paid Conversion RateShare of free trials that became paying subscriptions. at each trial's end is not the metric; paid conversions per sign-up at one fixed horizon, longer than the longer trial, is. See why both arms share a fixed horizon.
- Paying customers per visitor. The card step filters who starts, so the card arm wins per trial almost by definition; per visitor the gap is far smaller. See the per-visitor primary metric.
- No. A Guardrail MetricA metric an experiment must not worsen beyond an agreed bound, watched beside the primary metric that decides the test. breaks although refunds held: complaints rose sharply and a month later the arms are level, so the extra customers did not mean to pay. Keep the card-free trial and fix renewal clarity before any retest. See refunds, early churn and complaints as guardrails.
- No reliable gain. Acceptance is not a save. A Save OfferAn alternative such as a pause, smaller plan or discount shown once to a cancelling customer, beside the cancel route and declinable in one action. is judged on net retained revenue per starter at a horizon past the offer's end, after its cost. The interval spans zero and most accepters left when the discount ended: remove it and let people cancel directly. See the decision rule for save offers.
- No. Each change inflates the save rate by obstruction. Declining stays a single action as prominent as accepting, and steps, time to cancel and complaints are hard guardrails. Regulators treat easy cancellation as a consumer protection matter; take qualified advice on the flow. See keeping the exit easy.
- It cannot be read in a month. Only starters enter, revenue per starter is skewed, and the horizon runs a full billing period past the longest pause. Write the end date down before launch, and never read early. See sample size and duration for save offers.
- Keep the current trial. The rule written before launch covers this row: more paid conversions but early churn beyond its limit, so the gain is borrowed from retention and lost in LTVTotal value a customer brings over time.. No segment hunt rescues it. See the trial length decision rule.
The arithmetic behind answers two, six and seven
- Two: per trial, 12% against 33%; per visitor, 1.68% against 1.98%.
- Six: 4,800 starters at about 800 a month take six months to enrol; a pause of up to two months plus a month's billing puts the read about nine months after launch.
- Seven: early churn of 92 in 576 (16%) against 186 in 666 (28%), a rise of 12 points against a limit of 2.
Where to go next
Misses on questions one and seven send you back to the podcast hosting trial exercise; when every answer holds, move on to the dog walking marketplace exercise.
