What this checkpoint covers
The second module of the intermediate growth level: building and reading the table in the cohort analysis tutorial, the customer and revenue definitions in the guide to churn and retention metrics, and the holdouts, cohort ages and leading metrics of the experimentation loop for retention. It follows the project management tool cohort exercise.
Task
- Answer every question in a sentence or two, with your reason, before reading the explanations.
- Compare your reasoning with each explanation and note what you missed.
- Take the retention KPIA metric tied directly to an objective. on your own dashboard. If it is a blended rate, rewrite it as one cell of a cohort table, naming the cohort event, the measure and the age, and note what the change would have shown last quarter.
Questions
- A product with a free plan builds cohorts from sign-up date and finds its retention curve falls almost to nothing. The founder concludes paying customers do not stay. What is wrong with the table?
- Retention at the same age has risen steadily down the column for recent cohorts. The product lead credits a new feature. What do you check first?
- Every cohort, young and old, dips in the same calendar month. What does that pattern point to?
- A software company reports a falling customer churn rate. Its denominator is customers at the end of each month. Why is the figure suspect, and what else should sit beside it?
- A garden supplies retailer's "never lapsed" retention curve collapses within months, yet its customers return each spring. How should the table be rebuilt?
- A new welcome sequence is tested as a short split, ended when the planned sample is reached in its first week, and reported as a retention win. What would you change?
- A team wants "profile completed" as the leading metric for a retention test because it moves quickly. What must be shown before it is used?
- A win-back email is tested while the welcome series and push notifications are also being changed. Why will the result be hard to trust?
Answers explained
- The cohort event measures activation, not retention. Sign-up cohorts mix people who never became customers with those who did. The cohort analysis tutorial cohorts on the first paid event, and keeps a separate sign-up view for activation.
- The acquisition mix. A better figure at the same age may simply mean a channel or segment that retains badly now makes up less of each cohort. Split the column by channel and segment at acquisition before crediting the product; the tutorial lists this among its common mistakes.
- Something that happened to everyone in that month. A diagonal shift is a calendar effect, such as a price change, an outage or a competitor launch, rather than a change in cohort quality. Read the diagonal before blaming newer customers.
- Customers acquired during the month are in the base. That dilutes churn. The guide to churn and retention metrics uses customers at the start of the period. Revenue churn and net revenue retention belong beside it, with voluntary and involuntary churn reported separately.
- Measure "active in window" on a grain that matches the cycle. A seasonal business produces a smile: customers fall away and come back. A "never lapsed" definition marks every seasonal returner as lost and makes a healthy business look leaky.
- Keep a persistent holdout and read at fixed cohort ages. A short split measures the first-week bump. The experimentation loop for retention compares test and holdout at an early and a later age; a gap that shrinks was novelty. Call that comparison a holdout, and keep the term A/B TestControlled comparison of two variants. for short tests inside the programme.
- That it predicts staying, before and during the test. Check on past cohorts that customers who hit it retain better, then check after the test that it and later retention moved together in the test group. A step people are nagged into can rise without changing why they stay.
- Other messages move at the same time. The email's result is measured against whatever else changed that week and may flip when the rest of the programme settles. Freeze the other lifecycle messages for both groups, or test the programme as a unit.
Where to go next
If you missed the reading questions, return to the project management tool cohort exercise and read its table again, column by column, with the mix beside it. Retention errors carry straight into LTVTotal value a customer brings over time., so if you missed the definition questions, rebuild one lifetime value figure from your own cohorts. Then start the third module, on loops and experimentation at scale, with the invoicing app referral loop exercise.
