What this checkpoint covers
Saffronhill Recipes is an invented app offering a weekly recipe plan by subscription after a free trial. It grows through word of mouth, app store search and paid social install campaigns. The questions draw on the incrementality testing guide, the budget increase incrementality test and the marginal returns and forecasting guide, and they assume the budgeting app exercise.
Task
Write an answer and its reason for each question before reading the explanations; it counts only if the reasoning matches. Then write, for a channel you know, the threshold a marginal customer must clear and where it comes from.
Questions
- A well-known cook recommends Saffronhill on television. Trial starts jump for a fortnight, and paid social reports its cheapest cost per trial ever. The team wants to raise the paid budget while it is "working so well". What do you say?
- Saffronhill wants to test a budget increase. One proposal raises spend in matched regions; another alternates higher and normal spend week by week nationally. Which do you choose, and why?
- The analyst reading the budget test counts only the subscriptions credited to paid social in each region. What is wrong?
- Blended acquisition cost, all spend over all new subscribers, is comfortably low. Paid social's credited cost per subscriber is acceptable. The last budget step's marginal cost per incremental subscriber is high. Which figure decides the next step?
- Finance compares the marginal cost per incremental subscriber with a three-year revenue projection per average subscriber and approves the increase. What should the comparison be?
- Across three budget steps, each step adds fewer incremental subscribers than the last, frequency climbs and the share of new users among those reached falls. What is happening, and what follows?
- The chief executive asks for one number: subscribers next year if the paid budget doubles. How do you answer?
Answers explained
- Wait, then test. The broadcast created demand that flows through every channel, including paid ads, which claim the people who were already coming. Credited cost falls with no change in what the ads cause. Raising spend into the spike buys mostly credit. Let the spike pass and use a matched-region test to see what paid adds on normal weeks (why attribution cannot answer the budget question).
- Matched regions. Region tests run test and control in the same weeks, so season and news affect both. Weekly alternation lets trials started in a high week convert in a normal one, blurring the two, and every national event lands on one arm. Use time-based designs only when regions are impossible, with long blocks and a comparison series (design).
- The test exists to count everything. Extra paid spend can take subscribers from app store search, word of mouth or email, and only all-channel subscriptions from Saffronhill's own records show whether the total rose. Branded search and blended acquisition cost are the guardrails that reveal cannibalisation (metrics and guardrails).
- The marginal figure. Blended acquisition cost (CACCost to acquire one new customer.) includes organic subscribers who cost nothing; credited cost includes subscribers who would have come anyway. Only the marginal cost per incremental subscriber says what the next step buys (marginal cost per acquisition).
- Contribution within the payback window, for a marginal subscriber. Revenue overstates what the business keeps, three years outruns any cohort Saffronhill has observed, and marginal subscribers often lapse faster than the average. Compare with contribution inside the agreed window from the test cohort's own retention; lifetime value (LTVTotal value a customer brings over time.) projections belong in labelled appendices (where spend should stop).
- Saturation. The reachable, responsive audience is being used up: each step buys repeat exposure to the same people. Stop at the last step whose marginal cost cleared the threshold, and look for new audiences, creative or channels rather than more budget (diminishing returns and saturation).
- With scenarios and ranges. Give outcomes for holding, a modest increase and doubling, each built from marginal costs that rise with spend, with the assumptions written beside them. Doubling sits beyond anything tested, so its range is wide; recommend a staged increase with a test at each stage, measuring IncrementalityConversions caused by marketing. as it goes (forecasting from spend scenarios).
Where to go next
If you missed the design or counting questions, reread the budget increase test and redo the reading step of the budgeting app exercise. If you missed the marginal questions, rework the budget steps in the marginal returns guide with your own figures. Then continue to pricing with the scheduling tool pricing exercise, or revisit payback on the intermediate growth level.
