Why this matters
A growth team that scales on credited sign-ups scales into its own organic demand. The question a budget decision asks is narrower: what did the last slice of spend cause, and is a user bought at that price worth it? The budget increase incrementality test is the design that answers it, the incrementality testing guide explains how lift is read, and the marginal returns and forecasting guide explains why the last slice matters more than the average. At the intermediate level you built acquisition cost and payback from cohorts; here those figures become the threshold the test is judged against.
The app and its budget test
Coinwick Money is an invented personal budgeting app with a free tier and a paid subscription. It acquires users through app-install campaigns. To learn whether more spend would buy more paying users, it matched eighteen regions into groups of three on pre-period sign-ups and, within each group, randomly left one region alone, raised spend by about a third in another and by about three-fifths in the third.
Coinwick Money: the budget test, six weeks after an eight-week pre-period
| Cell | Pre-period sign-ups | Spend in test weeks (planned) | Spend in test weeks (actual) | Sign-ups in test weeks | Paying users within 30 days |
|---|---|---|---|---|---|
| Control | 5,000 | £15,000 | £15,000 | 5,400 | 1,080 |
| Cell A, about +33% | 4,500 | £13,500 | £18,000 | 5,240 | 1,022 |
| Cell B, about +60% | 4,500 | £13,500 | £21,600 | 5,400 | 1,042 |
- Sign-ups the platforms credited in each test cell: about 1,700 at planned spend; during the test, 2,500 in A and 2,950 in B.
- Contribution per paying user within the twelve-month payback window: £110 on average.
- Paying users in the test cells kept their subscription into month 2 at 71%, against 78% for paying users generally.
- Current national install budget: £90,000 a quarter.
From attributed to incremental to marginal
Three readings, in order. Incremental: the sign-ups each test cell gained over what the control cell predicts from the pre-period relationship. Paying: the same for paying users, because a free sign-up is not a customer and counting it makes acquisition cost (CACCost to acquire one new customer.) and value look low in the same stroke, as the LTV and CAC guide warns. Marginal: the extra spend of each step divided by the extra paying users that step bought, compared with what a marginal paying user is worth. Users bought at the margin are often less valuable than the average, so check before assuming they are not.
Task
- Predict each test cell's sign-ups and paying users without the increase, and calculate the incremental figures.
- Compare incremental sign-ups with the extra sign-ups the platforms credited.
- Calculate the marginal cost per incremental sign-up and per incremental paying user for each step.
- Estimate what a marginal paying user is worth and compare.
- Recommend next quarter's budget and state the caveats.
- Write the next test.
Worked answer
Coinwick's test, read
Pre-period ratio: 4,500 ÷ 5,000 = 0.9 for each test cell. Predicted sign-ups: 0.9 × 5,400 = 4,860. Predicted paying users: 0.9 × 1,080 = 972.
| Row label | Cell A | Cell B | Step from A to B |
|---|---|---|---|
| Extra spend | £4,500 | £8,100 | £3,600 |
| Incremental sign-ups | 380 | 540 | 160 |
| Extra sign-ups the platforms credited | 800 | 1,250 | 450 |
| Incremental paying users | 50 | 70 | 20 |
| Paying share of incremental sign-ups | ≈ 13% | ≈ 13% | 12.5% |
| Marginal cost per incremental sign-up | ≈ £11.84 | £15.00 | £22.50 |
| Marginal cost per incremental paying user | £90 | ≈ £116 | £180 |
Platform-credited cost per sign-up during the test: A £18,000 ÷ 2,500 = £7.20; B £21,600 ÷ 2,950 ≈ £7.32. Paying share of all control sign-ups: 1,080 ÷ 5,400 = 20%. Value of a marginal paying user, adjusted for weaker early retention: £110 × 71 ÷ 78 ≈ £100.
Attributed against incremental. The platforms credited roughly twice the sign-ups the extra spend caused, and their cost per sign-up barely moved between the two steps. The dashboard says the second step was as good as the first; the test says it was far worse.
Quality at the margin. Incremental sign-ups paid at about two-thirds the usual rate and lapsed faster in their first month. Marginal users are cheaper to count and dearer to keep, which is why the comparison runs on paying users and on a value adjusted for their retention, not on the average lifetime value (LTVTotal value a customer brings over time.).
Recommendation. The first step bought paying users below what they are worth; the second step's users cost nearly twice their value. Raise the national budget by about a third next quarter, roll it out region by region rather than at once, and keep a few regions at the old level as a standing control. Report the expected gain as a range, because the result rests on six weeks and a modest number of incremental payers, and the true IncrementalityConversions caused by marketing. of the step could sit well either side of the midpoint.
Caveats. The thirty-day paying window misses slower converters, which would lower the marginal cost; the weaker retention would raise it. Re-read both cells at sixty days before the rollout passes the halfway point.
Next test. Hold the new level and test a further step of about half the size of the failed one, so the crossing point between the two levels is found. Run a second cell that changes creative towards people likely to pay, because raising the paying share of marginal sign-ups would move the crossing point further than any budget change.
Common mistakes
- Judging the test on credited sign-ups. The dashboard's cost per sign-up hides the difference between steps.
- Comparing the marginal cost with the average value. Marginal users retain worse; adjust before comparing.
- Counting free sign-ups as customers. The threshold is set on paying users.
- Scaling to the largest cell because its total gain was biggest. The question is what the last step bought.
- Removing every control region at rollout. Without one, the next drift goes unseen.
