What this checkpoint covers
Lindenfield Opticians is an invented chain of opticians taking eye-test bookings online. Its paid search (SEMPaid search advertising (sometimes used to include SEO).) account is large enough to test by region. The questions draw on the bidding strategy geo split test, the budget increase incrementality test and the guide to results dropping after a bid strategy change.
Task
Write an answer and its reason for each question, then compare. The right action with the wrong reason counts as a miss; reread the linked page for every miss.
Questions
- To test automated bidding, a colleague proposes putting the regions with the most branches in the test group "so the test has enough data", and the rest in control. What is wrong, and what should happen instead?
- Lindenfield switched every region to a new bid strategy last month. Bookings are down on the month before, and the area manager wants to switch back. Why is the comparison weak, and what would a fair one need?
- Before a budget test starts, the finance lead asks when it will be stopped. What stopping rules and guardrails should be written down, and which reason for stopping early is not acceptable?
- Use the figures in the example below. Should Lindenfield keep the higher budget?
- A bid strategy test has a learning fortnight followed by a four-week read. Eye-test bookings usually follow the click within a few days. When should the read begin and end, and when should the results be pulled?
- A budget test shows extra bookings in test regions, but the pre-period showed that the test and control regions' weekly ratio wandered by about as much as the lift. What do you report?
- The platform reports that raised budgets produced many more conversions in test regions, while bookings from all channels in the booking system barely moved. What explains the gap, and which figure decides?
Lindenfield Opticians: budget test for question four
| Row label | Test regions | Control-based prediction |
|---|---|---|
| Paid search spend over the read | £28,000 | £20,000 |
| Eye-test bookings from all channels | 560 | 460 |
Before the test, the business set its affordable cost per incremental booking at £60, from the margin on an eye test and the glasses that usually follow. Average cost per booking across the account is £45.
Answers explained
- The groups must be matched and then randomised. Choosing test regions by size stacks the test with regions that may already behave differently. Match regions into pairs or groups on their booking history, check that they tracked each other in the pre-period, then assign at random within each group. See the design section of the geo split test.
- Before-and-after mixes the switch with everything else. Season, learning and conversion delay differ between the months. A fair comparison needs a control that did not change, such as matched regions or a campaign experiment, and a baseline with closed conversion windows. See confirming the fall against the right baseline.
- Write the threshold and the guardrails before launch. The decision rule is marginal cost per incremental booking against an affordability threshold; blended CPACost of each conversion. across channels and branded search volume are guardrails. Stop early only for real harm, such as a broken booking flow. Stopping because the number looks good, or bad, in the first weeks is not acceptable. See the budget increase test.
- No, not at this level. The average looks healthy, but the decision depends on what the extra money bought. The arithmetic is below. See reading the results in the budget increase test.
- Start after learning ends, and pull late. The read begins when the test regions are out of learning, runs whole weeks, and is pulled only once the last week's delayed bookings have arrived, or is read by conversion time. See conversion delay misread as decline.
- No measurable effect at this size. A lift no larger than the normal wander has not shown an effect. Report it as inconclusive, and if the decision matters, rerun with a larger lift, more regions or a longer read, sized from the pre-period. See sample size reasoning in the geo split test.
- Credit is not causation. The platform counts conversions it touched, including bookings that would have come through other channels anyway. The IncrementalityConversions caused by marketing. read uses all-channel bookings from Lindenfield's own system, and that figure decides. See the metrics section of the budget increase test.
Answer four: marginal cost per incremental booking
Extra spend: £28,000 − £20,000 = £8,000. Incremental bookings: 560 − 460 = 100. Marginal cost: £8,000 ÷ 100 = £80 per incremental booking, above the £60 threshold. Average cost in test regions: £28,000 ÷ 560 = £50, comfortably under the threshold, which is why the average hides the margin. Hold the budget at its previous level, or test a smaller step.
Where to go next
If you missed the matching or reading questions, redo the geo split bidding exercise for a machinery dealer. If question four caught you out, revisit marginal returns before the third module, which starts with the Performance Max brand demand exercise. For the bidding foundations, return to the intermediate Google Ads level.
