What this checkpoint covers
Stonepier Seafood Boxes is an invented firm delivering boxes of fresh fish and shellfish by subscription and as one-off orders, with Meta as its largest paid channel. The questions draw on the incrementality testing guide, the remarketing holdout test and the marginal returns and forecasting guide, and they assume the baby clothing holdout exercise.
Task
Write a short answer with its reason for every question before reading the explanations; an answer counts only if the reasoning matches. Reread the linked page for each miss. Finish by writing, for a channel you know, the decision rule you would agree before a test starts: the threshold, the range you would act on and the response to an inconclusive result.
Questions
- A geo test shows prospecting caused roughly half the orders Meta attributed to it over the same weeks. The marketing manager says the test must be broken because Ads Manager disagrees. What do you say, and what do you do with the result?
- Remarketing to basket abandoners has the best reported return in the account. A clean holdout shows a small lift. How can both be true, and what changes?
- During a remarketing holdout, the held-out group still receives the abandoned-basket email flow, and a catalogue campaign another colleague built also reaches them. Which of these threatens the test?
- Partway into a test planned for a full buying cycle, the treated regions are well ahead. The owner wants to stop and scale now. What do you advise?
- In a geo test, the team proposes moving the budget withheld from control regions into the treated regions so that "nothing is wasted". Treated-region frequency starts to climb. What is wrong?
- Prospecting's average return is comfortably above break-even. The last budget step's marginal return was below it. The owner wants another step because "the campaign is profitable". How do you answer?
- A remarketing holdout ends with a lift whose range includes zero and is wide. One person wants to cut remarketing entirely; another wants to scale it because reported return is high. What do you decide?
Answers explained
- The gap is the finding. Attribution credits every conversion an ad touched; the test measures the ones it caused. The ratio of incremental to attributed conversions is the channel's incrementality factor at that spend level. Use it to discount routine reports until the next test, and judge the incremental cost per order against what an order is worth (reading lift).
- Remarketing reaches people who were already likely to buy. Credit follows exposure whether or not the ad made the difference, so reported return is high and the IncrementalityConversions caused by marketing. is modest. Keep RemarketingAdvertising to previous visitors or engagers. only if the cost per incremental conversion is affordable, and try a lower frequency or smaller spend for the same lift (reading the results).
- The catalogue campaign. It is a leak: held-out people are being shown ads, so the lift is understated. Exclude the held-out group from every campaign that can reach the audience. The email flow is the normal state of the business; the test then measures remarketing on top of email, and the write-up says so (pitfalls).
- Keep going. Early results are the noisiest, and stopping when the number looks good means the test can no longer fail. Run the full cycle and the cool-down, then read the range against the threshold set before launch (duration).
- The test has changed its own spend level. Treated regions now receive more money per person than before, so the result describes a higher budget than the one in question, and rising FrequencyAverage times a person saw an ad. shows the extra money buying repeats. Cut the budget in proportion to the withheld regions and keep the frequency guardrail in view (guardrails).
- The next step will resemble the last step, not the average. The average blends the cheap early conversions with the expensive late ones. A step whose marginal return fell below break-even lost money even though the campaign is profitable overall; pull back to the last level that paid and look for new audiences or creative instead (where spend should stop).
- Neither. A range that includes zero has not shown an effect, and a wide range says the test may have been too small to see one. Compare the result with the minimum detectable effect set beforehand: if the test was underpowered, rerun it with a longer window or a pooled audience. Meanwhile, hold spend at a level whose cost per incremental conversion would be tolerable even at the pessimistic end, and do not scale on reported return (sample size reasoning).
Where to go next
If you missed the test-design questions, redo the split and decision-rule steps of the baby clothing exercise. If you missed the scaling questions, rework the budget steps in the marginal returns guide with your own figures. Then continue to the creative systems module with the activewear creative pipeline exercise, or revisit the scaling lessons on the intermediate Meta Ads level.
