Why this matters
Measuring whether AI helped a team was the work of the advanced AI in marketing level. The guide to AI in ad platforms treats handing a decision to the platform as a trade that the platform's own reports cannot judge, since they also count conversions that would have come anyway. This lesson chooses no keyword, audience or event; it writes, orders and reads the tests that decide.
The retailer and its agency's proposals
Tholumb Models is an invented online retailer of model railway kits, track and scenery, advertising on Google Ads and Meta through an agency.
Tholumb Models: the accounts and the agency's three proposals
A part-time bookkeeper matches platform purchases to the shop's orders each week.
| Campaign | Set-up today | A month, last 3 months |
|---|---|---|
| Google Ads, core range search | Exact and phrase; Smart Bidding at a target cost per conversion of £18 | 330 conversions; about half within 3 days of the click, the rest over 3 weeks |
| Google Ads, brand search | Brand keywords only | 410 conversions |
| Meta, prospecting | Interests in model railways and scale modelling; optimises for purchase; customers excluded | 190 purchases reported; 166 matched to shop orders |
| Meta, garden railway range | 2 ad sets sharing the budget; optimises for purchase; stuck in learning limited | 22 purchases, 180 adds to cart |
The agency's proposals, backed by screenshots of the platforms' recommendation pages:
- Move the core range keywords to broad match under the same target.
- Replace the prospecting interests with Advantage+ audience.
- Optimise the garden railway campaign for add to cart, where its cost per result is far lower.
Calendar: a spring sale over the 2 weeks around Easter; a site redesign in early September; the busy season from October.
One change, judged on the measure its test sets
The broad match expansion test reads conversion value at the account's target cost per conversion, since a trial arm that finds more searches will usually spend and convert more. The audience targeting breadth test reads cost per purchase reconciled with the shop's orders, guarded by the share of new customers. The optimisation event test reads cost per purchase in both arms, because the add-to-cart arm wins on its own event by construction. The agency's screenshots compare nothing with anything.
Task
- Write the broad match test plan: arms, what is held constant, the measure, guardrails with limits, the learning period, the reading and the decision rule.
- Write the audience and optimisation event plans in the same shape.
- Put the three tests in a calendar so that each can be read alone, and justify the order.
- Say what the agency may do while the tests run.
Worked answer
Broad match. A campaign experiment arm differs from the original in match typeSettings that decide how closely a search must match a keyword for the ad to enter the auction. alone, inside the same auctions. Both arms keep the Smart BiddingGoogle Ads' automated bid strategies that set each auction's bid from conversion signals. target, since changing it too would answer neither question, and the late conversions decide when the test is read.
Tholumb Models: the broad match test plan
| Element | Choice |
|---|---|
| Arms | Exact and phrase against broad match on the same core keywords, cookie-based split |
| Held constant | The £18 target, budgets, ads, landing pages and negatives; a negative added mid-test goes into both arms and is logged |
| Primary measure | Conversion value at the £18 target |
| Guardrails | Irrelevant search terms no more than 8% of trial-arm spend, against a list agreed before launch; brand conversions by arm, brand excluded in both |
| Learning and reading | 2 weeks set aside, 4 weeks of stable delivery, then 3 more for late conversions |
| Decision rule | The page's rule, with £18 as the target and 8% as the irrelevant-spend limit, written into the plan before launch |
Broad audiences. Two ad sets differ only in targeting, through Meta's A/B test tool. The control is the interest set the account really runs, and the customer exclusion is identical in both, or the broad arm could win on cheap repeat orders.
Tholumb Models: the audience test plan
| Element | Choice |
|---|---|
| Arms | Current interests against Advantage+ audience |
| Held constant | Creative, placements, purchase event, £70 a day per arm, attribution setting and the exclusion; no edits |
| Primary measure | Cost per purchase, purchases matched to shop orders by the bookkeeper |
| Guardrails | New-customer share of matched orders; frequency; average order value |
| Reading | After learning, 4 full weeks, read once |
| Decision rule | The page's rule, read on matched purchases, with the exclusion kept if broad wins |
The upstream event. An event cannot fix a fragmented structure, the event test advises, so the two garden railway ad sets become one first; the test then compares purchase with add to cart as the Optimisation EventThe conversion event an ad set or campaign is told to optimise delivery towards, such as a purchase, lead or install., both arms read on purchases.
Tholumb Models: the optimisation event test plan
| Element | Choice |
|---|---|
| Arms | Purchase against add to cart, through the A/B test tool, both events tested on the live site first |
| Held constant | Audience, creative, placements, £40 a day per arm, bid strategy and attribution |
| Primary measure | Cost per purchase in both arms, matched weekly to shop orders |
| Guardrails | Whether each arm leaves the Learning PhaseThe period after launch or a significant edit when an ad set's delivery is still being optimised.; purchases per add to cart; average order value |
| Reading | After learning, at least 6 full weeks; if purchases cannot separate the arms, the decision is recorded as judgement |
| Decision rule | The page's rule, read on matched purchases; if neither arm leaves learning, the structure is fixed before any retest |
The order. No change lands in a test's window, and none touches the sale, redesign or season.
Tholumb Models: the calendar
| When | Test | Why there |
|---|---|---|
| Mid-January to mid-March | Broad match, Google Ads | Clear of the spring sale; the wait for late conversions ends before Easter |
| Mid-January to late February | Audience breadth, Meta | A different platform with its own random split, so a Google change lands on both arms alike |
| May to July | Optimisation event, Meta | After the audience result is applied, so its targeting is the one the account keeps; done before the redesign |
What the agency may do. It runs the tests as written, applies no recommendation early, adds nothing to one arm only and reports each test on its own measure. The bookkeeper's matched orders settle the Meta tests.
Common mistakes
- Changing the target with the match type. Two changes answer neither question.
- Leaving existing customers in reach of the broad arm. Cheap repeat orders look like a prospecting win.
- Reading the add-to-cart arm on adds to cart. It wins on its own event by construction.
- Running both Meta tests at once. The event test would hold constant a targeting the audience test might change.
