Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

AI in Marketing › Expert

Testing Broad Match and Broad Audiences for a Model Railway Retailer

An agency wants an invented model railway retailer to hand three choices to platform automation on the strength of the platforms' own reports. You turn each proposal into a test that changes one thing, is judged on the measure its test page sets, sets the learning period aside and carries a decision rule written before launch. You finish with three test plans and a calendar that keeps every result readable on its own.
ExerciseExpert levelModule 3: Proving what platform automation adds5 minUpdated

Why this matters

Measuring whether AI helped a team was the work of the advanced AI in marketing level. The guide to AI in ad platforms treats handing a decision to the platform as a trade that the platform's own reports cannot judge, since they also count conversions that would have come anyway. This lesson chooses no keyword, audience or event; it writes, orders and reads the tests that decide.

The retailer and its agency's proposals

Tholumb Models is an invented online retailer of model railway kits, track and scenery, advertising on Google Ads and Meta through an agency.

Illustrative example — not a real result

Tholumb Models: the accounts and the agency's three proposals

A part-time bookkeeper matches platform purchases to the shop's orders each week.

CampaignSet-up todayA month, last 3 months
Google Ads, core range searchExact and phrase; Smart Bidding at a target cost per conversion of £18330 conversions; about half within 3 days of the click, the rest over 3 weeks
Google Ads, brand searchBrand keywords only410 conversions
Meta, prospectingInterests in model railways and scale modelling; optimises for purchase; customers excluded190 purchases reported; 166 matched to shop orders
Meta, garden railway range2 ad sets sharing the budget; optimises for purchase; stuck in learning limited22 purchases, 180 adds to cart

The agency's proposals, backed by screenshots of the platforms' recommendation pages:

  • Move the core range keywords to broad match under the same target.
  • Replace the prospecting interests with Advantage+ audience.
  • Optimise the garden railway campaign for add to cart, where its cost per result is far lower.

Calendar: a spring sale over the 2 weeks around Easter; a site redesign in early September; the busy season from October.

One change, judged on the measure its test sets

The broad match expansion test reads conversion value at the account's target cost per conversion, since a trial arm that finds more searches will usually spend and convert more. The audience targeting breadth test reads cost per purchase reconciled with the shop's orders, guarded by the share of new customers. The optimisation event test reads cost per purchase in both arms, because the add-to-cart arm wins on its own event by construction. The agency's screenshots compare nothing with anything.

Task

  1. Write the broad match test plan: arms, what is held constant, the measure, guardrails with limits, the learning period, the reading and the decision rule.
  2. Write the audience and optimisation event plans in the same shape.
  3. Put the three tests in a calendar so that each can be read alone, and justify the order.
  4. Say what the agency may do while the tests run.

Worked answer

Broad match. A campaign experiment arm differs from the original in match type alone, inside the same auctions. Both arms keep the Smart Bidding target, since changing it too would answer neither question, and the late conversions decide when the test is read.

Illustrative example — not a real result

Tholumb Models: the broad match test plan

ElementChoice
ArmsExact and phrase against broad match on the same core keywords, cookie-based split
Held constantThe £18 target, budgets, ads, landing pages and negatives; a negative added mid-test goes into both arms and is logged
Primary measureConversion value at the £18 target
GuardrailsIrrelevant search terms no more than 8% of trial-arm spend, against a list agreed before launch; brand conversions by arm, brand excluded in both
Learning and reading2 weeks set aside, 4 weeks of stable delivery, then 3 more for late conversions
Decision ruleThe page's rule, with £18 as the target and 8% as the irrelevant-spend limit, written into the plan before launch

Broad audiences. Two ad sets differ only in targeting, through Meta's A/B test tool. The control is the interest set the account really runs, and the customer exclusion is identical in both, or the broad arm could win on cheap repeat orders.

Illustrative example — not a real result

Tholumb Models: the audience test plan

ElementChoice
ArmsCurrent interests against Advantage+ audience
Held constantCreative, placements, purchase event, £70 a day per arm, attribution setting and the exclusion; no edits
Primary measureCost per purchase, purchases matched to shop orders by the bookkeeper
GuardrailsNew-customer share of matched orders; frequency; average order value
ReadingAfter learning, 4 full weeks, read once
Decision ruleThe page's rule, read on matched purchases, with the exclusion kept if broad wins

The upstream event. An event cannot fix a fragmented structure, the event test advises, so the two garden railway ad sets become one first; the test then compares purchase with add to cart as the Optimisation Event, both arms read on purchases.

Illustrative example — not a real result

Tholumb Models: the optimisation event test plan

ElementChoice
ArmsPurchase against add to cart, through the A/B test tool, both events tested on the live site first
Held constantAudience, creative, placements, £40 a day per arm, bid strategy and attribution
Primary measureCost per purchase in both arms, matched weekly to shop orders
GuardrailsWhether each arm leaves the Learning Phase; purchases per add to cart; average order value
ReadingAfter learning, at least 6 full weeks; if purchases cannot separate the arms, the decision is recorded as judgement
Decision ruleThe page's rule, read on matched purchases; if neither arm leaves learning, the structure is fixed before any retest

The order. No change lands in a test's window, and none touches the sale, redesign or season.

Illustrative example — not a real result

Tholumb Models: the calendar

WhenTestWhy there
Mid-January to mid-MarchBroad match, Google AdsClear of the spring sale; the wait for late conversions ends before Easter
Mid-January to late FebruaryAudience breadth, MetaA different platform with its own random split, so a Google change lands on both arms alike
May to JulyOptimisation event, MetaAfter the audience result is applied, so its targeting is the one the account keeps; done before the redesign

What the agency may do. It runs the tests as written, applies no recommendation early, adds nothing to one arm only and reports each test on its own measure. The bookkeeper's matched orders settle the Meta tests.

Common mistakes

  • Changing the target with the match type. Two changes answer neither question.
  • Leaving existing customers in reach of the broad arm. Cheap repeat orders look like a prospecting win.
  • Reading the add-to-cart arm on adds to cart. It wins on its own event by construction.
  • Running both Meta tests at once. The event test would hold constant a targeting the audience test might change.

Pages this lesson applies

Terms used on this page

Also related

Related