Hypothesis
If an ad set uses broad or Advantage+ audience targeting instead of detailed interests, with the same creative, budget and optimisation event, it will deliver purchases at a lower cost, because the delivery system finds buyers outside the interests the team would have chosen. The decision behind it is concrete: hand audience selection to the platform, or keep detailed interest targeting.
The hypothesis has a known failure mode, and the design is built around it. A broad audience includes the business's existing customers and recent site visitors, and the delivery system tends to find people likely to buy, who are often those same people. Broad targeting can then show a lower cost per purchase while buying mostly repeat orders, which looks like a win and is not. The Advantage+ audiences guide explains how the option expands beyond its suggestions and when manual targeting still wins; this page turns that question into a test plan for one account and reads it on new customers as well as cost.
Metrics and guardrails
The primary metric is cost per purchase, checked against the business's own order data. Read purchases from Ads Manager to compare the arms, then reconcile each arm's purchase count with orders in the shop's records for the same period, so a difference in tracking or attribution does not decide the test. The CPA calculator keeps the arithmetic consistent across arms; CPACost of each conversion. here is spend per purchase, not per click or per add to cart.
Three guardrails decide whether a lower cost is a real gain. New-customer share of purchases, matched against the customer file, is the most important: if broad targeting wins on cost but its purchases come mostly from existing customers, the decision rule keeps interests. FrequencyAverage times a person saw an ad. is the second: a broad audience is larger, so frequency should be lower; if it is higher, delivery is concentrating on a small group, often recent visitors. Average order value is the third, because cheaper purchases of smaller baskets can leave revenue flat.
Design
Use Meta's A/B test tool with targeting as the single variable. Build two ad sets in the same campaign structure: one with the account's best detailed interest targeting, the other with broad or Advantage+ audience. Creative, placements, optimisation event, budget, schedule and attribution setting are identical. Apply the same exclusions to both arms, normally a Custom AudienceAn ad audience built from the advertiser's own sources: customer lists, site visitors or engagers. of existing customers and recent purchasers, using the methods in the audience exclusions and overlap guide.
| Design element | Choice | Reason |
|---|---|---|
| Unit of randomisation | User, assigned by the A/B test tool | Keeps each person in one arm, so audiences do not overlap |
| Arms | Detailed interests versus broad or Advantage+ audience | One change: targeting breadth |
| Creative, budget, optimisation event | Identical, frozen for the test | Isolates targeting from the other levers |
| Exclusions | Same customer exclusions in both arms | Makes the test about prospecting, not retention |
| Primary metric | Cost per purchase, reconciled with own orders | Judges on the outcome the business pays for |
| Learning | Both arms through the learning phase before the read | Different learning states distort early cost |
If the account already uses a Lookalike AudienceAudience similar to a seed list. as its prospecting audience, it can be the control instead of interests; the decision is still whether to hand selection to the platform. Choose the control the account would really run, and keep the comparison to two arms unless the account has the volume to feed three. The academy exercise on planning a model railway retailer's tests of platform automation practises this design with the interests the account already runs as control and the customer exclusion matched in both arms, then schedules it ahead of an optimisation event test on the same platform so that test keeps whichever targeting wins.
Sample size reasoning
The arms are compared on purchases, so the test needs enough purchases per arm to separate the smallest cost difference that would change how the account is built. A small difference is not worth rebuilding the prospecting structure for, so set the threshold first. The sample size and significance guide sets out how the purchase rate and that difference fix the run length, and why cost per purchase needs more data than a click metric.
If the account cannot produce enough purchases per arm within a tolerable period, either test on the largest prospecting campaign, accept that only a large difference can be read, or decide on judgement and record that no test supported the choice.
Duration
Run both ad sets through the learning phase, then at least two full weekly cycles of stable delivery. Meta describes the learning phase as the period after launch or a significant edit when delivery is still being calibrated and performance is less stable, so read neither arm inside it. Do not edit either ad set during the test; a significant edit returns that arm to learning and puts the arms in different states.
Reading the results and the decision rule
Read the test once, at the planned end. Compare cost per purchase between arms with an interval rather than the point estimate, reconcile purchases with the shop's orders, then check new-customer share, frequency and order value. Write the decision rule into the plan before launch:
| Result | Decision |
|---|---|
| Broad wins on cost per purchase, new-customer share holds | Move prospecting to the broad setup; keep the customer exclusion in place |
| Broad wins on cost but new-customer share falls | Keep interests; broad is mostly reaching existing customers |
| Interests win on cost | Keep interests; revisit when creative or signal quality changes |
| No reliable difference | Choose the setup that is cheaper to manage, usually broad, and watch new-customer share |
If broad delivery keeps finding existing customers despite exclusions, the troubleshooting guide for Advantage+ audience reaching customers covers the causes and fixes before any retest.
A skincare brand tests broad targeting against its interest set
Orrinvale Skincare, an invented online skincare brand, prospects with an interest set built around natural beauty and wellbeing. It runs an A/B test with an Advantage+ audience arm, excluding customers from both, at £150 a day per arm for four weeks after learning. The interest arm records 190 purchases at £31.60; the broad arm records 236 at £25.40. Matched against the customer file, new-customer share is 81% in the interest arm and 78% in the broad arm, and average order value is within a pound. Frequency is lower in the broad arm. Orrinvale moves prospecting to the broad setup and keeps the exclusion.
Pitfalls
Existing customers inflating the broad arm is the pitfall that most often produces the wrong decision. The customer exclusion helps, but lists go stale and match rates are imperfect, so the new-customer guardrail must be read from the business's records, not assumed. Different learning states are the second: if one arm leaves learning a week after the other, its early cost drags down its average, so read only the period after both have stabilised.
Creative changed mid-test is the third. A new ad added to one arm, or a winning ad paused in one, makes creative the variable. Freeze creative for the test, and test creative separately with the creative testing guide once targeting is settled.
Common mistakes
Targeting tests fail more often on their setup than their arithmetic. The ad creative hook test varies the creative with targeting fixed and is the mirror of this design, and the experiment library lists every design.
- Running two ad sets outside the A/B test tool. The audiences overlap and the comparison is unfair.
- Leaving customers in the broad arm only. Exclusions must be identical in both arms.
- Judging on platform purchases alone. Reconcile with the shop's orders.
- Testing broad against a weak interest set. Compare with the targeting you would really run.
- Editing during the test. A significant edit resets learning in one arm.
- Ignoring new-customer share. Cheap repeat orders are not prospecting success.
