Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Experiment Library

Brand Search Holdout Test

An experiment design for deciding whether to keep bidding on your own brand terms: matched regions paused, paid and organic brand clicks read together.
Experimentadvanced8 min readUpdated

Hypothesis

If brand search ads are paused in a set of matched regions, total brand-search conversions there will fall by far less than the paid conversions the platform reported, because organic listings recapture most clicks unless competitors bid on the brand. The decision behind it is concrete: keep, trim or stop paid search ads on the company's own brand terms, and where.

The hypothesis is written to be falsified in two directions. Brand campaigns report conversions that look cheap, because the people searching for a company by name were already looking for it. Some of those conversions would have arrived through the organic result anyway, and the test measures how many. But the results page is shared: when a brand stops advertising on its own name, a competitor, a reseller or an affiliate can take the top position, and then the pause loses real business. The SEO versus PPC guide describes this cannibalisation of clicks and why its size is found by testing rather than assumed. Record the regions, the brand terms in scope and the expected behaviour of each in the experiment hypothesis template before anything is paused.

Metrics and guardrails

The primary metric is incremental conversions from brand searches, counting paid and organic together. In each region, add the conversions from brand ads to the conversions from organic brand visits, then compare treated regions with what the control regions predict they would have been. That combined figure is the incremental answer: it shows what the business keeps when the ads stop, not what the ads were credited with. Count conversions in your own order or lead records where possible, because the platform cannot report the conversions it no longer drives.

Three guardrails explain the result. Competitor Impression Share on brand terms comes first: the auction insights available in Google Ads show which other advertisers appear on your brand queries and how often, and a rise in the paused regions is the clearest sign that the vacated position is being taken. The second is organic brand click-through rate, read from the Search Console Performance report, which shows clicks and impressions for queries and can be filtered by country; the Google Search Console guide covers how to filter it to brand queries. Where the regions are countries, that reads organic click-through by arm directly; where they are smaller areas inside one country, read organic brand visits by location from site analytics and use Search Console for the overall trend. If organic traffic rises in the paused regions, the organic listing is absorbing the paid clicks as the hypothesis expects. The third is cost per incremental conversion: brand spend in the control regions divided by the conversions the ads actually added, which the CPA calculator turns into a figure the finance team can compare with what a brand conversion is worth.

Design

Randomise at region level. Choose regions that can be targeted separately and that each carry steady Branded Search volume, match them into pairs or groups on their brand conversions over a pre-period, and assign one region in each group at random to the treated arm. In treated regions, brand campaigns are paused or their location targeting excludes the region; in control regions nothing changes. Generic and competitor campaigns keep running everywhere, because the question is about brand ads alone. How a competitor campaign is built and judged in its own right is set out in the guide to bidding on competitor names. The treated regions together form the Holdout Group.

Design elementChoiceReason
Unit of randomisationRegion, within matched groupsBrand queries cannot be split by person without the ads still showing to some
ArmsBrand ads paused in treated regions, unchanged in controlOne change, read against regions that kept it
Primary metricBrand conversions, paid plus organic, per regionMeasures what the business keeps, not what the ads were credited with
Other campaignsUnchanged everywhereIsolates the brand ads from the rest of the account
Pre-periodSeveral weeks of brand conversions per regionShows how treated and control regions normally move together
Competitor watchAuction insights on brand terms, by armConquesting is the outcome that reverses the decision

How brand campaigns are separated from generic ones matters before the test starts. If brand terms also run inside generic or shopping campaigns, pausing the brand campaign moves the traffic rather than stopping it; the account structure guide explains how brand and non-brand are kept apart, and brand terms should be added as negatives elsewhere in the treated regions. The same matched-region method is laid out in more detail in the bidding strategy geo split test, which uses it to compare bid strategies rather than to pause a campaign.

Sample size reasoning

In a regional test the sample is the number of matched regions and the number of weeks, not the number of searches. Power comes from how steadily the regions' brand conversions moved together before the test: if treated and control regions tracked each other closely in the pre-period, a modest fall after the pause stands out; if they wandered apart week to week, only a large fall can be read. Write down the smallest fall in brand conversions that would change the decision, usually the point at which the conversions lost would cost more than the brand spend saved, and use the region logic in the incrementality testing guide to check that the available regions can detect it.

If they cannot, do not run the test anyway and read a noisy result as evidence. Use more and smaller regions where each still carries steady brand volume, lengthen the pause, or narrow the question to the markets where brand spend is largest and the answer is worth most.

Duration

Run a pre-period long enough to show the regions moving together, then pause for at least one full buying cycle, followed by a short cool-down in which the ads return and the regions are watched for any lasting effect. Avoid launches, promotions and seasonal peaks, which change brand demand unevenly and can land in one arm more than the other. A brand pause that is too short also misses competitor behaviour: rivals and affiliates do not always react within days, and the guardrail needs time to show whether anyone moves into the space.

Reading the results and the decision rule

Read the test once, after the pause and the cool-down. Compare treated regions' combined brand conversions with the prediction from control regions and the pre-period relationship, report the interval rather than the point estimate, then read the guardrails by region. Write the decision rule into the plan before launch:

ResultDecision
Combined brand conversions barely fall and competitor impression share stays lowTrim or stop brand bidding in markets like the treated ones; keep watching auction insights
Combined conversions fall and competitors take the vacated positionsKeep brand ads; they are defending the position, and the cost is the price of that defence
Combined conversions fall with no competitor, at a cost per incremental conversion below what a brand conversion is worthKeep brand ads; they add conversions at an affordable price
Cost per incremental conversion exceeds a brand conversion's value and no conquesting appearsTrim or stop, and reinvest the budget where incremental return is higher

Put the result next to the wider allocation question. The brand versus performance guide explains why brand demand is built by more than search ads, and the test answers only the narrower question of whether paying for your own name adds conversions once that demand exists.

Illustrative example — not a real result

A window installer pauses brand ads in half its regions

Ashgrove Glazing, an invented double-glazing installer, spends £4,000 a month on brand search across fourteen regions. It matches them into seven pairs on twelve weeks of brand enquiries and pauses brand ads in one region of each pair for six weeks. The platform had credited brand ads in the treated regions with about 180 enquiries over a comparable period. Against the control prediction, combined paid and organic brand enquiries in the treated regions fall by roughly 25, with an interval of about 5 to 45. Competitor impression share on brand terms stays low in six pairs but rises sharply in one city, where a rival starts bidding. Ashgrove stops brand ads in the six quiet regions and keeps them in the contested city.

Pitfalls

Competitor conquesting is the pitfall that changes the answer. A pause can look harmless in its first weeks and costly once a rival notices; read the competitor guardrail by region and over the whole pause, not at the start. Affiliates bidding on the brand are the second. Where affiliate terms allow brand bidding, an affiliate can take the paid position and then claim commission on sales that the organic listing would have delivered; the affiliate reporting and incrementality guide covers how those terms are set and checked, and the guide to setting brand bidding rules for affiliates covers the term list, exceptions and monitoring that keep partners off the brand's name before the test starts.

Promotions landing in one arm are the third. A sale, a press story or a television burst that reaches some regions more than others shifts brand demand and swamps the effect of the pause. Note every such event in the record, and if one lands unevenly, extend the test or drop the affected pair. Finally, leakage: brand terms running through broad generic keywords or shopping campaigns in the treated regions mean the ads never truly stopped, so check the search terms in those campaigns during the pause.

Common mistakes

Brand holdouts fail more often on measurement than on statistics. The experiment library lists the other designs, and the Google Ads Help Centre documents the location targeting and auction insights reports this design depends on.

  • Counting paid conversions only. The organic listing recaptures much of what the ads claimed; count both.
  • Comparing before and after. Seasonality and promotions move brand demand; use control regions.
  • Ignoring competitors. Conquesting is the result that reverses the decision, so watch it by region.
  • Leaving brand terms in other campaigns. The traffic moves instead of stopping, and the test reads nothing.
  • Pausing for a few days. Rivals and affiliates react slowly; a short pause misses the cost.
  • Applying one answer to every market. Read the result per region and keep ads where they defend.

Frequently asked questions

Why not simply pause brand ads everywhere for a fortnight and compare with the weeks before?

Because brand demand moves with seasons, promotions, press coverage and other channels, and a before-and-after comparison cannot separate those movements from the pause. Matched control regions that keep the ads experience the same calendar, so the difference between the two groups is the effect of the pause rather than the effect of the month.

Should the organic clicks really count in the primary metric?

Yes. The question is how many brand-search conversions the business keeps without the ads, and most people who searched for the brand still arrive through the organic listing. Counting paid conversions alone would show the pause as a loss of everything the ads were credited with, which is exactly the overstatement the test exists to correct.

What if a competitor starts bidding on our brand during the test?

Record it, because it is the result the guardrail is there to catch. Conquesting in the paused regions is evidence that brand ads protect the position, and the decision rule keeps them where it appears. Do not restart the ads mid-test unless the loss breaks the limit agreed before launch; a test stopped at the first sign of a competitor teaches nothing about the size of the effect.

Terms used on this page

Tools to try

Frameworks

Where this page is used

Also related

Sources

  1. Google Ads Help — Google Ads Help
  2. Performance report (Search results) — Google Search Console Help

Related