Hypothesis
Switching from manual bidding to a target-based automated strategy will deliver more conversions at the same spend than matched control regions that keep the current strategy. The reasoning is that automated strategies can use auction-time signals no manual bidder sees, and so place each bid closer to the value of that particular search. The smart bidding strategies guide explains those signals and the conditions under which automated strategies tend to work well, such as enough conversion data and a reliable conversion action.
The hypothesis is phrased at equal spend for a reason. Automated strategies often change how much they spend, and a strategy that spends more will usually deliver more conversions without being more efficient. Fixing spend makes the comparison about efficiency. Record which strategy and which target will be used, how the target was chosen, and the evidence that the account has enough conversion volume, in the experiment hypothesis template.
Metrics and guardrails
The primary metric is conversions per region at equal spend: conversions in test regions compared with what the control regions predict they would have been, after accounting for differences in the pre-period. Count conversions in your own systems where possible, using the same definition the bidding strategy optimises for, so that the strategy is judged on the outcome it was given.
CPACost of each conversion. is the first guardrail. If spend drifts despite your controls, cost per acquisition shows whether the extra conversions came at an acceptable price. Conversion value per conversion is the second: an automated strategy optimising for conversion count may favour cheaper, lower-value conversions, such as smaller orders or less qualified leads. If the business uses value-based targets, compare ROASRevenue generated per unit of ad spend. as well. A strategy that wins on count while losing on value has moved spend towards the wrong customers.
Design
Divide the market into regions that can be targeted separately, such as cities, counties or postal areas, and that are large enough to generate steady conversions. Match them into pairs or groups using a pre-period of historical data: regions whose conversions moved together over time make good matches. Assign regions at random within each matched group to test or control. The geo-targeted campaigns guide explains how to set up location targeting so each region receives only its own campaign.
| Design element | Choice | Reason |
|---|---|---|
| Unit of randomisation | Region, within matched groups | Bidding changes cannot be split by person here |
| Variants | Current manual bidding versus target-based automated strategy | One strategy change |
| Pre-period | Several weeks of history before the switch | Establishes how test and control normally relate |
| Learning period | Run but excluded from the read | Strategy calibration is not representative |
| Spend | Held level relative to the pre-period | Makes the test about efficiency |
| Read period | Whole weeks after learning ends | Covers weekly patterns |
The learning period length depends on the strategy and the conversion volume; the help centre documents how the platform reports learning status. Treat the read period as starting only once the test regions are out of learning.
Sample size reasoning
In a geo test, the sample is the number of regions and the number of weeks, not the number of searches. With few regions, a single unusual region can swing the result, so more, smaller regions usually give a more reliable answer than a handful of large ones, as long as each region still produces steady conversions. The pre-period tells you how noisy the relationship between test and control regions is; the noisier it is, the longer the read period must be. The incrementality testing guide covers the methods used to estimate this.
For the general logic of effect size, baseline and confidence, use the sample size and significance guide. The conversion rate calculator is useful for checking the post-click conversion rate in each region, which helps explain the result.
Reading the results
At the end of the read period, compare test regions' actual conversions with the conversions predicted from control regions and the pre-period relationship. The difference is the effect of the strategy switch. Confirm first that spend stayed level; if it did not, adjust the read to conversions per unit of spend and say so in the record.
Then check the guardrails. A gain in conversions with stable cost per acquisition and stable value per conversion is a clean win; roll out the strategy region by region and keep watching. A gain in count with a fall in value suggests switching to a value-based target. A flat result often means the manual bidding was already close to efficient for this account, or that conversion volume is too low for the automated strategy to learn well. Read the result per matched group too, so one group does not carry the whole finding.
A regional furniture retailer tests automated bidding by city
Oakline, an invented furniture retailer, advertises in twelve UK cities. It matches them into six pairs using sixteen weeks of conversion history and randomly switches one city in each pair to a target-based automated strategy. Spend is held level. After a two-week learning period, the read runs for six weeks. Test cities record 1,140 conversions against a predicted 1,010 from controls, roughly 13% more at equal spend. Cost per acquisition falls, and average order value per conversion holds steady. Oakline rolls the strategy out across all cities.
Pitfalls
Spill-over is the first pitfall. People live in one region and search from another, and location targeting is not perfect, so some test-region ads reach control-region people and vice versa. Choose regions with clear boundaries and leave buffer regions out of the test where commuting is heavy.
Unmatched regions are the second. Randomly assigning regions without matching on history can leave the test group with all the fast-growing cities. Use the pre-period to match, and check that the matched groups track each other before the switch. Reading during learning is the third: the early weeks after a strategy change often look worse, and a team that reads too early abandons a strategy that would have worked. Finally, watch for anything else that changes in only some regions, such as a local promotion, a store opening or a competitor's campaign, and note it in the record.
Common mistakes
Geo tests are slow and expensive, so a mistake costs weeks rather than days. The same matched-region method answers the scaling question in the budget increase incrementality test, and the experiment library routes simpler questions to simpler designs.
- Letting spend float. A strategy that spends more will win on count without being better.
- Skipping the pre-period. Without it, you cannot tell normal regional differences from the effect.
- Including the learning period. Calibration costs are temporary and distort the read.
- Too few regions. One unusual city can decide the result.
- Changing targets mid-test. Each change restarts learning and muddies the comparison.
- Judging on platform-reported conversions alone. Use your own sales or lead data where possible.
