The test-and-learn loop
The experimentation loop framework describes the cycle: observe, hypothesise, test, learn, act. In paid acquisition each turn has a specific meaning. Observe where the result falls short of the target and at which stage of the chain from impression to margin. Hypothesise which lever would close the gap and by roughly how much. Test that lever in a design the platform cannot flatter. Learn by reading the result against the prediction, not against hope. Act by rolling out what won, scaling it until the guardrails trip, and recording the decision.
Scaling is part of the loop, not a separate activity. Each winning test raises the question of how much more budget the winner can absorb at an acceptable marginal cost, and answering it is itself a test. Each scaling step that hits diminishing returnsEach extra unit of spend or effort buys less extra result than the one before it. raises the question of which lever would push the ceiling up, which is the next hypothesis.
The loop runs at a pace the data can support. A team with abundant conversions can run several tests in a month; one with few should run fewer, longer tests. The experimentation cadence guide in the growth root covers how to set that rhythm across a whole programme. The optimisation hub places this loop alongside reporting.
What to test, in what order
Levers differ in how much they can move results and what they cost to test. Ordering them stops a team spending a quarter on bid tweaks while the creative is the real constraint.
| Lever | Expected effect | Cost to test | Notes |
|---|---|---|---|
| Offer and pricing | Large; changes whether people act at all | High; needs business sign-off | Often outside the marketer's control, and worth arguing for |
| Creative concept | Large, especially in paid social | Medium; needs production | New concepts beat variations on old ones |
| Landing page | Large; multiplies every click | Medium; needs development | Speed, clarity of offer, proof and friction |
| Conversion event and value | Large where the event is wrong | Low to medium | Fix before any other test if tracking is unreliable |
| Audience approach | Medium | Low | Broad against defined, seeded against interest-led |
| Structure | Medium; mostly via data volume | Low, but resets learning | Consolidation tests often win |
| Bidding strategy and target | Small to medium | Low | Automation already does most of this |
| Ad copy and creative variations | Small per test | Low | Useful once a strong concept exists |
The order is not fixed. If the conversion event is wrong, fix it first, because every other test will be read through a broken measure. If creative has been tested heavily and the landing page never, the landing page moves up. The Meta creative testing guide covers creative test mechanics on the platform where creative matters most. Beyond individual concepts, how often to replace creative is itself testable, as the creative refresh cadence test shows.
Designing a fair test across platforms
A fair test changes one thing, randomises who sees each version, runs long enough to separate signal from noise, and measures an outcome both versions can be compared on. Each of those is harder in paid media than it sounds.
Randomisation. Two campaigns launched side by side are not a test: the platform's delivery system sends them to different people at different prices. Use the platform's built-in experiment or split testing tools, which Google's and Meta's help centres document, so that the audience is divided randomly and auction overlap is controlled. For an A/B TestControlled comparison of two variants. on a landing page, split traffic on the site rather than in the ad platform.
One variable. A new creative with a new audience and a new landing page is three tests at once, and whichever wins teaches nothing reusable. If several changes must ship together, call it a package test and accept that you cannot attribute the result.
Duration and volume. A test should run across at least full weekly cycles and collect enough conversions for the difference to be meaningful. The statistical mechanics of sample size and significance belong to the CRO section, and the A/B testing guide covers them in full. Decide duration in advance and do not stop early because a variant is ahead.
Cross-platform tests. Questions such as whether a channel adds sales at all, or whether moving budget from one platform to another helps, cannot be answered inside a single platform. They need geographic or time-based holdouts measured in the business's own data, which is where IncrementalityConversions caused by marketing. tests come in.
Reading results without fooling yourself
Most test errors happen at the reading stage. A few habits prevent the commonest ones.
- Write the prediction first. Record which variant you expect to win and by how much before the test starts. A result read without a prediction can always be explained.
- Look at the business outcome. A creative that wins on click-through rate but loses on cost per acquisition (CPACost of each conversion.) has lost. Use the outcome the target is set on, ideally confirmed in your own data.
- Respect the attribution window. Some variants produce slower conversions; a test read before the window closes will favour the faster one.
- Distrust large wins on small samples. The bigger and earlier the apparent effect, the more likely it is noise or a novelty effectA temporary change in behaviour caused by a change being new, which fades as users get used to it. that will fade.
- Check the rollout. A winning variant should keep winning when rolled out. If results revert, the test was wrong or the conditions changed, and either way it is a finding.
The widget below shows how sample size and the true difference between variants decide whether a test can reach a reliable answer.
A/B test significance checker
Scaling paths: budget, audiences, channels, markets
Once something works, there are four ways to buy more of it, roughly in increasing order of effort and risk.
Budget. Raise spend on the winning campaign. It is the fastest path and the first to hit diminishing returns. Platforms' help centres advise gradual changes on automated campaigns, because large jumps can reset learning, so raise in steps and read the marginal result after each. The Meta scaling guide covers the platform-specific tactics. For Google Ads, the guide to scaling Google Ads spend orders the levers inside one account by risk, from budget-limited campaigns to new campaign types.
Audiences. Widen who the campaign can reach: broader targeting, new seed audiences, new placements. It extends the pool before the budget path saturates it, and it usually needs fresh creative suited to the newer, colder audience.
Channels. Take the winning proposition to another platform with a different audience. The concept may transfer; the execution rarely does without adaptation, and the new channel needs its own role and target.
Markets. Take the proposition to a new country or region. It is the largest path and the most demanding, with new pricing, language, regulation and competition, and it is closer to a new launch than to scaling.
Diminishing returns, marginal cost and guardrails
Every scaling path eventually hits diminishing returns: each further increment buys fewer or more expensive results, because the most responsive people were reached first and FrequencyAverage times a person saw an ad. on the rest rises. The average cost per result climbs slowly while the marginal cost, the cost of the last increment of results, climbs fast. Scaling decisions should be made on the marginal figure. In cost terms that figure is the marginal CPAThe extra spend divided by the extra acquisitions it produced when spend changed; usually above the average, and the figure scaling and cutting decisions need.: a step's extra spend divided by the extra results it bought. Return on ad spend (ROASRevenue generated per unit of ad spend.) works the same way.
Guardrails turn this into practice. Before scaling, agree three things: the marginal cost at which scaling stops, the signals that trigger a pause, and the review point at which the step is judged. Typical signals are marginal cost per result above target, frequency rising steadily, new-customer shareThe proportion of conversions or orders that come from first-time customers, flagged from the business's own records rather than a platform's report. falling, and the blended cost in the business's own data moving against the platform's figures. A CPA calculator helps convert a budget step and its incremental results into a marginal cost.
A fitness app scales until the marginal cost says stop
Stridewell, a running app, found a creative concept that cut cost per subscription on Meta from £21 to £15 in a split test. With a target of £24, the team scaled budget in 20% steps each week from £4,000. Average cost per subscription rose gently: £15, £16, £17, £18. Marginal cost per subscription, calculated from the extra spend and extra subscriptions at each step, rose faster: £19, £22, £27. At the third step the marginal cost passed the target, so the team held budget at £5,760 a week, briefed new creative for a broader audience, and started a geographic test on TikTok as the next scaling path.
Results of each test and each scaling step should be recorded where the next review will see them; the performance reporting guide describes the decision log that holds them. The marketing decision log template gives each scaling step a row with its expected effect, guardrail and check date.
If a scaling step raised spend without raising sales, the flat sales after a budget increase diagnosis separates diminishing returns from spend absorbed by existing demand, saturated audiences and results read too soon.
Before a budget step begins, the tutorial on using the marketing budget calculator for scaling spend turns the extra results wanted and the last step's marginal cost into the increment and its monthly pace.
Common mistakes
- Comparing two live campaigns and calling it a test. Without randomisation, the platform's delivery decides the winner.
- Testing bids while the creative is the constraint. Order the levers by expected effect; small levers first wastes a quarter.
- Stopping a test when a variant pulls ahead. Early leads are often noise; decide duration in advance and keep to it.
- Judging on the platform's metric rather than the business outcome. Click-through and platform-attributed conversions can disagree with what the business recorded.
- Scaling on average returns. The average hides the marginal cost; stop when the last increment breaks the target.
- Large, sudden budget jumps. Automated campaigns can re-enter learning and give a misleading read on whether scaling worked.
- Not recording results. A test nobody can find six months later will be run again.
