Hypothesis
If one arm replaces its ads on a fixed schedule while the other keeps winners until fatigue shows, the scheduled arm will hold cost per result steadier over the period, but only if that gain outweighs the extra production cost and the relearning each refresh triggers. The decision behind it is concrete: replace ad creative on a fixed schedule, or keep running winners until fatigue signals appear.
The two policies trade one cost for another. A schedule refreshes before Creative FatigueThe decline in an ad's response as the same people see it again and again. can bite, but it retires ads that might have run profitably for weeks more, needs a steady supply of new creative and introduces edits that can send ad sets back into learning. Waiting for fatigue wastes nothing on early replacement but pays for the weeks in which a tiring ad gets dearer before the signal is clear. The frequency and creative fatigue guide argues that refresh is best decided by signals rather than the calendar; this test checks that for one account, with its own audience size, budget and production costs.
Metrics and guardrails
The primary metric is cost per result over the whole test period, including production cost: media spend plus the cost of producing the creative each arm used, divided by the results the arm delivered. Reading the whole period matters because the policies differ over time, not at any one moment; a scheduled arm may look better in the weeks after each refresh and worse during each relearning. CPACost of each conversion. computed with the CPA calculator on spend plus production keeps the comparison on the true cost.
Three guardrails explain the result. FrequencyAverage times a person saw an ad. per arm shows how far the run-until-fatigue arm lets repetition build and whether the scheduled arm actually kept it lower. Learning phase resets count how often each policy's edits sent an ad set back into the Learning PhaseThe period after launch or a significant edit when an ad set's delivery is still being optimised., which Meta and TikTok both describe as following significant edits; frequent resets are a hidden cost of a tight schedule. The click-through rate trend, read across the period, shows wear-out building in one arm and being reset in the other.
Design
Randomise at region level, or across mutually exclusive audiences, because rotation plays out over many weeks. Match regions on cost per result and volume over a pre-period and assign each matched group's members at random to the two policies. Both arms run the same campaign structure, budget share, targeting and optimisation, and both draw on one creative pool, produced to the same brief and standard, so the policy is the only difference.
| Design element | Choice | Reason |
|---|---|---|
| Unit of randomisation | Region, within matched groups, or exclusive audiences | Each policy needs its own audience for the whole period |
| Arms | Scheduled refresh versus refresh on agreed fatigue signals | One change: when ads are replaced |
| Creative pool | Shared, produced to one brief, assigned in the same order | Keeps creative quality out of the comparison |
| Fatigue rule | Written before launch and applied mechanically | Prevents week-by-week judgement becoming a third policy |
| Primary metric | Cost per result including production, over the whole period | Counts every cost of each policy |
| Pre-period | Several weeks of cost per result per unit | Shows how the units normally relate |
The creative pool deserves the most care. If the scheduled arm draws more new ads, it must get them in the same order and at the same standard as the other arm would; the creative production playbook describes how a team keeps a pipeline of concepts at a consistent standard. How individual ads are compared belongs to the creative testing guide; this test takes the pool as given and varies only when it is used.
Sample size reasoning
The comparison is a cost trend over weeks rather than a single conversion rate, so the sample is the number of matched units and weeks. A handful of regions over a few refresh cycles gives a small effective sample, and the incrementality testing guide explains why few units mean only a large difference can be read. Decide before launch how large a difference in cost per result, including production, would change the account's policy, and check that the units and weeks available can see it.
If they cannot, either run the test on the account's largest market only, extend it to more refresh cycles, or make the decision on judgement and record the reasoning, rather than reading a noisy trend as a result.
Duration
Run for several refresh cycles of the scheduled arm, long enough for fatigue to appear in the other arm. A test that ends before the run-until-fatigue arm has tired measures nothing, because the policies only differ once fatigue starts. Long tests cross seasons, so match the units on the pre-period and make sure both arms face the same calendar; avoid starting just before a peak that will end mid-test.
Reading the results and the decision rule
Read the test once, at the end of the planned period. Compare cost per result including production between arms, with an interval derived from the matched units, then read frequency, learning resets and the click-through trend. Write the decision rule into the plan before launch:
| Result | Decision |
|---|---|
| Scheduled arm has lower cost per result including production | Adopt scheduled refreshes at the tested cadence |
| Scheduled arm lower on media cost but higher once production is added | Refresh on fatigue signals; spend the production budget on new concepts |
| Run-until-fatigue arm lower overall | Refresh on fatigue signals and keep a reserve of tested ads ready |
| No reliable difference | Refresh on fatigue signals, which needs less production, and revisit when budget or audience size changes |
Policy results depend on spend and audience size, so revisit them when either changes materially; the scaling campaigns guide explains how higher budgets speed up frequency and fatigue.
A bike retailer tests a fortnightly refresh
Fenmoor Cycles, an invented online bicycle retailer, runs prospecting in ten regions matched into five pairs on twelve weeks of cost per order. For twelve weeks, one region in each pair replaces its lead ad every fortnight from a shared pool; the other replaces an ad only when frequency rises and click-through falls for two consecutive weeks. The scheduled arm uses eleven new ads and the other arm five, at £400 each to produce. Media cost per order is £38.10 in the scheduled arm and £39.60 in the other, but with production included the figures are £40.30 and £40.60, with intervals that overlap widely. Fenmoor keeps refreshing on fatigue signals and puts the saved production budget into new concepts.
Pitfalls
New creative quality differing between arms is the pitfall that most often decides these tests unnoticed. If the scheduled arm's extra ads are rushed, it loses on quality, not on policy; if the best concepts happen to go to one arm first, it wins on luck. Produce to one brief and assign in a fixed order. Seasonality across a long test is the second: a test spanning weeks crosses promotions and demand shifts that hit units unevenly, so note every such event and check that the matched pairs still track each other.
Production cost left out is the third, and it flatters the scheduled arm every time. Record the cost of every ad produced for each arm, including design time and any paid creators, and add it to spend before reading.
Common mistakes
Policy tests fail on discipline more often than on arithmetic. The ad creative hook test compares creative variants at one moment and is the right design for choosing what goes into the pool, and the experiment library lists every design.
- Leaving production cost out. The scheduled arm then wins by construction.
- Judging the fatigue rule week by week. Write it down and apply it mechanically.
- Giving the arms different creative. The test then compares ads, not policies.
- Ending before fatigue appears. The policies only differ once ads tire.
- Ignoring learning resets. Each refresh has a relearning cost.
- Applying the result after a big budget change. Spend changes how fast ads tire.
