Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Experiment Library

Creative Refresh Cadence Test

An experiment design for how often to replace ad creative: a scheduled refresh arm against a run-until-fatigue arm, judged on cost per result over time.
Experimentadvanced6 min readUpdated

Hypothesis

If one arm replaces its ads on a fixed schedule while the other keeps winners until fatigue shows, the scheduled arm will hold cost per result steadier over the period, but only if that gain outweighs the extra production cost and the relearning each refresh triggers. The decision behind it is concrete: replace ad creative on a fixed schedule, or keep running winners until fatigue signals appear.

The two policies trade one cost for another. A schedule refreshes before Creative Fatigue can bite, but it retires ads that might have run profitably for weeks more, needs a steady supply of new creative and introduces edits that can send ad sets back into learning. Waiting for fatigue wastes nothing on early replacement but pays for the weeks in which a tiring ad gets dearer before the signal is clear. The frequency and creative fatigue guide argues that refresh is best decided by signals rather than the calendar; this test checks that for one account, with its own audience size, budget and production costs.

Metrics and guardrails

The primary metric is cost per result over the whole test period, including production cost: media spend plus the cost of producing the creative each arm used, divided by the results the arm delivered. Reading the whole period matters because the policies differ over time, not at any one moment; a scheduled arm may look better in the weeks after each refresh and worse during each relearning. CPA computed with the CPA calculator on spend plus production keeps the comparison on the true cost.

Three guardrails explain the result. Frequency per arm shows how far the run-until-fatigue arm lets repetition build and whether the scheduled arm actually kept it lower. Learning phase resets count how often each policy's edits sent an ad set back into the Learning Phase, which Meta and TikTok both describe as following significant edits; frequent resets are a hidden cost of a tight schedule. The click-through rate trend, read across the period, shows wear-out building in one arm and being reset in the other.

Design

Randomise at region level, or across mutually exclusive audiences, because rotation plays out over many weeks. Match regions on cost per result and volume over a pre-period and assign each matched group's members at random to the two policies. Both arms run the same campaign structure, budget share, targeting and optimisation, and both draw on one creative pool, produced to the same brief and standard, so the policy is the only difference.

Design elementChoiceReason
Unit of randomisationRegion, within matched groups, or exclusive audiencesEach policy needs its own audience for the whole period
ArmsScheduled refresh versus refresh on agreed fatigue signalsOne change: when ads are replaced
Creative poolShared, produced to one brief, assigned in the same orderKeeps creative quality out of the comparison
Fatigue ruleWritten before launch and applied mechanicallyPrevents week-by-week judgement becoming a third policy
Primary metricCost per result including production, over the whole periodCounts every cost of each policy
Pre-periodSeveral weeks of cost per result per unitShows how the units normally relate

The creative pool deserves the most care. If the scheduled arm draws more new ads, it must get them in the same order and at the same standard as the other arm would; the creative production playbook describes how a team keeps a pipeline of concepts at a consistent standard. How individual ads are compared belongs to the creative testing guide; this test takes the pool as given and varies only when it is used.

Sample size reasoning

The comparison is a cost trend over weeks rather than a single conversion rate, so the sample is the number of matched units and weeks. A handful of regions over a few refresh cycles gives a small effective sample, and the incrementality testing guide explains why few units mean only a large difference can be read. Decide before launch how large a difference in cost per result, including production, would change the account's policy, and check that the units and weeks available can see it.

If they cannot, either run the test on the account's largest market only, extend it to more refresh cycles, or make the decision on judgement and record the reasoning, rather than reading a noisy trend as a result.

Duration

Run for several refresh cycles of the scheduled arm, long enough for fatigue to appear in the other arm. A test that ends before the run-until-fatigue arm has tired measures nothing, because the policies only differ once fatigue starts. Long tests cross seasons, so match the units on the pre-period and make sure both arms face the same calendar; avoid starting just before a peak that will end mid-test.

Reading the results and the decision rule

Read the test once, at the end of the planned period. Compare cost per result including production between arms, with an interval derived from the matched units, then read frequency, learning resets and the click-through trend. Write the decision rule into the plan before launch:

ResultDecision
Scheduled arm has lower cost per result including productionAdopt scheduled refreshes at the tested cadence
Scheduled arm lower on media cost but higher once production is addedRefresh on fatigue signals; spend the production budget on new concepts
Run-until-fatigue arm lower overallRefresh on fatigue signals and keep a reserve of tested ads ready
No reliable differenceRefresh on fatigue signals, which needs less production, and revisit when budget or audience size changes

Policy results depend on spend and audience size, so revisit them when either changes materially; the scaling campaigns guide explains how higher budgets speed up frequency and fatigue.

Illustrative example — not a real result

A bike retailer tests a fortnightly refresh

Fenmoor Cycles, an invented online bicycle retailer, runs prospecting in ten regions matched into five pairs on twelve weeks of cost per order. For twelve weeks, one region in each pair replaces its lead ad every fortnight from a shared pool; the other replaces an ad only when frequency rises and click-through falls for two consecutive weeks. The scheduled arm uses eleven new ads and the other arm five, at £400 each to produce. Media cost per order is £38.10 in the scheduled arm and £39.60 in the other, but with production included the figures are £40.30 and £40.60, with intervals that overlap widely. Fenmoor keeps refreshing on fatigue signals and puts the saved production budget into new concepts.

Pitfalls

New creative quality differing between arms is the pitfall that most often decides these tests unnoticed. If the scheduled arm's extra ads are rushed, it loses on quality, not on policy; if the best concepts happen to go to one arm first, it wins on luck. Produce to one brief and assign in a fixed order. Seasonality across a long test is the second: a test spanning weeks crosses promotions and demand shifts that hit units unevenly, so note every such event and check that the matched pairs still track each other.

Production cost left out is the third, and it flatters the scheduled arm every time. Record the cost of every ad produced for each arm, including design time and any paid creators, and add it to spend before reading.

Common mistakes

Policy tests fail on discipline more often than on arithmetic. The ad creative hook test compares creative variants at one moment and is the right design for choosing what goes into the pool, and the experiment library lists every design.

  • Leaving production cost out. The scheduled arm then wins by construction.
  • Judging the fatigue rule week by week. Write it down and apply it mechanically.
  • Giving the arms different creative. The test then compares ads, not policies.
  • Ending before fatigue appears. The policies only differ once ads tire.
  • Ignoring learning resets. Each refresh has a relearning cost.
  • Applying the result after a big budget change. Spend changes how fast ads tire.

Frequently asked questions

Why include production cost in the primary metric?

Because a scheduled refresh policy needs more new ads than a run-until-fatigue policy, and those ads cost money and time to make. A policy that lowers media cost per result while doubling production spend may cost the business more overall. Adding production cost to spend before dividing by results puts both policies on the true cost of running them.

Why not test this inside one ad set with the platform's A/B test tool?

Rotation policies play out over many weeks and several refresh cycles, longer than most platform split tests are designed to run, and each refresh is a set of edits that the policy itself requires. Matched regions or mutually exclusive audiences let each policy run as it would in practice for the whole period, with its own learning resets and its own fatigue.

What counts as a fatigue signal for the run-until-fatigue arm?

Agree it before launch and write it down: typically rising frequency together with falling click-through rate and rising cost per result against the ad's own early weeks, as the frequency and creative fatigue guide describes. The arm replaces an ad only when that rule fires, so the policy is mechanical rather than a judgement made week by week.

Terms used on this page

Tools to try

Frameworks

Also related

Sources

  1. Meta Business Help Centre — Meta
  2. About the learning phase — Meta Business Help Centre
  3. TikTok Ads Help Center — TikTok for Business

Related