Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Growth Marketing › Growth Fundamentals

Experimentation Cadence

How to run a growth experimentation cadence: the backlog and prioritisation, weekly and sprint rituals, test velocity versus test quality, and the learning log.
Guideintermediate9 min readUpdated

Why cadence beats heroics

Most teams' experiment history is a series of heroics: a big test when someone had a strong opinion, a redesign when a new leader arrived, a burst of activity before a board meeting. Between the bursts, nothing. The results of the bursts are remembered selectively, the losers quietly forgotten, and a year later nobody can say what the team knows that it did not know before.

A cadence replaces bursts with a rhythm. Every week some experiments start, some finish, and some are read out and recorded. The rhythm has three effects that heroics cannot produce. It makes learning cumulative, because every result goes into the same log against the same hypotheses. It makes prioritisation honest, because the backlog is compared as a whole rather than one idea against the enthusiasm of the moment. And it protects the programme from the loudest voice, because a strong opinion becomes a hypothesis in the queue rather than a decision.

The cadence also fixes the relationship between experiments and the growth model. The growth loops guide explains how a loop is mapped and where its leaks are; the cadence is how the team works on those leaks in order, so that each test claims a path to the north star and the KPI tree beneath it. Without that link, a team can run a hundred tests and improve a hundred local measures while the business stands still. The growth fundamentals hub describes the model the cadence serves.

The backlog and prioritisation

The backlog is a single list of candidate experiments, each written to a template that forces the author to think. A usable entry has: the leak or opportunity it addresses and the evidence for it; a hypothesis in the form "if we change this, then that measure will move, because"; the primary metric and the decision rule (what result ships it, what result kills it); the smallest version that would be informative; who is needed to build it; and an estimate of effort.

Prioritisation is where the backlog earns its keep. The usual scoring approaches weigh expected impact, confidence in the hypothesis and ease of execution, and the prioritising tests guide in the CRO section compares the common schemes and their failure modes. Two rules matter more than the scheme chosen. First, impact is scored against the north star or the loop stage, not against the metric the test happens to move, so that a test on a minor page cannot outrank a test on activation by promising a large local lift. Second, confidence comes from evidence (research, data, prior tests), not from seniority, and an entry with no evidence is scored low however senior its author.

The backlog is reviewed weekly and pruned monthly. Ideas that have sat unprioritised for a quarter are either rewritten with evidence or removed; a backlog of hundreds of unranked ideas is a graveyard, not a plan. A healthy backlog has a short ranked top, a middle of entries waiting for evidence or capacity, and a tail that is regularly cleared. The strongest entries often come from above the backlog: a learning agenda built from the plan's assumptions names the questions the year's plan cannot afford to get wrong and the dates by which each needs an answer.

Rituals

Rituals are the meetings and documents that happen on a schedule regardless of mood. The set below is the common shape; small teams merge several into one weekly hour, and the tutorial on running a weekly growth meeting sets out the agenda, pre-read and decisions for that merged hour.

RitualCadenceWhoOutput
Results read-outWeeklyGrowth lead, analyst, test ownersEach finished test called as win, loss or inconclusive against its decision rule; log entry written
PrioritisationWeeklyGrowth lead, product and engineering partnersNext tests pulled from the ranked backlog; capacity confirmed
Backlog groomingMonthlyGrowth lead, analystEntries rewritten with new evidence; stale entries removed; scoring recalibrated
Sprint or planning syncPer sprintGrowth team and the product team whose roadmap it sharesExperiments that need builds scheduled; conflicts with product releases resolved
Cohort and economics reviewMonthlyGrowth lead, analyst, finance partnerNorth star, loop stage rates and unit economics by cohort; experiments reassessed against them
Model reviewQuarterlyWhole growth team and stakeholdersGrowth model updated; next quarter's leak targets agreed; learning log themes summarised

The read-out is the ritual most often skipped and the one that matters most. Its discipline is that every test is called against the decision rule written before it started, by someone other than the person who built it, and that the call is recorded the same day. An inconclusive result is recorded as inconclusive, not reinterpreted as a narrow win.

Velocity vs quality

Test velocity, the number of experiments completed per period, is a tempting measure of a programme because it is easy to count and easy to raise. It is also the measure most likely to corrupt the programme. Pushed too hard, velocity produces small tests on trivial pages, tests stopped the moment a difference appears, and a log full of "wins" that never show up in the business.

Quality has its own failure mode. A programme that insists on perfect tests runs few of them, waits months for sample sizes on low-traffic pages, and learns slowly. The balance depends on traffic and on where in the loop the test sits.

The practical rules are these. Tests on stages with high traffic (landing pages, sign-up forms, checkout) can run as proper A/B Test designs with pre-registered sample sizes, and the A/B testing guide sets out how. Tests on stages with low traffic (late activation steps, referral prompts, win-back emails to a small lapsed base) often cannot reach significance in useful time, and the choice is between a longer test, a bolder change that would produce a bigger effect, or a sequential or before-and-after design read with appropriate caution. Not every learning needs a controlled test: a sharp drop in a stage's conversion rate (CVR) after a release is evidence enough to roll back. For website changes on stages that a split test cannot settle in useful time, the guide to conversion work without an A/B test sets out which changes to simply ship, how to make a test possible and how to read a before-and-after change honestly.

Velocity should be reported alongside two quality measures: the share of tests called inconclusive (too low suggests results are being forced; too high suggests tests are under-powered or changes too timid), and the share of shipped wins whose effect is still visible in the cohort data a quarter later. The test pitfalls guide lists the ways a fast programme fools itself, from peeking to multiple comparisons, and every growth analyst should know it.

The learning log

The learning log is the programme's memory. It is a single, searchable record of every experiment, including the losers and the inconclusives, in a format that lets someone two years later find out whether a given idea has been tried and what happened.

Each entry carries: the hypothesis as written before the test; the loop stage and north star input it targeted; the design (variants, traffic split, duration, sample); the primary and guardrail metrics with their results; the call made and by whom; what shipped; and, most valuable of all, the interpretation: what the team now believes about customers that it did not before, written in one or two sentences. The last field is what turns a results list into learning, because a test that lost often teaches more than one that won, and a log that records only outcomes cannot say why. The experiment results report template is a fixed format for the readout each entry is drawn from, with the learning as its own field.

The log is reviewed as a body at the quarterly model review, where themes are drawn out: the stages where changes rarely move anything, the kinds of offers customers ignore, the messages that consistently lift activation. Those themes feed the next quarter's backlog and, over time, become the team's institutional knowledge.

Illustrative example — not a real result

A quarter in a subscription box company's learning log

Over one quarter the growth team completes 19 experiments. Eleven target the first-box experience, five target the pricing page and three target win-back. Of the eleven onboarding tests, four win (a simplified first-box choice lifts second-box rate from 61% to 66%; a shipping-date reminder lifts it a further two points; two copy tests add a point each), five are inconclusive after reaching their planned sample of 3,200 subscribers per variant, and two lose. All five pricing-page tests are inconclusive at 14 days and are extended; one reaches significance at 23 days with a 9% lift in trial starts, but the guardrail metric, first-box cancellation, rises from 12% to 15%, and the test is called a loss on the decision rule. The three win-back tests cannot reach sample size on a lapsed base of 1,800 and are run as a sequential design; one shows a credible lift and is shipped with a note that the evidence is weaker. The quarterly review draws one theme: everything that reduces decisions in the first week wins, and everything that adds an offer loses or is inconclusive. Next quarter's backlog is rebuilt around that theme.

Sharing results and avoiding theatre

Experiment programmes are vulnerable to theatre: the ritual of testing performed for an audience, with results chosen for their story. The signs are familiar. Only wins are presented. Lifts are quoted from the test rather than from the cohort data afterwards. The same idea is retested until it wins. Decision rules are rewritten after the data arrives.

The defences are procedural rather than moral. Results are shared in a fixed format, the log entry itself, so a presentation cannot omit the guardrails or the sample. Every result, including losses and inconclusives, goes to the same audience at the same ritual. Shipped wins are revisited a quarter later against cohort data, and the revisit is logged next to the original call. The person calling a test is not the person who built it. And the programme's own measure is learning that changed a decision, not the count of wins.

Sharing beyond the team is where theatre is most tempting, because stakeholders want stories. Give them the theme from the quarterly review, the two or three learnings that changed the model, and the honest tally of tests run, won, lost and inconclusive. A leadership team that hears only wins will eventually ask why the business does not grow at the rate the wins imply, and that question ends programmes.

To turn a testing rhythm into a lasting operating model, the experimentation programme playbook covers roles, backlog management, reporting and the governance that keeps a programme funded. When part of the programme is bought in from an agency, consultant or freelancer, the guide to hiring and evaluating a growth marketing agency sets out what stays with the business: the backlog, the learning log, the definitions and who calls each test.

Common mistakes

  • Velocity as the goal. Counting tests rewards trivial tests; count learnings that changed a decision.
  • No decision rule before the test. Without one, every result can be argued into a win.
  • The builder calls the result. Separate the person who owns the variant from the person who reads it.
  • Losers vanish. A log without losses cannot say what the team knows.
  • Skipping the quarterly revisit. Test lifts fade; cohort data a quarter later is the real result.
  • Backlog as graveyard. Hundreds of unranked ideas are not a plan; prune monthly.
  • Testing the brand. Experiments vary mechanics and offers inside a consistent brand frame, not the brand itself.
  • Reading statistics from memory. The A/B testing guide and the test pitfalls guide exist so that the cadence does not have to carry the maths.

Frequently asked questions

How many experiments should a growth team run each week?

As many as it can run properly against a ranked backlog, which for most teams is a handful rather than dozens. The limiting factor is usually traffic on the stages that matter, not build capacity, so a team with modest traffic should run fewer, bolder tests on high-leverage stages rather than many small ones. Report velocity alongside the share of inconclusive results and the share of wins that persist in cohort data.

What should go in a learning log entry?

The hypothesis as written before the test, the loop stage and north star input it targeted, the design and sample, the primary and guardrail results, the call and who made it, what shipped, and one or two sentences on what the team now believes about customers that it did not before. The last field is the one that makes the log worth keeping, because it turns outcomes into knowledge that the next quarter's backlog can use.

Who should own the experimentation cadence?

One person, usually the growth lead, owns the rituals and the log, but the backlog is shared with product, engineering and the channel teams whose stages the tests touch. The analyst who reads results should not report to the person whose variants are being read. In small companies the founder often holds the cadence; the important thing is that the rituals happen on schedule regardless of who is busy.

Terms used on this page

Where this page is used

Part of these learning paths

Also related

Related