Why cadence beats heroics
Most teams' experiment history is a series of heroics: a big test when someone had a strong opinion, a redesign when a new leader arrived, a burst of activity before a board meeting. Between the bursts, nothing. The results of the bursts are remembered selectively, the losers quietly forgotten, and a year later nobody can say what the team knows that it did not know before.
A cadence replaces bursts with a rhythm. Every week some experiments start, some finish, and some are read out and recorded. The rhythm has three effects that heroics cannot produce. It makes learning cumulative, because every result goes into the same log against the same hypotheses. It makes prioritisation honest, because the backlog is compared as a whole rather than one idea against the enthusiasm of the moment. And it protects the programme from the loudest voice, because a strong opinion becomes a hypothesis in the queue rather than a decision.
The cadence also fixes the relationship between experiments and the growth model. The growth loops guide explains how a loop is mapped and where its leaks are; the cadence is how the team works on those leaks in order, so that each test claims a path to the north star and the KPIA metric tied directly to an objective. tree beneath it. Without that link, a team can run a hundred tests and improve a hundred local measures while the business stands still. The growth fundamentals hub describes the model the cadence serves.
The backlog and prioritisation
The backlog is a single list of candidate experiments, each written to a template that forces the author to think. A usable entry has: the leak or opportunity it addresses and the evidence for it; a hypothesis in the form "if we change this, then that measure will move, because"; the primary metric and the decision ruleA rule written before a test starts that says which result ships the change, which leads to another round and which abandons it. (what result ships it, what result kills it); the smallest version that would be informative; who is needed to build it; and an estimate of effort.
Prioritisation is where the backlog earns its keep. The usual scoring approaches weigh expected impact, confidence in the hypothesis and ease of execution, and the prioritising tests guide in the CRO section compares the common schemes and their failure modes. Two rules matter more than the scheme chosen. First, impact is scored against the north star or the loop stage, not against the metric the test happens to move, so that a test on a minor page cannot outrank a test on activation by promising a large local lift. Second, confidence comes from evidence (research, data, prior tests), not from seniority, and an entry with no evidence is scored low however senior its author.
The backlog is reviewed weekly and pruned monthly. Ideas that have sat unprioritised for a quarter are either rewritten with evidence or removed; a backlog of hundreds of unranked ideas is a graveyard, not a plan. A healthy backlog has a short ranked top, a middle of entries waiting for evidence or capacity, and a tail that is regularly cleared. The strongest entries often come from above the backlog: a learning agenda built from the plan's assumptions names the questions the year's plan cannot afford to get wrong and the dates by which each needs an answer.
Rituals
Rituals are the meetings and documents that happen on a schedule regardless of mood. The set below is the common shape; small teams merge several into one weekly hour, and the tutorial on running a weekly growth meeting sets out the agenda, pre-read and decisions for that merged hour.
| Ritual | Cadence | Who | Output |
|---|---|---|---|
| Results read-out | Weekly | Growth lead, analyst, test owners | Each finished test called as win, loss or inconclusive against its decision rule; log entry written |
| Prioritisation | Weekly | Growth lead, product and engineering partners | Next tests pulled from the ranked backlog; capacity confirmed |
| Backlog grooming | Monthly | Growth lead, analyst | Entries rewritten with new evidence; stale entries removed; scoring recalibrated |
| Sprint or planning sync | Per sprint | Growth team and the product team whose roadmap it shares | Experiments that need builds scheduled; conflicts with product releases resolved |
| Cohort and economics review | Monthly | Growth lead, analyst, finance partner | North star, loop stage rates and unit economics by cohort; experiments reassessed against them |
| Model review | Quarterly | Whole growth team and stakeholders | Growth model updated; next quarter's leak targets agreed; learning log themes summarised |
The read-out is the ritual most often skipped and the one that matters most. Its discipline is that every test is called against the decision rule written before it started, by someone other than the person who built it, and that the call is recorded the same day. An inconclusive result is recorded as inconclusive, not reinterpreted as a narrow win.
Velocity vs quality
Test velocity, the number of experiments completed per period, is a tempting measure of a programme because it is easy to count and easy to raise. It is also the measure most likely to corrupt the programme. Pushed too hard, velocity produces small tests on trivial pages, tests stopped the moment a difference appears, and a log full of "wins" that never show up in the business.
Quality has its own failure mode. A programme that insists on perfect tests runs few of them, waits months for sample sizes on low-traffic pages, and learns slowly. The balance depends on traffic and on where in the loop the test sits.
The practical rules are these. Tests on stages with high traffic (landing pages, sign-up forms, checkout) can run as proper A/B TestControlled comparison of two variants. designs with pre-registered sample sizes, and the A/B testing guide sets out how. Tests on stages with low traffic (late activation steps, referral prompts, win-back emails to a small lapsed base) often cannot reach significance in useful time, and the choice is between a longer test, a bolder change that would produce a bigger effect, or a sequential or before-and-after design read with appropriate caution. Not every learning needs a controlled test: a sharp drop in a stage's conversion rate (CVRShare of visits or clicks that convert.) after a release is evidence enough to roll back. For website changes on stages that a split test cannot settle in useful time, the guide to conversion work without an A/B test sets out which changes to simply ship, how to make a test possible and how to read a before-and-after change honestly.
Velocity should be reported alongside two quality measures: the share of tests called inconclusive (too low suggests results are being forced; too high suggests tests are under-powered or changes too timid), and the share of shipped wins whose effect is still visible in the cohort data a quarter later. The test pitfalls guide lists the ways a fast programme fools itself, from peeking to multiple comparisons, and every growth analyst should know it.
The learning log
The learning logA single, searchable record of every experiment, losers and inconclusive results included, with the hypothesis, design, result, decision and what it taught. is the programme's memory. It is a single, searchable record of every experiment, including the losers and the inconclusives, in a format that lets someone two years later find out whether a given idea has been tried and what happened.
Each entry carries: the hypothesis as written before the test; the loop stage and north star input it targeted; the design (variants, traffic split, duration, sample); the primary and guardrail metricsA metric an experiment must not worsen beyond an agreed bound, watched beside the primary metric that decides the test. with their results; the call made and by whom; what shipped; and, most valuable of all, the interpretation: what the team now believes about customers that it did not before, written in one or two sentences. The last field is what turns a results list into learning, because a test that lost often teaches more than one that won, and a log that records only outcomes cannot say why. The experiment results report template is a fixed format for the readout each entry is drawn from, with the learning as its own field.
The log is reviewed as a body at the quarterly model review, where themes are drawn out: the stages where changes rarely move anything, the kinds of offers customers ignore, the messages that consistently lift activation. Those themes feed the next quarter's backlog and, over time, become the team's institutional knowledge.
A quarter in a subscription box company's learning log
Over one quarter the growth team completes 19 experiments. Eleven target the first-box experience, five target the pricing page and three target win-back. Of the eleven onboarding tests, four win (a simplified first-box choice lifts second-box rate from 61% to 66%; a shipping-date reminder lifts it a further two points; two copy tests add a point each), five are inconclusive after reaching their planned sample of 3,200 subscribers per variant, and two lose. All five pricing-page tests are inconclusive at 14 days and are extended; one reaches significance at 23 days with a 9% lift in trial starts, but the guardrail metric, first-box cancellation, rises from 12% to 15%, and the test is called a loss on the decision rule. The three win-back tests cannot reach sample size on a lapsed base of 1,800 and are run as a sequential design; one shows a credible lift and is shipped with a note that the evidence is weaker. The quarterly review draws one theme: everything that reduces decisions in the first week wins, and everything that adds an offer loses or is inconclusive. Next quarter's backlog is rebuilt around that theme.
Sharing results and avoiding theatre
Experiment programmes are vulnerable to theatre: the ritual of testing performed for an audience, with results chosen for their story. The signs are familiar. Only wins are presented. Lifts are quoted from the test rather than from the cohort data afterwards. The same idea is retested until it wins. Decision rules are rewritten after the data arrives.
The defences are procedural rather than moral. Results are shared in a fixed format, the log entry itself, so a presentation cannot omit the guardrails or the sample. Every result, including losses and inconclusives, goes to the same audience at the same ritual. Shipped wins are revisited a quarter later against cohort data, and the revisit is logged next to the original call. The person calling a test is not the person who built it. And the programme's own measure is learning that changed a decision, not the count of wins.
Sharing beyond the team is where theatre is most tempting, because stakeholders want stories. Give them the theme from the quarterly review, the two or three learnings that changed the model, and the honest tally of tests run, won, lost and inconclusive. A leadership team that hears only wins will eventually ask why the business does not grow at the rate the wins imply, and that question ends programmes.
To turn a testing rhythm into a lasting operating model, the experimentation programme playbook covers roles, backlog management, reporting and the governance that keeps a programme funded. When part of the programme is bought in from an agency, consultant or freelancer, the guide to hiring and evaluating a growth marketing agency sets out what stays with the business: the backlog, the learning log, the definitions and who calls each test.
Common mistakes
- Velocity as the goal. Counting tests rewards trivial tests; count learnings that changed a decision.
- No decision rule before the test. Without one, every result can be argued into a win.
- The builder calls the result. Separate the person who owns the variant from the person who reads it.
- Losers vanish. A log without losses cannot say what the team knows.
- Skipping the quarterly revisit. Test lifts fade; cohort data a quarter later is the real result.
- Backlog as graveyard. Hundreds of unranked ideas are not a plan; prune monthly.
- Testing the brand. Experiments vary mechanics and offers inside a consistent brand frame, not the brand itself.
- Reading statistics from memory. The A/B testing guide and the test pitfalls guide exist so that the cadence does not have to carry the maths.
