Why backlogs need a method
Without a method, a backlog is ordered by whoever spoke last. The chief executive's idea runs first, the designer's redesign second, and the hypothesis with the strongest evidence waits because nobody with authority proposed it. The results are predictable: tests on pages that matter to stakeholders rather than to visitors, variants that bundle several people's ideas so that everyone is represented, and a programme judged on how many tests it ran rather than on what it learned.
A method does three things. It makes the reasoning visible, so that an idea's place in the queue can be challenged on its evidence rather than its sponsor. It forces the estimates that decide feasibility, above all how many visitors a test needs and how long that will take, to be made before the test is built rather than discovered when it has run for two months without a verdict. And it exposes dependencies: tests that cannot run at the same time, tests that must wait for a release, and tests whose answer changes the value of other tests.
The method does not have to be elaborate. A consistent set of fields, filled honestly, reviewed monthly, is enough for most teams, and the backlog those fields live in is described in the conversion research hub. What matters is that every hypothesis is scored the same way, that the score is a conversation starter rather than a verdict, and that the traffic constraint is applied before any score is read. A key performance indicator (KPIA metric tied directly to an objective.) for the backlog itself is the share of tests that reached a conclusion in the time planned; a backlog full of tests that ran out of traffic is a prioritisation failure, not a statistics one.
Scoring models and their limits
Most teams use one of a few scoring models, all of which rate each hypothesis on a handful of dimensions and combine the ratings into a single number for sorting.
| Model | Dimensions | Strength | Limit |
|---|---|---|---|
| Impact, confidence, ease | How much the metric could move, how sure the team is, how cheap it is to build | Fast; everyone can use it | All three are guesses, and confidence is usually the author's enthusiasm |
| Potential, importance, ease | How much room the page has to improve, how much traffic and value it carries, how cheap the change is | Importance weights the page by what it is worth | Potential is subjective; ease dominates when estimates are rough |
| Evidence-weighted checklist | A set of yes-or-no questions: above the fold, noticeable within seconds, adds or removes an element, backed by analytics, recordings, polls, usability sessions | Replaces opinion with checks; rewards evidence | Binary questions flatten big differences; the list needs tailoring to the site |
| Expected value | Probability of winning multiplied by the value of a win, set against cost | The most honest framing of what a test is worth | Needs calibrated probabilities most teams do not have |
Each model is useful as a way of making people state their assumptions, and each fails when the number it produces is treated as the truth. The common failures are the same across models. Confidence and impact are rated by the person who wrote the hypothesis, who is not neutral. Ease is rated by the person who will not build it. A small, cheap, confident change scores above a large, costly, evidence-backed one, so the backlog fills with button tests. And none of the models, as usually applied, asks whether the test can run at all: whether the page has enough traffic to detect the effect the hypothesis predicts in a time anyone will wait. Outside test backlogs, the comparison of the impact-effort matrix and the weighted scoring matrix explains when a quick two-by-two sort is enough and when a few options deserve criteria weighted before scoring. The comparison of expected value and the weighted scoring matrix sets out when a probability-weighted value in money is the better way to choose between options.
Use a model, then, as a first sort and a shared vocabulary, with two amendments. Rate evidence explicitly, by counting the research methods that support the hypothesis, rather than letting it hide inside confidence. And apply the traffic constraint as a filter before the scores are compared, which is the subject of the next section.
Traffic and minimum detectable effect as the hard constraint
Every test needs a certain number of visitors in each arm before it can tell a real effect from noise. That number depends on the baseline conversion rate (CVRShare of visits or clicks that convert.) of the page or step, on the smallest effect the team would act on, called the minimum detectable effect, and on the error rates the team is willing to accept. The relationship runs the wrong way for impatient teams: the smaller the effect you want to detect, the many times more visitors you need, and halving the detectable effect roughly quadruples the sample. The sample size and significance guide explains the statistics in plain language, and the conversion rate calculator turns raw counts into the baseline rate the estimate starts from; this guide uses the result.
For prioritisation, the practical procedure is to attach to each hypothesis the smallest effect that would be worth acting on, estimate the visitors per arm that effect requires at the page's baseline, and divide by the page's eligible traffic to get a run length. Then set a ceiling. Tests that would run longer than the ceiling, commonly a few weeks to a couple of months, are not feasible on that page in that form, however well they score, and they are parked, redesigned to predict a bigger effect, moved to a higher-traffic page, or replaced by a fix shipped without a test where the evidence is strong enough.
This single step changes most backlogs more than any scoring model. It removes small changes on low-traffic pages, which can never reach a verdict. It favours bolder changes on busy pages, which have both the reach and the plausible effect size to conclude quickly. It also reveals when a site should not be running tests at all and should rely on research, usability testing and judgement instead, which is a legitimate programme and a better one than a queue of tests that never finish. The test pitfalls page describes what happens when teams ignore this and stop tests early instead: results that look significant and are not.
Effort, risk and dependencies
Once the feasible hypotheses are known, three further factors shape the order. Effort is the cost to design, build, check and analyse the test, estimated by the people who will do the work rather than the person proposing it, and expressed in days rather than a vague scale. A test that needs a new checkout component is not comparable to one that changes a headline, and a scoring model that rates both as medium ease hides that.
Risk has three faces. Technical risk is that the variant breaks something: a form that fails on one browser, a script that slows the page, a test tool that causes flicker. Business risk is that the change, even if it wins on the primary metric, harms something the test does not measure: a discount that raises conversion and lowers margin, a shorter form that raises leads and lowers their quality. Measurement risk is that the metric's event is not reliable on the page, so the test could not be read even if it ran. Each is reduced by work before launch, and the work belongs in the effort estimate.
Dependencies are the constraints that come from outside the test. A hypothesis about a checkout step cannot be tested while the checkout is being rebuilt. A pricing test needs sign-off from finance. A seasonal page is only worth testing in season. A test on a step upstream changes the population that reaches a step downstream, which affects the baseline of any test there. Record each dependency against the hypothesis, with the date or event that clears it, so that the backlog shows what is blocked and why, and so that a hypothesis is not scored down for being impossible this month when it is merely waiting.
Sequencing interacting tests
Two tests interact when the result of one changes the meaning of the other. The clearest case is two tests on the same page or step at the same time: if the headline test and the form test run together with independent assignment, some visitors see both variants, some see neither, and the effect each test reports is an average over a mix the hypothesis never described. The subtler case is two tests on different steps of the same funnel, where the upstream test changes who reaches the downstream one and therefore its baseline and its effect.
There are three ways to handle this, in rising order of cost. Run interacting tests in sequence, so that each result is read on a stable page; this is the default for small programmes. Run them as a single multivariateAn experiment that tests several elements at once by measuring every combination of their versions. test that includes every combination, which answers the interaction question directly but needs far more traffic, since each combination is in effect its own arm. Or use mutually exclusive groups, where visitors are allocated to one test only, which keeps results clean at the cost of splitting the traffic each test can use, and therefore lengthening every run.
The practical rule for a backlog is to tag each hypothesis with the page, the funnel step and the element it touches, and to treat two hypotheses with the same tag as a sequence rather than a pair. Order the sequence by the size of the predicted effect, largest first, because the bigger change settles the baseline the smaller one will be read against. Where a test is site-wide, such as a navigation or template change, it should either run alone or be acknowledged as a background condition in every result read during its run. The hypothesis tutorial asks for scope at the moment of writing precisely so that these tags exist before prioritisation starts.
The quarterly roadmap
Prioritisation ends in a roadmap: the tests the programme expects to run in the coming quarter, in order, with the slot each occupies on the site. A quarter is long enough to run a handful of tests to their planned sample on a mid-traffic site and short enough that the research behind the hypotheses is still current. The roadmap is a plan, not a contract; a test that concludes early frees its slot, and a loss that overturns a diagnosis may remove the tests queued behind it.
Build it in four passes. First, filter the backlog to feasible hypotheses using the traffic constraint, and park the rest with their reason. Second, score what remains with the chosen model, with evidence counted explicitly and effort estimated by the builders. Third, group by page and step, and turn each group into a sequence. Fourth, lay the sequences against the calendar, respecting dependencies and the ceiling on run length, and leave slack for the tests that overrun or the research that produces something urgent. Reserve some capacity for fixes without tests, which usually produce the quarter's most reliable gains. Then review monthly: results written back, scores revised, parked hypotheses rechecked as traffic or priorities change, and the A/B test (A/B TestControlled comparison of two variants.) slots for the coming month confirmed with the people who will build them.
Sorting a backlog of eight hypotheses
An online furniture retailer has eight hypotheses after a research sprint and a mid-traffic site: roughly 180,000 sessions a month, with 22,000 reaching checkout. The traffic pass removes three at once. A hypothesis predicting a 5% relative lift on the delivery-options page, at a 48% step rate, needs about 24,000 visitors per arm and would take over four months on that page's 11,000 monthly visitors; it is parked. A trust-badge test on a low-traffic category page and a copy change on the returns page fail the same filter. Of the five that remain, the two strongest on evidence both touch the product page: a stock-level message supported by analytics, recordings and poll responses, predicting a 12% lift in add-to-basket, and an image-gallery change supported by recordings only. They are sequenced, stock message first. A checkout guest-option test, backed by four research methods and predicting a 15% lift in checkout completion from a 31% baseline, needs about 4,100 visitors per arm, under two weeks, and takes the first slot. The quarter's roadmap holds the checkout test, then the two product-page tests in sequence, then a search-results test, with the fifth hypothesis in reserve and two bug fixes shipped in week one without a test.
Prioritisation works best inside a wider process, and the experimentation programme playbook shows how the backlog, test reviews and reporting fit together as a repeatable programme.
When a test from the backlog runs for weeks without a result, the page on an A/B test that is not reaching significance shows whether to re-plan it, move it to a busier page or record a genuine null.
Common mistakes
- Scoring before filtering. A high-scoring test the page cannot support wastes a slot for months. Apply the traffic constraint first.
- Ease rated by the proposer. Effort estimates belong to the people who will build and check the test.
- Confidence as a proxy for evidence. Count the research methods behind the hypothesis instead of asking how sure the author feels.
- Running everything at once. Overlapping tests on one page or funnel produce results nobody can read. Sequence them.
- A roadmap with no slack. Tests overrun, research surprises, releases slip. Leave room, or the plan is fiction by month two.
- Treating the model's number as a decision. The score starts the conversation; the backlog review, with the evidence on the table, ends it.
- Judging the programme by tests run. The measure is what was learned and what moved; the experimentation hub covers how to read and report results so that losses count as learning too.
