Why this matters
The guide to prioritising tests argues that a score is a first sort, not a verdict, and that traffic and the smallest effect worth detecting decide which tests are possible at all. The experimentation loop framework shows how scores written in private drift towards whoever is most enthusiastic. This exercise applies both to a real-looking list.
The idea backlog
Lumenfield Lighting is an invented online retailer of pendant lights, lamps and bulbs. Proposers scored their own ideas for impact, confidence and ease. The developer estimated build effort, and the analyst added each page's traffic, its baseline rate and the visitors per arm needed for the proposer's predicted lift.
Lumenfield's backlog, as submitted
| ID | Idea (from) | Evidence offered | Page, monthly visitors, baseline | Lift, visitors per arm | Effort | Proposer's ICE |
|---|---|---|---|---|---|---|
| A | Bulb fitting and compatible bulbs beside the button (customer service) | Fitting tickets; fitting returns; fitting-code searches | Product template, 60,000, add-to-basket 6.0% | 10%, 25,700 | 3 days | 294 |
| B | "Which bulb fits?" filter (agency) | None | Product template, 60,000, 6.0% | 10%, 25,700 | 8 days | 280 |
| C | Orange add-to-basket button (managing director) | None | Product template, 60,000, 6.0% | 3%, 277,000 | 0.5 days | 810 |
| D | Delivery cost and arrival date in the basket (exit poll) | Exit poll; recordings | Basket, 9,000, to checkout 50% | 8%, 2,450 | 2 days | 288 |
| E | Free delivery banner (agency) | "Competitors do it" | Site-wide, 95,000, orders 1.8% | 10%, 89,800 | 2 days | 512 |
| F | Room-setting photos on pendant pages (reviews) | Reviews on judging size; recordings | Pendant pages, 12,000, add-to-basket 5.0% | 10%, 31,200 | 6 days | 168 |
| G | Dimension diagram on pendant pages (agency) | None | Pendant pages, 12,000, 5.0% | 10%, 31,200 | 2 days | 288 |
| H | Guest checkout (analyst) | Funnel report; recordings | Account step, 4,500, step rate 64% | 10%, 840 | 10 days; blocked until next month's checkout upgrade | 144 |
| I | Homepage carousel redesign (designer) | None | Homepage, 30,000, to category 40% | 5%, 9,500 | 8 days | 384 |
| J | Live chat on every page (sales) | Sales team's impression | Site-wide, 95,000, orders 1.8% | 10%, 89,800 | 5 days plus staffing | 336 |
Run length in weeks = visitors per arm × 2 ÷ monthly visitors × 4.33, rounded up to whole weeks. The ceiling for any single test is six weeks.
Scoring consistently
Rescore the survivors on one scale, written down before you start, instead of the proposers' numbers.
- Feasible: yes or no, from run length against the ceiling, applied before anything else is read.
- Impact: high, medium or low, from the page's reach and how close the change sits to the order.
- Evidence: the number of independent research sources, counted rather than felt.
- Ease: from the developer's estimate, never the proposer's.
Task
- Merge ideas that address the same problem on the same page, and say which version survives.
- Drop or return every idea with no evidence, with a one-line reason.
- Work out each remaining idea's run length, and remove or redesign any that break the ceiling.
- Rescore and order the survivors, marking ideas that share a page or adjacent steps as a sequence.
- Record any dependency, with the event that clears it.
- Design the first test: hypothesis, primary metric, guardrails, unit, allocation, sample and run length.
Worked answer
Merges. Idea B treats the same problem as idea A on the same template, so it folds into A; the cheaper build survives and the filter becomes a follow-up if the message works. Idea G joins idea F, because both answer the complaint that size is hard to judge, and together they make one bolder variant.
Drops. Ideas C, E, I and J arrive with no research behind them, and each goes back to its proposer with the question the experimentation loop asks of every hypothesis: which observation is this built on? Idea C topped the list on its proposer's score yet could never conclude: the case for a stated rationale and a second opinion.
Run lengths for the survivors
| ID | Arithmetic | Weeks | Verdict |
|---|---|---|---|
| A | 25,700 × 2 ÷ 60,000 × 4.33 = 3.7 | 4 | Feasible |
| D | 2,450 × 2 ÷ 9,000 × 4.33 = 2.4 | 3 | Feasible |
| F at a 10% lift | 31,200 × 2 ÷ 12,000 × 4.33 = 22.5 | 23 | Not feasible |
| F and G combined, predicting 20% | 8,150 × 2 ÷ 12,000 × 4.33 = 5.9 | 6 | Feasible, at the ceiling |
| H | 840 × 2 ÷ 4,500 × 4.33 = 1.6 | 2 | Feasible once unblocked |
| C, for comparison | 277,000 × 2 ÷ 60,000 × 4.33 = 40 | 40 | Not feasible |
Scores and order. Idea A has high impact, three evidence sources and a short build. Idea D has medium impact, two sources and the cheapest build. Idea H has high impact and two sources, but is recorded as blocked until the checkout upgrade rather than scored down. Combined F and G has medium impact, two sources and no slack against the ceiling.
Ideas A and F share the product template and idea D sits on the next step, so the three form a sequence. Idea A leads it: more evidence, the whole template's reach and slack against the ceiling, whereas F and G's bolder prediction was partly made to fit the ceiling. The roadmap is A, then D, then H when the upgrade lands, with F and G in reserve.
The first test. If product pages show the fitting type and compatible bulbs beside the button, add-to-basket will rise, because tickets, returns and searches all show shoppers unsure the light fits. It is an A/B TestControlled comparison of two variants. with the visitor as the unit and an even split. The primary metric is add-to-basket rate, a CVRShare of visits or clicks that convert. counted per visitor; the guardrails are order conversion and returns coded "wrong fitting". It needs the visitors per arm in the table above, over four whole weeks, read once. The A/B testing guide covers pre-launch QA.
Common mistakes
- Sorting by the proposers' scores. Self-scored ideas favour their authors.
- Scoring before the traffic check. An idea that needs most of a year is not a candidate.
- Keeping duplicates as separate tests. Two fixes for one problem on one template collide.
- Treating "blocked" as "low priority". Record the dependency so the idea returns when it clears.
Next, take the test prioritisation and design checkpoint or return to the intermediate CRO level.
