Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Conversion Rate Optimisation › Intermediate

Prioritising a Test Backlog for a Lighting Retailer

An exercise in turning an untidy list of test ideas from staff, an agency and customers into a short, ordered roadmap. You merge duplicates, drop ideas with no evidence, apply the traffic constraint before any score is read, and design the first test in enough detail to build.
ExerciseIntermediate levelModule 1: Prioritising and designing tests5 minUpdated

Why this matters

The guide to prioritising tests argues that a score is a first sort, not a verdict, and that traffic and the smallest effect worth detecting decide which tests are possible at all. The experimentation loop framework shows how scores written in private drift towards whoever is most enthusiastic. This exercise applies both to a real-looking list.

The idea backlog

Lumenfield Lighting is an invented online retailer of pendant lights, lamps and bulbs. Proposers scored their own ideas for impact, confidence and ease. The developer estimated build effort, and the analyst added each page's traffic, its baseline rate and the visitors per arm needed for the proposer's predicted lift.

Illustrative example — not a real result

Lumenfield's backlog, as submitted

IDIdea (from)Evidence offeredPage, monthly visitors, baselineLift, visitors per armEffortProposer's ICE
ABulb fitting and compatible bulbs beside the button (customer service)Fitting tickets; fitting returns; fitting-code searchesProduct template, 60,000, add-to-basket 6.0%10%, 25,7003 days294
B"Which bulb fits?" filter (agency)NoneProduct template, 60,000, 6.0%10%, 25,7008 days280
COrange add-to-basket button (managing director)NoneProduct template, 60,000, 6.0%3%, 277,0000.5 days810
DDelivery cost and arrival date in the basket (exit poll)Exit poll; recordingsBasket, 9,000, to checkout 50%8%, 2,4502 days288
EFree delivery banner (agency)"Competitors do it"Site-wide, 95,000, orders 1.8%10%, 89,8002 days512
FRoom-setting photos on pendant pages (reviews)Reviews on judging size; recordingsPendant pages, 12,000, add-to-basket 5.0%10%, 31,2006 days168
GDimension diagram on pendant pages (agency)NonePendant pages, 12,000, 5.0%10%, 31,2002 days288
HGuest checkout (analyst)Funnel report; recordingsAccount step, 4,500, step rate 64%10%, 84010 days; blocked until next month's checkout upgrade144
IHomepage carousel redesign (designer)NoneHomepage, 30,000, to category 40%5%, 9,5008 days384
JLive chat on every page (sales)Sales team's impressionSite-wide, 95,000, orders 1.8%10%, 89,8005 days plus staffing336

Run length in weeks = visitors per arm × 2 ÷ monthly visitors × 4.33, rounded up to whole weeks. The ceiling for any single test is six weeks.

Scoring consistently

Rescore the survivors on one scale, written down before you start, instead of the proposers' numbers.

  • Feasible: yes or no, from run length against the ceiling, applied before anything else is read.
  • Impact: high, medium or low, from the page's reach and how close the change sits to the order.
  • Evidence: the number of independent research sources, counted rather than felt.
  • Ease: from the developer's estimate, never the proposer's.

Task

  1. Merge ideas that address the same problem on the same page, and say which version survives.
  2. Drop or return every idea with no evidence, with a one-line reason.
  3. Work out each remaining idea's run length, and remove or redesign any that break the ceiling.
  4. Rescore and order the survivors, marking ideas that share a page or adjacent steps as a sequence.
  5. Record any dependency, with the event that clears it.
  6. Design the first test: hypothesis, primary metric, guardrails, unit, allocation, sample and run length.

Worked answer

Merges. Idea B treats the same problem as idea A on the same template, so it folds into A; the cheaper build survives and the filter becomes a follow-up if the message works. Idea G joins idea F, because both answer the complaint that size is hard to judge, and together they make one bolder variant.

Drops. Ideas C, E, I and J arrive with no research behind them, and each goes back to its proposer with the question the experimentation loop asks of every hypothesis: which observation is this built on? Idea C topped the list on its proposer's score yet could never conclude: the case for a stated rationale and a second opinion.

Illustrative example — not a real result

Run lengths for the survivors

IDArithmeticWeeksVerdict
A25,700 × 2 ÷ 60,000 × 4.33 = 3.74Feasible
D2,450 × 2 ÷ 9,000 × 4.33 = 2.43Feasible
F at a 10% lift31,200 × 2 ÷ 12,000 × 4.33 = 22.523Not feasible
F and G combined, predicting 20%8,150 × 2 ÷ 12,000 × 4.33 = 5.96Feasible, at the ceiling
H840 × 2 ÷ 4,500 × 4.33 = 1.62Feasible once unblocked
C, for comparison277,000 × 2 ÷ 60,000 × 4.33 = 4040Not feasible

Scores and order. Idea A has high impact, three evidence sources and a short build. Idea D has medium impact, two sources and the cheapest build. Idea H has high impact and two sources, but is recorded as blocked until the checkout upgrade rather than scored down. Combined F and G has medium impact, two sources and no slack against the ceiling.

Ideas A and F share the product template and idea D sits on the next step, so the three form a sequence. Idea A leads it: more evidence, the whole template's reach and slack against the ceiling, whereas F and G's bolder prediction was partly made to fit the ceiling. The roadmap is A, then D, then H when the upgrade lands, with F and G in reserve.

The first test. If product pages show the fitting type and compatible bulbs beside the button, add-to-basket will rise, because tickets, returns and searches all show shoppers unsure the light fits. It is an A/B Test with the visitor as the unit and an even split. The primary metric is add-to-basket rate, a CVR counted per visitor; the guardrails are order conversion and returns coded "wrong fitting". It needs the visitors per arm in the table above, over four whole weeks, read once. The A/B testing guide covers pre-launch QA.

Common mistakes

  • Sorting by the proposers' scores. Self-scored ideas favour their authors.
  • Scoring before the traffic check. An idea that needs most of a year is not a candidate.
  • Keeping duplicates as separate tests. Two fixes for one problem on one template collide.
  • Treating "blocked" as "low priority". Record the dependency so the idea returns when it clears.

Next, take the test prioritisation and design checkpoint or return to the intermediate CRO level.

Pages this lesson applies

Terms used on this page

Also related

Related