Brief
Brackenford Opticians is an invented independent optician that sells prescription glasses online alongside its single high-street practice. Shoppers choose a frame, configure lenses, enter their prescription, and buy; a free home try-on service sends a few frames to try before buying. The owner is the practice's optometrist. An operations manager runs dispatch and returns, a marketing coordinator runs the site and email, and a freelance developer works on the site a few days a week. Nobody owns testing.
A testing tool was installed last year. Since then the team has run a handful of tests, each chosen by whoever asked loudest, two stopped early when they "looked good", and nobody can say what was learned. The owner has hired you to set up a programme for the next quarter and to leave it running when you go.
The constraints:
- The developer's time is the scarcest resource, and every test competes with routine site work.
- The owner insists that shoppers create an account so their prescription is kept on record. Changing that needs her sign-off after she checks her professional obligations, which she will do this quarter but not this month.
- The annual sale runs late in the quarter. Nothing should be tested on affected pages during it.
- The testing tool is client-side, so flicker is a live risk on heavier pages.
- The owner reads one page a month and wants to know what changed and what was learned, not how many tests ran.
The store and its data
Brackenford's monthly funnel
| Step | Shoppers | Step rate |
|---|---|---|
| Sessions | 70,000 | |
| Frame product page visitors | 38,000 | |
| Lens configurator started | 4,200 | 11.1% |
| Prescription entered | 2,500 | 59.5% |
| Added to basket | 2,300 | 92.0% |
| Checkout started | 1,500 | 65.2% |
| Orders | 1,050 | 70.0% |
Average order value: £118. Home try-on requests: 600 a month. Returns: 9% of orders, most often because the frame was too wide or too narrow. The quarter is 13 weeks; the sale takes weeks 10 and 11. No single test may run longer than six weeks.
The research pack
| ID | Finding | Evidence |
|---|---|---|
| R-a | Shoppers stall at the pupillary distance field on the prescription step; many leave there | Recordings; support emails asking how to measure it |
| R-b | Shoppers leaving the lens step say they did not know which lenses they needed | Exit poll; recordings of repeated switching between options |
| R-c | Frame width and bridge measurements sit in a closed tab; fit is the top return reason | Returns data; recordings |
| R-d | Glasses are made to order and arrive in 7 to 10 days, first stated at checkout | Walkthrough; "where is my order" emails |
| R-e | An account is required before payment | Walkthrough; funnel data |
| R-f | Reviews praise home try-on, which is linked only from the footer | Reviews |
| R-g | The owner would like a video on the homepage | None |
| R-h | A vendor offers a virtual try-on tool | The vendor's own claims |
Task
Produce a roadmap document with the parts below, in this order. Use the experimentation programme playbook as the backbone: intake, prioritisation, design and QA, run and monitor, read-out and learning log.
- Intake. Turn every finding into a hypothesis card on the experiment hypothesis template, naming the observation, the change, the expected effect, the primary metric and the guardrails. Findings with no evidence go back with a recorded reason; say what research would earn them a place.
- Feasibility. For each hypothesis, record the page or step, its eligible traffic, its baseline CVRShare of visits or clicks that convert. from the funnel (the conversion rate calculator turns counts into rates), the smallest effect worth shipping, the visitors per arm and the run length in whole weeks. Follow the reasoning in the guide to sample size and significance, and park, redesign or move any hypothesis that breaks the ceiling.
- Scoring and sequence. Score what survives on a scale you state in advance, counting evidence sources rather than rating confidence, as the guide to prioritising tests recommends. Tag each by page and step, turn shared tags into sequences, and record dependencies with the event that clears them.
- Fixes without a test. Identify findings better answered by shipping a change and watching the step rate, and explain why each does not need a test slot.
- Two full test plans. For the first two tests, write the hypothesis, unit of randomisation, allocation, primary metric, guardrails (including AOVAverage revenue per order. and the return rate where relevant), sample, duration, stopping rule, QA steps including a flicker check, and the decision you will take for each possible result. The A/B testing guide sets out the procedure.
- The calendar. Lay the tests and fixes across the quarter, respecting the ceiling, the sale, the account dependency and the developer's capacity, with slack for overruns.
- Roles and cadence. Assign the playbook's roles to Brackenford's people, including who alone declares a result, and set the weekly stand-up, fortnightly read-out and quarterly review.
- Reporting. Design the learning log fields and the one-page monthly read-out for the owner, stating results as ranges and recording losses and inconclusive tests with the same care as wins.
Draw on the level's friction work as you write the hypotheses: the guide to checkout optimisation for the account and delivery findings, and the guide to product page optimisation for frame measurements, fit and home try-on. The experimentation loop framework is the check that every test ends in a write-up that feeds the next round of research.
One row of a strong feasibility table, for comparison
| Hypothesis | Step and eligible traffic | Baseline | Smallest effect | Visitors per arm | Run length |
|---|---|---|---|---|---|
| A "how to measure" helper beside the pupillary distance field, with an option to add it later, raises the prescription step rate | Prescription step, 4,200 configurator starts a month, about 970 a week | 59.5% | 10% relative, to 65.5% | About 1,030 | 2,060 ÷ 970 = 2.1 weeks, so 3 whole weeks |
The row shows its arithmetic, rounds up to whole weeks, and ties the hypothesis to two independent sources from the research pack.
Evaluation criteria
Assess your roadmap against each row, or ask a peer to. A strong roadmap meets every row; most first attempts miss two or three.
| Criterion | What a strong answer shows | Common gap |
|---|---|---|
| Evidence at intake | Every hypothesis traces to a named finding and source; the video and the vendor's tool are returned with the research that would justify them | Unevidenced ideas kept because of who proposed them |
| Feasibility first | Run lengths computed for every hypothesis and applied before scores are read, with a decision for each that breaks the ceiling | Scores sorted first, feasibility discovered after launch |
| Consistent scoring | A stated scale, evidence counted, effort from the developer | Scores with no rationale, or everything rated high |
| Sequencing and dependencies | Tests on the same page or adjacent steps sequenced; the account change held for the owner's sign-off; nothing scheduled on sale pages during the sale | Overlapping tests on the lens and prescription steps |
| Fixes without tests | Low-risk, well-evidenced changes such as surfacing frame measurements shipped and monitored, freeing test slots | Test slots spent confirming obvious fixes |
| Test plans | Unit, metric, guardrails, sample, duration, stopping rule and a pre-agreed decision for each outcome, plus a flicker check | A plan with no sample size or no guardrails |
| Programme governance | Named roles with one person alone declaring results, and a cadence that keeps health checks apart from read-outs | Channel owners calling their own winners |
| Honest reporting | Results reported as ranges; losses and inconclusive tests logged; an "implemented?" field tracked | A monthly page listing winners only |
| Fit to the business | The calendar respects the developer's capacity and leaves slack; the owner's page answers what changed and what was learned | A roadmap that needs a full-time team |
Submission checklist
When you have finished, compare your programme with the phases in the experimentation programme playbook, then return to the intermediate CRO level to mark the level complete. Your plans should treat every A/B TestControlled comparison of two variants. as one turn of a loop that keeps running after you leave.
