What this checkpoint covers
The first module of the intermediate CRO level: scoring and ordering ideas as the guide to prioritising tests sets out, the cycle described by the experimentation loop framework, and the procedure for an A/B TestControlled comparison of two variants. in the A/B testing guide: unit, metric, guardrails, allocation and QA. It follows the backlog prioritisation exercise.
Task
- Answer every question in a sentence or two, with your reason, before reading the explanations.
- Mark each answer right, partly right or wrong, and note the idea you missed.
- Take one test idea from your own work, or from a backlog you know, and rewrite it so that it names the evidence, the page's traffic, the primary metric, the guardrails and the unit of randomisation. If you cannot fill one of those, that gap is the next thing to research.
Questions
- The founder's idea scores highest in the backlog on impact, confidence and ease. The page it touches would need many months to reach the planned sample. Where should it go in the roadmap?
- Two ideas both target the product page: a new image gallery and a delivery message near the button. The team wants to run them at the same time to save a month. What do you advise?
- A colleague rates an idea's confidence highly because "everyone on the team agrees". How would you rate it instead?
- An online furniture shop tests a new checkout. Most buyers visit several times before ordering. The tool defaults to assigning by session. Is that acceptable?
- A test of a bolder pricing message is planned with click-through to the basket as the primary metric, because it moves faster than orders. What is the risk, and what would you change?
- A discount prompt is being tested on the basket page. The plan names the order rate as the primary metric and nothing else. What is missing?
- The developer can launch a test today, but the tracking walk in each arm has not been done. The team argues it can be checked once results come in. What do you say?
- A high-evidence fix for a broken postcode field is in the backlog as a test. Several research methods agree on the problem and the change carries little risk. Should it be tested?
Answers explained
- Out of the roadmap: park it, redesign it or move it. A score says nothing about feasibility, and the guide to prioritising tests applies the traffic constraint before any score is read. The options are a bolder version predicting a larger effect, the same change on a busier page, or a decision on judgement, recorded with its reason.
- Sequence them, or run one test with combined arms. Two tests on the same page interact: visitors who see both variants, or neither, blur what each test reports. Run the change with the larger predicted effect first, so the smaller one is read against a settled page, as the prioritisation guide recommends.
- By counting the research sources behind it. Agreement is not evidence. The experimentation loop framework warns that confidence inflates when it is rated by feel; analytics, recordings, polls and support contacts that point at the same problem are what raise it.
- No. Assign by visitor. With session-level assignment one person can see both checkouts across their visits, so the A/B TestControlled comparison of two variants. measures neither. The A/B testing guide matches the unit to the level at which the decision and the metric happen.
- The variant can win on clicks and lose on orders. Choose the metric closest to the business outcome the traffic can detect, and if clicks must be the primary metric, make orders a guardrail. A click-level win with flat orders is a finding about attention, not a reason to ship.
- Guardrails, above all revenue per visitor or order value. A discount can raise the order rate while lowering what each order is worth. The guardrails belong in the plan before launch, so a breach becomes a business decision rather than a surprise after rollout.
- Do not launch until the walk is done. A tag that fires twice in one arm, or not at all on a redesigned page, manufactures a result that no analysis can repair. QA of the experience, the tracking and the written plan all pass before the test counts, as the A/B testing guide sets out.
- Usually not; ship it and measure it. The prioritisation guide reserves capacity for fixes without tests, where evidence from several methods is strong and the risk is low. Watch the step's rate afterwards to confirm the fix landed.
Where to go next
If you missed the ordering questions, rework the backlog prioritisation exercise with a different ceiling on run length. If you missed the design questions, write a full plan for your rewritten idea on the experiment hypothesis template. Then move on to the second module, which starts with the statistics behind sample size and significance.
