Illustrative scenario
The retailer, its checkout, the people and every figure on this page are invented so that a testing programme can be shown from research to decision. Nothing here describes a real client, a real retailer or the author's own results, and the test results in the table are a worked exercise in reading statistics, not evidence of what any change to a checkout will do. No number on this page is a benchmark.
About this scenario
This scenario is invented. The retailer, which the page calls Hollowmere, does not exist; its checkout, its traffic, its team and the results of its tests were made up so that the full cycle — research, hypothesis, test, decision — could be shown with figures that are internally consistent and correctly interpreted. Nothing here describes the author's own work or any real company.
In the scenario Hollowmere sells homeware online: lighting, textiles, tableware. Its checkout is built in-house rather than taken from a platform, which means the team can change any step. The team is a product manager, a designer, two engineers and an analyst, and it has a testing set-up that assigns visitors to arms server-side at the start of checkout and records which arm each order came from. Before the programme, changes to the checkout were shipped on opinion, and the analyst had no way to say whether any of them had helped. The product manager's proposal is modest: find the biggest leaks with research, write a small number of hypotheses, test them one at a time to a sample fixed in advance, and record every result, including the ones that embarrass somebody. The programme runs for a quarter. The point of the page is how the three tests were planned, read and acted on, and especially how the team avoided the two mistakes that undo most testing programmes: stopping early when a result looks good, and treating a non-significant result as proof of no effect.
The starting point and research
The research followed the methods in the conversion research guide in a deliberately ordered way: quantitative first, to find where people left; qualitative second, to find out why. The analyst built a step-by-step view of the checkout FunnelModel of the customer journey stages. using the approach in the funnel analysis guide, counting people who reached each step rather than page views, and segmenting by device and by new versus returning customers. Then the team watched session recordings from the steps with the biggest drop, read a quarter's worth of support tickets tagged "checkout", ran a short exit survey on the checkout pages, and sat five customers down to buy something while thinking aloud. The same order of work, run as a bounded study of one funnel that ends in backlog rows, is laid out step by step in the tutorial on running a conversion research sprint.
Four findings came out of it. The first step, which demanded an account before anything else, lost the largest share of people, and new visitors on mobile lost most. The delivery cost appeared only at the shipping step, and the exit survey's most common free-text answer at that step was a complaint about the cost arriving late. The address form caused visible struggle in recordings, particularly with flat numbers and postcode lookup. And a number of people paused at payment, scrolled, and left; the usability sessions suggested unfamiliarity with the payment set-up rather than price. The checkout optimisation guide describes each of these friction points in general; the research told the team which ones applied here and roughly in what order of size.
Hollowmere: where the checkout leaked, invented figures
In a representative fortnight, 10,000 visitors started checkout. 7,900 got past the account step, 6,400 completed the address form, 5,200 reached payment and 3,800 placed an order, a checkout conversion rate of 38.0%. The account step lost 21% of starters, the address form 19% of those who reached it, the delivery step 19% of those who reached it, and payment 27% of those who reached it. On mobile, the account step alone lost 26% of starters against 14% on desktop. Checkout starts ran at about 7,500 a week, which set the pace at which any test could reach its sample.
The hypotheses
The team wrote hypotheses in the format from the writing hypotheses tutorial: the evidence, the problem it implies, the change, the expected effect and the metric that would show it. Three survived prioritisation under the method in the prioritising tests guide, chosen by the size of the leak they addressed and the confidence the research gave, and ordered so that the one addressing the largest leak ran first.
The first hypothesis: because the account step loses the largest share of starters, especially on mobile, and recordings show people abandoning at the password field, offering guest checkoutLetting a shopper complete an order without creating an account, with the account offered afterwards when the details are already captured. as the default and inviting account creation after the order will raise checkout conversion. The second: because the delivery cost is first shown at the shipping step and the exit survey names it as the reason for leaving, showing a delivery estimate on the first checkout step will raise conversion by removing a late surprise, at some risk of losing people earlier instead. The third: because people pause and leave at payment and usability sessions suggest uncertainty rather than price, placing an express wallet option first and adding a short reassurance line about payment security will raise conversion at the payment step.
Each hypothesis named the same primary metric, checkout conversion rate (CVRShare of visits or clicks that convert.), defined as orders divided by checkout starts, and the same guardrails: average order value (AOVAverage revenue per order.), payment error rate and support contacts per order. Each stated a minimum effect worth detecting, from which the analyst computed the sample size per arm and the run length in whole weeks, using the method in the sample size and significance guide. The run length, the sample and the decision rule were written down before each test started, and the rule was that results would be read once, at the end. The reasoning behind that rule is set out in the article in the sources on how not to run a test: looking repeatedly and stopping when the result crosses a line inflates the false-positive rate well beyond the level the test claims.
The tests
The tests ran one at a time, because each A/B TestControlled comparison of two variants. touched the same flow and the same metric, and each ran for three full weeks so that weekday and weekend behaviour were covered in every arm. Assignment was by visitor at the first checkout step, and the analyst checked, before reading any result, that the arms had been assigned in the planned proportions and that the two arms looked alike on device mix and new-versus-returning share. The A/B testing guide describes those pre-launch and in-flight checks; the scenario simply followed them. The results, all invented, are in the example.
Hollowmere: three tests, invented results, read correctly
Planning: baseline checkout conversion 38.0%, minimum detectable effect 5% relative (about 1.9 percentage points), two-sided test at a 0.05 significance level with 0.8 power, which gives about 10,300 checkout starts per arm. After the first test shipped, the baseline moved to about 40.5% and the same relative effect needed about 9,300 per arm. Each test ran three full weeks regardless and was read once, at the end. Differences are in percentage points (pp) with a 95% confidence interval.
| Test | Change | Starts per arm | Control CVR | Variant CVR | Difference (95% CI) | p-value | Decision |
|---|---|---|---|---|---|---|---|
| A | Guest checkout by default; account offered after the order | 11,200 | 38.1% (4,267 orders) | 40.6% (4,547 orders) | +2.5 pp (+1.2 to +3.8) | below 0.001 | Shipped |
| B | Delivery cost estimate on the first checkout step | 10,900 | 40.5% (4,414 orders) | 41.1% (4,480 orders) | +0.6 pp (−0.7 to +1.9) | 0.36 | Not shipped; planned-size effect ruled out |
| C | Express wallet first, plus a payment reassurance line | 10,100 | 40.6% (4,101 orders) | 42.3% (4,272 orders) | +1.7 pp (+0.3 to +3.1) | 0.015 | Shipped, then partly reverted |
Guardrails: average order value was £63.80 against £64.20 in test A, £64.10 against £63.90 in test B, and £64.30 against £62.60 in test C; none of the three differences was statistically distinguishable from zero, though the team flagged the test C figure for follow-up. Payment error rates did not differ in any test. In test C, a reading taken on day 5 showed the variant ahead by 9% relative with a nominal p-value of 0.03; the pre-registered decision rule meant nobody acted on it, and the lift at the planned end was 4.2% relative, less than half the day-5 figure.
Reading the table correctly is most of the lesson. Test A's p-value says that a difference this large would be very unlikely if guest checkout had no effect, and its intervalA range of values for an effect, such as a lift in conversion rate, that is consistent with the data at a chosen confidence level. says the true effect is plausibly anywhere from a modest lift to a large one; the team shipped on the interval's lower bound being comfortably above zero, not on the headline. Test B's p-value does not say the change did nothing. Its interval runs from a loss of 0.7 points to a gain of 1.9 points, and the minimum effect the team had planned to detect was about 2.0 points on the new baseline. So the test rules out an effect at or above that planned size, but it cannot rule out a smaller gain or a small loss. The honest reading is "not the lift we planned for, and we do not know about anything smaller", and the decision was to leave the control in place and ask whether a gain below the planned minimum would be worth the cost of building and maintaining it. Test C was significant with an observed effect of 1.7 points, smaller than the roughly 2.0 points the planned minimum implied. That is an ordinary outcome, not a quirk of the sample: at 80% power the smallest difference that reaches significance is already about 70% of the minimum detectable effect even at exactly the planned sample, which here is about 1.4 points, so a significant result somewhere between that and the planned size is expected. It is a reminder that the minimum detectable effect is a planning quantity, not a threshold a result must clear. The test pitfalls guide covers the early-stopping temptation that test C's day-five reading illustrates. The conversion rate calculator is enough to turn order counts into the rates in the table; the intervals need the sample size guide.
What shipped and what was reverted
Test A's change shipped to everyone within a week of the read, and the account prompt after the order was kept because a share of guests used it. Test B did not ship. The product manager was pressed to ship it anyway on the grounds that it "could not hurt", and declined, because the interval included a loss and the test had been designed exactly to answer that question; shipping on a result that had not shown the planned effect would have meant the test was theatre. The change went back on the backlog with a note that a retest was only worth running if the team decided a smaller effect than the planned minimum was worth having, and that it would then need a much larger sample, since halving the detectable effect roughly quadruples the number of visitors needed.
Test C shipped and was then partly taken back. Within a fortnight of launch, support contacts per order rose, and the tickets had a pattern: customers who paid through the express wallet had skipped the delivery-instructions field, which the wallet flow bypassed, and their parcels were going to the wrong door. The guardrail of support contacts per order had been measured during the test, but the lag between order and complaint meant the signal arrived after the test had been read. The team reverted the wallet-first placement, kept the reassurance line, and recorded two lessons. The first is that a guardrail with a lag longer than the test needs a post-launch watch window, with a reversal criterion agreed in advance. The second is that bundling two changes in one variant made the lift unattributable: nobody could say whether the wallet placement or the reassurance line had produced it, so the reassurance line was queued for a retest on its own.
What generalises
The sequence generalises: research that ranks leaks by size before anyone proposes a fix, hypotheses that cite the evidence they came from, prioritisation by leak size and confidence, one test at a time on a shared flow, a sample and run length fixed in advance, and a single read at the end. The statistical reading generalises exactly, because it is not a property of this retailer: a p-value measures surprise under the assumption of no effect, a confidence interval bounds the plausible size of the effect and is the thing to decide on, an inconclusive test is a question the sample could not fully answer rather than an answer of "no", although its interval still shows which sizes of effect have been ruled out, and a result observed early is biased towards whatever made someone look. Running tests over whole weeks, checking that arms were assigned as planned before reading, and naming guardrails alongside the primary metric generalise to every test on every site. So does the post-launch discipline test C exposed: a shipped winner is still a hypothesis about the long term, and guardrails with lag need a watch window and a reversal rule. Writing results up, losses and ambiguities included, is what turns three tests into a programme rather than three events.
What does not generalise
The figures are invented, and so is the convenient fact that the largest leak produced the clearest win. Real programmes often find that the biggest drop-off is not where the easiest fix is, and that the first test is inconclusive. The baseline conversion rate, the weekly checkout volume and the sample sizes are properties of this made-up retailer; a site with a tenth of the traffic cannot detect effects of this size in three weeks and has to accept larger minimum effects, longer runs or different methods. A platform-hosted checkout limits what can be changed at all, which rules out some of these hypotheses before research begins. The particular findings — forced accounts, late delivery costs, a wallet flow that skipped a field — are common enough to be worth checking for but are not predictions about any other checkout. The team's ability to assign server-side and to log orders by arm is a prerequisite that many smaller retailers lack. And the statistics here are the fixed-horizon kind; teams using sequential or Bayesian methods read results differently and should follow those methods' rules rather than this page's. The method of deciding transfers; the decisions do not. The case studies library holds other labelled scenarios built the same way.
