Hypothesis
If visitors are served an optimised build of the page that loads and responds faster, more of them will convert, because fewer leave before the page is usable, which lets the speed work be valued against its cost. The decision is whether to fund page speed work on a key template, judged by the conversions it earns.
Speed work competes for engineering time with features that promise visible returns, and it often loses because its return is assumed rather than measured. The speed and mobile guide for landing pages explains why speed is a conversion factor and how to find the causes of a slow page; this test measures what fixing them is worth on your template. The direction matters: the test compares a faster build with today's build, never a deliberately slowed page with today's. Record the template, the changes in the optimised build, the expected speed gain and the cost of the work in the experiment hypothesis template.
Metrics and guardrails
The primary metric is the conversion rate of the template's primary action per assigned visitor, with Revenue per VisitorRevenue divided by the number of visitors in the same period. as the primary metric for stores, because a faster page can change basket size as well as the decision to buy. Use a single definition for both arms, counted from the business's own records where possible.
Three guardrails make the result interpretable. Core Web Vitals measured per arm, from real visitors, confirm that the optimised arm was actually faster in the field; web.dev documents the metrics and the web-vitals library that collects them from a site's own visitors, and without this guardrail a null result cannot be told apart from a build that was not faster. Bounce RateShare of sessions that were not engaged (GA4); historically single-page sessions. per arm shows the mechanism the hypothesis names: fewer visitors leaving before the page is usable. Tracking parity checks that the analytics and conversion tags fire the same way in both builds, because speed work often defers or removes scripts, and a conversion tag that loads later or not at all in the faster arm would bias the result.
Design
Randomise at the level of the visitor, split on the server or at the edge so the testing mechanism adds no delay, and keep each visitor in one arm on return visits. The control serves the current build of the template. The variant serves an optimised build of the same template: identical content, layout, offer and tags, with the speed work applied, such as lighter images, deferred non-essential scripts and reserved space for late-loading elements.
| Design element | Choice | Reason |
|---|---|---|
| Unit of randomisation | Visitor, split on the server and kept in one arm | The testing tool must not slow either arm |
| Arms | Current build against an optimised build of the same template | Speed is the only change |
| Held constant | Content, layout, offer, analytics and conversion tags | Isolates speed from design |
| Field speed | Core Web Vitals collected from real visitors in each arm | Proves the variant was faster where it counts |
| Devices | Pooled result first; mobile and desktop split named before launch | Slow devices may respond more |
| Direction | Faster than today, never slower | No deliberate harm to real visitors |
The optimised build should be one the team would ship. A heavily stripped page that drops features to win on speed tests a different page, not faster delivery of the same one. The Core Web Vitals guide lists the template-level fixes that usually move many pages, and Google Search Central's documentation on Core Web Vitals and search results covers why the same measures matter for page experience in search.
Sample size reasoning
Speed gains usually move conversion by a small amount, so the sample is large. The inputs are the template's current conversion rate, the smallest rise that would pay for the speed work, and the confidence the team requires. Work back from the cost: the conversions needed to repay the engineering time over a reasonable period define the minimum effect worth detecting. The sample size and significance guide explains how those inputs set the visitors each arm needs, and why revenue per visitor needs more than a conversion rate because it varies more between visitors. The conversion rate calculator gives the baseline rates to start from. If the template cannot reach the sample, choose a higher-traffic template or pool several that share the same speed fixes.
How sample size scales with the lift you want to detect
Take a baseline conversion rate of 1.80% and a smallest worthwhile rise to 1.98%, a 10% relative lift. Assume a two-sided 5% significance level and 80% power, meaning an 80% chance of detecting a lift of that size if it is real. The usual two-proportion approximation, visitors per arm = (1.96 + 0.84)² × (0.018 × 0.982 + 0.0198 × 0.9802) ÷ (0.0198 − 0.018)², gives about 89,700, so plan on 90,000 visitors per arm. At 60,000 per arm the same lift would be detected only about 63% of the time, so more than a third of the time a test that small would miss a real gain of that size.
Duration
Run at least two full weekly cycles across device types, read once the planned sample is reached. Weekday and weekend traffic differ in device mix and connection quality, which is exactly what speed interacts with, so whole weeks matter more here than in most tests. Avoid releases to either build during the run; any change must reach both arms at once or wait.
Reading the results and the decision rule
Read once the planned sample is reached. First confirm from the field data that the optimised arm was measurably faster and that tracking fired equally in both arms. Then compare conversion rates with their confidence interval, and revenue per visitor for stores. Read the device split named before launch last. The rule, written before launch:
| Result | Decision |
|---|---|
| Optimised arm converts more and its field speed is measurably better | Fund the speed work and ship the optimised build |
| Field speed improved but conversion did not change reliably | Record the null; prioritise speed for search and experience reasons instead |
| Field speed did not improve | The test did not test speed; fix the build and rerun |
| Tracking differed between arms | Discard the reading, fix the tags and rerun |
For a conversion rate, the significance checker below reads the confidence of the difference between the two arms from each arm's visitors and conversions once the test has finished. It does not plan the test, size it or read revenue per visitor.
A/B test significance checker
Pitfalls
Client-side testing tools slowing both arms is the first pitfall. A tool that loads its script in the browser and then rewrites the page adds delay before anything useful appears, in both arms, and can apply the variant late enough to cause visible flicker. The guide to common testing pitfalls explains flicker and why moving the split to the server avoids it; for a speed test, a server-side split is a requirement, not a refinement. For tests of anything other than speed, the guide to choosing between client-side and server-side testing sets out which changes can stay in the browser and which belong on the server or on separate pages.
Tracking firing differently in the faster build is the second. Speed work often defers scripts, and if the analytics or conversion tag is among them, the optimised arm may record fewer conversions than it earned, or record them later. Check tag firing in both builds before launch and compare counts with the order or lead system during the run. The third is reading lab scores instead of field data. Lab tools test one simulated device and connection and are for finding causes; only field measurements from each arm's real visitors show whether the variant was faster for the people in the test.
A knitwear shop values its speed work
Glenvarra Knitwear, an invented online knitwear shop, estimates that faster product pages would cost six weeks of developer time. It builds the optimised product template and splits visitors on the server for four weeks, with 90,000 visitors per arm, the sample the calculation above calls for. Field data shows Largest Contentful Paint about 1.1 seconds sooner at the 75th percentile in the optimised arm, and tag firing matches the order system in both arms. The control converts at 1.80% (1,620 orders) and the optimised arm at 1.98% (1,782 orders), a confidence of about 99.5% from the significance checker, with the gain concentrated on mobile, as the plan anticipated. Bounce rate falls in the optimised arm. At Glenvarra's margins, the extra orders repay the developer time within a few months, so the team ships the build and schedules the same work for category pages.
Common mistakes
- Slowing pages for real visitors. Test a faster build against today's, never the reverse.
- Using a client-side testing tool. It adds delay to both arms and hides the difference.
- Reading lab scores. Field data per arm is the only proof the variant was faster.
- Changing content along with speed. Then the test measures a redesign.
- Not checking tag parity. Deferred tags can undercount the faster arm.
- Treating a null as failure. Record it and prioritise speed on its other merits.
- Reporting conversion without field speed. Report CVRShare of visits or clicks that convert. and field speed side by side in the write-up, so the result can be repeated on the next template.
- Under-sizing the test. A sample below the planned size, such as 60,000 per arm where 90,000 are needed, misses a real gain far more often than the plan allows.
Next step. Where phones convert far below desktops, the diagnosis for a mobile conversion rate far below desktop separates speed from layout and checkout causes before any build is funded. The social proof placement test is a typical next test once the template is fast, and the experiment library lists every design.
