Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Experiment Library

Page Speed Impact Test

An experiment design for valuing a faster page: a visitor split between the current and an optimised build, read on conversion with field speed per arm.
Experimentadvanced8 min readUpdated

Hypothesis

If visitors are served an optimised build of the page that loads and responds faster, more of them will convert, because fewer leave before the page is usable, which lets the speed work be valued against its cost. The decision is whether to fund page speed work on a key template, judged by the conversions it earns.

Speed work competes for engineering time with features that promise visible returns, and it often loses because its return is assumed rather than measured. The speed and mobile guide for landing pages explains why speed is a conversion factor and how to find the causes of a slow page; this test measures what fixing them is worth on your template. The direction matters: the test compares a faster build with today's build, never a deliberately slowed page with today's. Record the template, the changes in the optimised build, the expected speed gain and the cost of the work in the experiment hypothesis template.

Metrics and guardrails

The primary metric is the conversion rate of the template's primary action per assigned visitor, with Revenue per Visitor as the primary metric for stores, because a faster page can change basket size as well as the decision to buy. Use a single definition for both arms, counted from the business's own records where possible.

Three guardrails make the result interpretable. Core Web Vitals measured per arm, from real visitors, confirm that the optimised arm was actually faster in the field; web.dev documents the metrics and the web-vitals library that collects them from a site's own visitors, and without this guardrail a null result cannot be told apart from a build that was not faster. Bounce Rate per arm shows the mechanism the hypothesis names: fewer visitors leaving before the page is usable. Tracking parity checks that the analytics and conversion tags fire the same way in both builds, because speed work often defers or removes scripts, and a conversion tag that loads later or not at all in the faster arm would bias the result.

Design

Randomise at the level of the visitor, split on the server or at the edge so the testing mechanism adds no delay, and keep each visitor in one arm on return visits. The control serves the current build of the template. The variant serves an optimised build of the same template: identical content, layout, offer and tags, with the speed work applied, such as lighter images, deferred non-essential scripts and reserved space for late-loading elements.

Design elementChoiceReason
Unit of randomisationVisitor, split on the server and kept in one armThe testing tool must not slow either arm
ArmsCurrent build against an optimised build of the same templateSpeed is the only change
Held constantContent, layout, offer, analytics and conversion tagsIsolates speed from design
Field speedCore Web Vitals collected from real visitors in each armProves the variant was faster where it counts
DevicesPooled result first; mobile and desktop split named before launchSlow devices may respond more
DirectionFaster than today, never slowerNo deliberate harm to real visitors

The optimised build should be one the team would ship. A heavily stripped page that drops features to win on speed tests a different page, not faster delivery of the same one. The Core Web Vitals guide lists the template-level fixes that usually move many pages, and Google Search Central's documentation on Core Web Vitals and search results covers why the same measures matter for page experience in search.

Sample size reasoning

Speed gains usually move conversion by a small amount, so the sample is large. The inputs are the template's current conversion rate, the smallest rise that would pay for the speed work, and the confidence the team requires. Work back from the cost: the conversions needed to repay the engineering time over a reasonable period define the minimum effect worth detecting. The sample size and significance guide explains how those inputs set the visitors each arm needs, and why revenue per visitor needs more than a conversion rate because it varies more between visitors. The conversion rate calculator gives the baseline rates to start from. If the template cannot reach the sample, choose a higher-traffic template or pool several that share the same speed fixes.

Illustrative example — not a real result

How sample size scales with the lift you want to detect

Take a baseline conversion rate of 1.80% and a smallest worthwhile rise to 1.98%, a 10% relative lift. Assume a two-sided 5% significance level and 80% power, meaning an 80% chance of detecting a lift of that size if it is real. The usual two-proportion approximation, visitors per arm = (1.96 + 0.84)² × (0.018 × 0.982 + 0.0198 × 0.9802) ÷ (0.0198 − 0.018)², gives about 89,700, so plan on 90,000 visitors per arm. At 60,000 per arm the same lift would be detected only about 63% of the time, so more than a third of the time a test that small would miss a real gain of that size.

Duration

Run at least two full weekly cycles across device types, read once the planned sample is reached. Weekday and weekend traffic differ in device mix and connection quality, which is exactly what speed interacts with, so whole weeks matter more here than in most tests. Avoid releases to either build during the run; any change must reach both arms at once or wait.

Reading the results and the decision rule

Read once the planned sample is reached. First confirm from the field data that the optimised arm was measurably faster and that tracking fired equally in both arms. Then compare conversion rates with their confidence interval, and revenue per visitor for stores. Read the device split named before launch last. The rule, written before launch:

ResultDecision
Optimised arm converts more and its field speed is measurably betterFund the speed work and ship the optimised build
Field speed improved but conversion did not change reliablyRecord the null; prioritise speed for search and experience reasons instead
Field speed did not improveThe test did not test speed; fix the build and rerun
Tracking differed between armsDiscard the reading, fix the tags and rerun

For a conversion rate, the significance checker below reads the confidence of the difference between the two arms from each arm's visitors and conversions once the test has finished. It does not plan the test, size it or read revenue per visitor.

A/B test significance checker

Pitfalls

Client-side testing tools slowing both arms is the first pitfall. A tool that loads its script in the browser and then rewrites the page adds delay before anything useful appears, in both arms, and can apply the variant late enough to cause visible flicker. The guide to common testing pitfalls explains flicker and why moving the split to the server avoids it; for a speed test, a server-side split is a requirement, not a refinement. For tests of anything other than speed, the guide to choosing between client-side and server-side testing sets out which changes can stay in the browser and which belong on the server or on separate pages.

Tracking firing differently in the faster build is the second. Speed work often defers scripts, and if the analytics or conversion tag is among them, the optimised arm may record fewer conversions than it earned, or record them later. Check tag firing in both builds before launch and compare counts with the order or lead system during the run. The third is reading lab scores instead of field data. Lab tools test one simulated device and connection and are for finding causes; only field measurements from each arm's real visitors show whether the variant was faster for the people in the test.

Illustrative example — not a real result

A knitwear shop values its speed work

Glenvarra Knitwear, an invented online knitwear shop, estimates that faster product pages would cost six weeks of developer time. It builds the optimised product template and splits visitors on the server for four weeks, with 90,000 visitors per arm, the sample the calculation above calls for. Field data shows Largest Contentful Paint about 1.1 seconds sooner at the 75th percentile in the optimised arm, and tag firing matches the order system in both arms. The control converts at 1.80% (1,620 orders) and the optimised arm at 1.98% (1,782 orders), a confidence of about 99.5% from the significance checker, with the gain concentrated on mobile, as the plan anticipated. Bounce rate falls in the optimised arm. At Glenvarra's margins, the extra orders repay the developer time within a few months, so the team ships the build and schedules the same work for category pages.

Common mistakes

  • Slowing pages for real visitors. Test a faster build against today's, never the reverse.
  • Using a client-side testing tool. It adds delay to both arms and hides the difference.
  • Reading lab scores. Field data per arm is the only proof the variant was faster.
  • Changing content along with speed. Then the test measures a redesign.
  • Not checking tag parity. Deferred tags can undercount the faster arm.
  • Treating a null as failure. Record it and prioritise speed on its other merits.
  • Reporting conversion without field speed. Report CVR and field speed side by side in the write-up, so the result can be repeated on the next template.
  • Under-sizing the test. A sample below the planned size, such as 60,000 per arm where 90,000 are needed, misses a real gain far more often than the plan allows.

Next step. Where phones convert far below desktops, the diagnosis for a mobile conversion rate far below desktop separates speed from layout and checkout causes before any build is funded. The social proof placement test is a typical next test once the template is fast, and the experiment library lists every design.

Frequently asked questions

Why not slow the page down for some visitors to measure the cost of slowness?

Because it harms real customers to answer a question the business can answer another way. Deliberate slowdowns degrade the experience of people who came to buy, and they measure a different thing: the cost of a page slower than today, not the value of one faster than today. Build the faster version and test it against the current one.

Can a client-side testing tool run this test?

It is the wrong tool for it. A client-side tool loads its own script and applies the variant after the page starts rendering, which adds delay to both arms and can erase the very difference being tested. Split visitors on the server or at the edge, so each arm is delivered as a complete page.

What if the optimised build is faster but conversion does not change?

Record the null result honestly. Speed still matters for page experience in search and for visitors on slow devices, so the work may be worth doing for those reasons, but it should be prioritised on them rather than on a conversion gain the test did not find.

Terms used on this page

Tools to try

Frameworks

Where this page is used

Also related

Sources

  1. Web Vitals — web.dev
  2. Understanding Core Web Vitals and Google search results — Google Search Central

Related