Why shipping is a step of its own
The A/B testing guide ends where most test write-ups end: at the decision and the documentation. The experiment results report template records that decision, with roll-out tasks and owners in its follow-ups field. What neither covers in detail is the release itself, and the release is where a good number of wins are lost.
They are lost in three ways. Some never ship at all: the experimentation programme playbook gives its "implemented" field more weight than any other, because winning tests often sit in a development queue behind other work. Some ship in a different form from the one tested, because someone improved the design on the way to production. And some ship as tested but never deliver the improvement the test reported, for reasons the testing pitfalls guide diagnoses: novelty that faded, a result read too early, or tracking that inflated one arm. The steps below close each of those gaps.
Before you start
- The A/B testing guide, read. This tutorial assumes a test planned and read as that guide describes, with a written plan, a primary metric, guardrails and a planned sample. It is the prerequisite.
- The test planThe written design of a test, fixed before launch: hypothesis, unit of randomisation, primary metric, guardrails, planned sample, duration and decision rule. and the results report. The hypothesis, the decision ruleA rule written before a test starts that says which result ships the change, which leads to another round and which abandons it., the confidence interval for the primary metric and the guardrail results, in the format of the results report template or its equivalent.
- Screenshots of both arms on the devices that carry most of the traffic, taken while the test was running.
- A developer who can build the change into the site's templates or components, and time booked for it.
- The measurement change log, or whatever record the team keeps of releases that move the numbers.
- The learning log where the test has an open entry.
- About ninety minutes for the release work around the build itself: the checks, the log entry, the expectation, the watch plan and the clean-up.
The steps
Confirm the decision against the plan
Before anything is built, confirm that the win is a win by the rules written before the test started. Open the results report and check its first fields in order. Was the planned sample reached, and was the test read once rather than stopped the first time it looked significant? Did the sample ratio hold? Was the tracking verified in both arms during the run? Did every Guardrail MetricA metric an experiment must not worsen beyond an agreed bound, watched beside the primary metric that decides the test. stay within its bound? And does the result meet the decision rule quoted from the plan? If any answer is no, the result is not yet a winner: it is a lead for a new test, or an inconclusive result. The playbook's rule is that the analyst, not the person who proposed the idea, signs off a win, and this is the moment that sign-off matters most.
Decide exactly what ships
Ship what was tested. Write down, from the screenshots and the test's own build, every element of the winning variant: copy, layout, images, behaviour on each device, and the audience it was shown to. A material change made "while we are at it", a new headline because the brand team prefers it or a second element moved because it looked better, is a new HypothesisA testable statement of the change to make, the effect expected on a named metric and the evidence-based reason for expecting it. and does not inherit the test's result. If the test ran only on mobile or only for new visitors, the result covers that audience; extending the change to everyone is a decision to record, not a conclusion the test supports. If the variant bundled two changes, ship the bundle, or retest the parts separately; the illustrative checkout test case study shows how a bundled win becomes unattributable.
Rebuild the variant in the site's own code
Build the change into the site's templates or components, where the rest of the page lives, rather than leaving the testing script to serve it to everyone. A script left serving a winner has three costs. It adds weight: the speed and mobile guide lists testing tools among the third-party tags that together are often the largest part of a page, and notes that a client-side script which hides the page until it decides what to show adds a delay to every visit. It can flicker: the testing pitfalls guide describes the brief flash of the old version before the change is applied on slower devices. And it competes for the main thread: the troubleshooting page for slow interaction responsiveness names testing tools among the scripts the business added that keep the browser busy. Using the testing tool as a short bridge while the rebuild is scheduled is reasonable; give the bridge an end date. Once the change lives in the templates, the winner no longer depends on a script that could cause flickerThe brief flash of the original version before a browser-side test script applies the variant, which turns a result about the change into one about the fault..
Check it on devices and in the tracking
Check the rebuilt version against the screenshots of the winning arm on the devices and browsers that carry most of the traffic, including the slowest common phone. Look for differences the rebuild introduced: a font size, an image crop, the order of elements on a narrow screen. Complete the journey with the keyboard alone, because a component rebuilt from scratch can lose behaviour the test version had. Then work through the tracking verification checklist: every Key EventA GA4 event marked as important to the business, the property's measure of success. the test measured must fire once per real action on the rebuilt page, with the same parameters, because the expectation in step six assumes the metric is counted the way the test counted it. Reconcile a closed day with the business's own figures once the release is live.
Release it with a dated log entry
Release the change through the site's normal process and record it the same day in the measurement change log: the date and time it went live, the pages and templates it changed, a link to the results report, and the expected effect, which step six fills in. Log it as planned before the release and switch the status to live on the day, as the change log tutorial recommends, so that anyone reading reports that week knows what is coming. If the release slips, the entry is where the slip is recorded, so a later analyst does not credit the wrong week.
Write the expectation as a range
Write down what the release should do, taken from the test's confidence interval rather than its point estimate. The sample size and significance guide explains why: the observed lift is the single most likely value, but the result is the range, and a win reported as the point estimate will disappoint when the true effect turns out to sit nearer the lower bound. Translate the range into the units the business uses, such as orders, leads or bookings a week, and name the reading date and the comparison period. Add what would count as a problem: the primary metric below the lower end of the range on the reading date, or any guardrail outside its bound. The expectation is what the next steps are judged against.
Watch the metric for new and returning visitors
Monitor for faults from the first hour, and read the result on the reading date. Monitoring means confirming the key events arrive, the guardrails hold and nothing has broken; it is not reading the primary metric for a verdict. On the reading date, compare the primary metric and the guardrails with the expectation, separately for new and returning visitors. The testing pitfalls guide explains why: a Novelty EffectA temporary change in behaviour caused by a change being new, which fades as users get used to it. lives among returning visitors, who react to the change itself and then settle, so a strong early lead in that group that shrinks over the weeks is expected, and new visitors' results are the better guide to the long-run effect. A guardrail with a lag, such as returns or support contacts that arrive weeks after the order, needs its own watch window and a reversal rule, which the checkout case study learned the hard way.
Keep a holdback where the change is large
For a large change, one that touches several steps, alters pricing or packaging, or carries a lagging guardrail, keep a small random group on the old experience for a fixed period after release. The test hub's table of designs describes this as a Holdout GroupA randomly chosen part of an audience kept from exposure to measure what exposure caused.: a group kept away from something already live, to measure its cumulative value, with the costs of a long duration and a group that forgoes the benefit. The group must be random and concurrent with everyone else, and each visitor must stay in the same group on every visit. Set the end date before release. When it passes, read the comparison once, then retire the old version and move the group onto the new one. For small, self-contained changes, skip this step.
Remove the test and its code
Switch the test off in the testing tool and remove everything it left behind: the variant's code and the old control path, any page-hiding snippet that was only needed while the test ran, and test-specific events or parameters, such as the exposure event the tutorial on analysing A/B tests in GA4 sends with the test and variant names. Once the test is switched off, an exposure eventAn analytics event fired when a visitor sees a test arm, carrying the test and variant, so each arm's behaviour can be counted outside the testing tool. left firing only clutters the analysis of the next test. Archive the screenshots of both arms and the final version with the results report, because the next team to work on the page will want to see what was compared. A programme calendar that still shows the test as running will also block the next test on the same page.
Close the learning-log entry
Close the test's entry in the learning log the experimentation cadence guide describes: what shipped, when, in what form, and whether it matched the test exactly. Add the post-release reading against the expectation and, if a holdback ran, its result. The cadence guide's practice is to revisit shipped wins a quarter later against cohort data and log the revisit beside the original call; put that date in the entry now. Finally, mark the test as implemented, the field the experimentation playbook counts when it reports the share of winning tests that have actually shipped. The academy conversion strategy capstone asks you to write the shipping, holdback and learning-log rules for a whole conversion programme.
A rainwear shop ships a sizing change (illustrative)
Puddlebrace Rainwear ships its size-advice panel
Puddlebrace Rainwear, an invented rainwear retailer, tests a size-advice panel on its jacket pages that tells shoppers how each style fits compared with their usual size. The test runs to its planned 26,000 visitors per arm over four full weeks. Orders per visitor are 2.40% in the control and 2.71% in the variant, a relative lift of 13%, with a 95% interval running from about 2% to 24%. Returns, the guardrail, are flat during the test, though most returns arrive later. The analyst signs it off. A designer suggests adding a fit video to the panel at the same time; the team logs that as a separate hypothesis and ships the panel exactly as tested. The developer rebuilds it in the product template, the testing script is switched off for that page, and the release is logged for 3 March with an expected relative lift of 2% to 24% on orders per visitor and a reading date of 31 March. Because returns lag, 5% of visitors are kept on the old template for four weeks. On the reading date, the rebuilt panel shows orders per visitor 7% above the holdback group: smaller than the test's point estimate, inside its interval. Returning visitors started further ahead and settled by the third week; new visitors held steady at about 7% throughout. Returns among buyers who saw the panel fall slightly over the following six weeks. The holdback ends, the test code is removed, and the learning-log entry records a revisit in June.
Common mistakes
- Shipping a result that was read early. Confirm the planned sample was reached and read once before anything is built.
- Improving the variant on the way to production. A change to the winner is a new hypothesis, not part of the win.
- Leaving the testing tool to serve the winner indefinitely. It adds weight and delay, can flicker, and hides the change outside the site's code.
- Skipping the tracking check after the rebuild. A rebuilt page can count the key event differently from the test, and the expectation then means nothing.
- Promising the point estimate. Write the expectation as the interval, in the business's own units.
- Reading the first week as the verdict. Returning visitors' novelty fades; compare new and returning visitors on the reading date.
- Watching lagging guardrails only during the test. Returns, refunds and complaints need a watch window after release.
- Keeping a holdback with no end date. Set the date before release and retire the old version when it passes.
- Leaving test code behind. Old variants, control paths and exposure events clutter the site and confuse the next test.
- Never marking the test as implemented. The programme cannot see how many of its wins have reached customers.
