Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Conversion Rate Optimisation › Experimentation

Shipping a Winning Test Variant

What to do after an A/B test wins: confirm the decision, rebuild the variant properly, check tracking, log the release, watch the metric and retire the test.
Tutorialintermediate10 min readUpdated

Why shipping is a step of its own

The A/B testing guide ends where most test write-ups end: at the decision and the documentation. The experiment results report template records that decision, with roll-out tasks and owners in its follow-ups field. What neither covers in detail is the release itself, and the release is where a good number of wins are lost.

They are lost in three ways. Some never ship at all: the experimentation programme playbook gives its "implemented" field more weight than any other, because winning tests often sit in a development queue behind other work. Some ship in a different form from the one tested, because someone improved the design on the way to production. And some ship as tested but never deliver the improvement the test reported, for reasons the testing pitfalls guide diagnoses: novelty that faded, a result read too early, or tracking that inflated one arm. The steps below close each of those gaps.

Before you start

  • The A/B testing guide, read. This tutorial assumes a test planned and read as that guide describes, with a written plan, a primary metric, guardrails and a planned sample. It is the prerequisite.
  • The test plan and the results report. The hypothesis, the decision rule, the confidence interval for the primary metric and the guardrail results, in the format of the results report template or its equivalent.
  • Screenshots of both arms on the devices that carry most of the traffic, taken while the test was running.
  • A developer who can build the change into the site's templates or components, and time booked for it.
  • The measurement change log, or whatever record the team keeps of releases that move the numbers.
  • The learning log where the test has an open entry.
  • About ninety minutes for the release work around the build itself: the checks, the log entry, the expectation, the watch plan and the clean-up.

The steps

  1. Confirm the decision against the plan

    Before anything is built, confirm that the win is a win by the rules written before the test started. Open the results report and check its first fields in order. Was the planned sample reached, and was the test read once rather than stopped the first time it looked significant? Did the sample ratio hold? Was the tracking verified in both arms during the run? Did every Guardrail Metric stay within its bound? And does the result meet the decision rule quoted from the plan? If any answer is no, the result is not yet a winner: it is a lead for a new test, or an inconclusive result. The playbook's rule is that the analyst, not the person who proposed the idea, signs off a win, and this is the moment that sign-off matters most.

  2. Decide exactly what ships

    Ship what was tested. Write down, from the screenshots and the test's own build, every element of the winning variant: copy, layout, images, behaviour on each device, and the audience it was shown to. A material change made "while we are at it", a new headline because the brand team prefers it or a second element moved because it looked better, is a new Hypothesis and does not inherit the test's result. If the test ran only on mobile or only for new visitors, the result covers that audience; extending the change to everyone is a decision to record, not a conclusion the test supports. If the variant bundled two changes, ship the bundle, or retest the parts separately; the illustrative checkout test case study shows how a bundled win becomes unattributable.

  3. Rebuild the variant in the site's own code

    Build the change into the site's templates or components, where the rest of the page lives, rather than leaving the testing script to serve it to everyone. A script left serving a winner has three costs. It adds weight: the speed and mobile guide lists testing tools among the third-party tags that together are often the largest part of a page, and notes that a client-side script which hides the page until it decides what to show adds a delay to every visit. It can flicker: the testing pitfalls guide describes the brief flash of the old version before the change is applied on slower devices. And it competes for the main thread: the troubleshooting page for slow interaction responsiveness names testing tools among the scripts the business added that keep the browser busy. Using the testing tool as a short bridge while the rebuild is scheduled is reasonable; give the bridge an end date. Once the change lives in the templates, the winner no longer depends on a script that could cause flicker.

  4. Check it on devices and in the tracking

    Check the rebuilt version against the screenshots of the winning arm on the devices and browsers that carry most of the traffic, including the slowest common phone. Look for differences the rebuild introduced: a font size, an image crop, the order of elements on a narrow screen. Complete the journey with the keyboard alone, because a component rebuilt from scratch can lose behaviour the test version had. Then work through the tracking verification checklist: every Key Event the test measured must fire once per real action on the rebuilt page, with the same parameters, because the expectation in step six assumes the metric is counted the way the test counted it. Reconcile a closed day with the business's own figures once the release is live.

  5. Release it with a dated log entry

    Release the change through the site's normal process and record it the same day in the measurement change log: the date and time it went live, the pages and templates it changed, a link to the results report, and the expected effect, which step six fills in. Log it as planned before the release and switch the status to live on the day, as the change log tutorial recommends, so that anyone reading reports that week knows what is coming. If the release slips, the entry is where the slip is recorded, so a later analyst does not credit the wrong week.

  6. Write the expectation as a range

    Write down what the release should do, taken from the test's confidence interval rather than its point estimate. The sample size and significance guide explains why: the observed lift is the single most likely value, but the result is the range, and a win reported as the point estimate will disappoint when the true effect turns out to sit nearer the lower bound. Translate the range into the units the business uses, such as orders, leads or bookings a week, and name the reading date and the comparison period. Add what would count as a problem: the primary metric below the lower end of the range on the reading date, or any guardrail outside its bound. The expectation is what the next steps are judged against.

  7. Watch the metric for new and returning visitors

    Monitor for faults from the first hour, and read the result on the reading date. Monitoring means confirming the key events arrive, the guardrails hold and nothing has broken; it is not reading the primary metric for a verdict. On the reading date, compare the primary metric and the guardrails with the expectation, separately for new and returning visitors. The testing pitfalls guide explains why: a Novelty Effect lives among returning visitors, who react to the change itself and then settle, so a strong early lead in that group that shrinks over the weeks is expected, and new visitors' results are the better guide to the long-run effect. A guardrail with a lag, such as returns or support contacts that arrive weeks after the order, needs its own watch window and a reversal rule, which the checkout case study learned the hard way.

  8. Keep a holdback where the change is large

    For a large change, one that touches several steps, alters pricing or packaging, or carries a lagging guardrail, keep a small random group on the old experience for a fixed period after release. The test hub's table of designs describes this as a Holdout Group: a group kept away from something already live, to measure its cumulative value, with the costs of a long duration and a group that forgoes the benefit. The group must be random and concurrent with everyone else, and each visitor must stay in the same group on every visit. Set the end date before release. When it passes, read the comparison once, then retire the old version and move the group onto the new one. For small, self-contained changes, skip this step.

  9. Remove the test and its code

    Switch the test off in the testing tool and remove everything it left behind: the variant's code and the old control path, any page-hiding snippet that was only needed while the test ran, and test-specific events or parameters, such as the exposure event the tutorial on analysing A/B tests in GA4 sends with the test and variant names. Once the test is switched off, an exposure event left firing only clutters the analysis of the next test. Archive the screenshots of both arms and the final version with the results report, because the next team to work on the page will want to see what was compared. A programme calendar that still shows the test as running will also block the next test on the same page.

  10. Close the learning-log entry

    Close the test's entry in the learning log the experimentation cadence guide describes: what shipped, when, in what form, and whether it matched the test exactly. Add the post-release reading against the expectation and, if a holdback ran, its result. The cadence guide's practice is to revisit shipped wins a quarter later against cohort data and log the revisit beside the original call; put that date in the entry now. Finally, mark the test as implemented, the field the experimentation playbook counts when it reports the share of winning tests that have actually shipped. The academy conversion strategy capstone asks you to write the shipping, holdback and learning-log rules for a whole conversion programme.

A rainwear shop ships a sizing change (illustrative)

Illustrative example — not a real result

Puddlebrace Rainwear ships its size-advice panel

Puddlebrace Rainwear, an invented rainwear retailer, tests a size-advice panel on its jacket pages that tells shoppers how each style fits compared with their usual size. The test runs to its planned 26,000 visitors per arm over four full weeks. Orders per visitor are 2.40% in the control and 2.71% in the variant, a relative lift of 13%, with a 95% interval running from about 2% to 24%. Returns, the guardrail, are flat during the test, though most returns arrive later. The analyst signs it off. A designer suggests adding a fit video to the panel at the same time; the team logs that as a separate hypothesis and ships the panel exactly as tested. The developer rebuilds it in the product template, the testing script is switched off for that page, and the release is logged for 3 March with an expected relative lift of 2% to 24% on orders per visitor and a reading date of 31 March. Because returns lag, 5% of visitors are kept on the old template for four weeks. On the reading date, the rebuilt panel shows orders per visitor 7% above the holdback group: smaller than the test's point estimate, inside its interval. Returning visitors started further ahead and settled by the third week; new visitors held steady at about 7% throughout. Returns among buyers who saw the panel fall slightly over the following six weeks. The holdback ends, the test code is removed, and the learning-log entry records a revisit in June.

Common mistakes

  • Shipping a result that was read early. Confirm the planned sample was reached and read once before anything is built.
  • Improving the variant on the way to production. A change to the winner is a new hypothesis, not part of the win.
  • Leaving the testing tool to serve the winner indefinitely. It adds weight and delay, can flicker, and hides the change outside the site's code.
  • Skipping the tracking check after the rebuild. A rebuilt page can count the key event differently from the test, and the expectation then means nothing.
  • Promising the point estimate. Write the expectation as the interval, in the business's own units.
  • Reading the first week as the verdict. Returning visitors' novelty fades; compare new and returning visitors on the reading date.
  • Watching lagging guardrails only during the test. Returns, refunds and complaints need a watch window after release.
  • Keeping a holdback with no end date. Set the date before release and retire the old version when it passes.
  • Leaving test code behind. Old variants, control paths and exposure events clutter the site and confuse the next test.
  • Never marking the test as implemented. The programme cannot see how many of its wins have reached customers.

Frequently asked questions

Can we leave the winning variant running in the testing tool instead of rebuilding it?

As a short bridge while the rebuild is scheduled, with an end date, yes. As the permanent way of serving the change, no. A client-side testing script adds weight to every page it runs on, can show the old version for a moment before the new one appears, and competes with the site's own code for the browser's main thread, and the site's speed and responsiveness guides list testing tools among the scripts that slow pages down. The change also lives outside the site's code, where the next developer will not find it.

Why is the improvement after rollout smaller than the test showed?

Often because the test's point estimate was read as the result. The point estimate is the middle of a range, and the true effect can sit anywhere in the confidence interval, frequently nearer its lower end. Other common causes are a novelty effect among returning visitors that faded, a test stopped early when it first looked significant, and a different season or traffic mix after launch. Write the expectation as the interval before release, and the rollout can be judged against it.

Does every winning test need a holdback group?

No. A holdback costs something, because the group kept on the old experience misses the improvement and the old version has to stay available. It earns that cost when the change is large, touches several steps of the journey, or carries a guardrail that only shows its effect weeks later, such as returns, refunds or cancellations. For a small, self-contained change with clean guardrails, a dated release, a written expectation and a check on the reading date are enough.

Terms used on this page

Frameworks

Where this page is used

Part of these learning paths

Also related

Related