Why this matters
A relaunch breaks the intermediate habit of testing every change: many changes arrive together, most pages lack the orders a split needs, and the tracking often changes the same day. The guide to evaluating a website redesign sets the evaluation design and the guide to conversion work without an A/B test sorts what should simply ship; here you apply both under a launch date. If sample-size arithmetic is unfamiliar, revisit the intermediate CRO level.
The cookware maker and its relaunch
Vellerhoe Cookware, an invented cast-iron and enamel cookware maker selling direct online, has an agency that wants everything live on one day in October, before the Christmas peak. Addresses will not change, and the site can serve either set of templates to a visitor assigned at random and keep them on it.
Vellerhoe Cookware: the proposal and the traffic
| Proposed change | Note |
|---|---|
| New product template, all 140 products | Gallery, layout, rewritten copy |
| New navigation and category pages | Site-wide |
| One-page checkout | Replaces a three-step checkout |
| New tag set-up | Purchase event on a new confirmation page |
| Zoom fix | Gallery freezes on some phones |
| Basket errors | Shown only as a red outline |
| Delivery charge | First shown at payment |
| New homepage photography | Cheap to swap back |
| Page group | Visitors or starts a week | Step measured | Visitors per arm the team's calculation needs |
|---|---|---|---|
| Product template, all products | 16,000 | Add to basket, 7.0% | 22,000 |
| Flagship casserole page alone | 1,400 | Add to basket, 9.0% | 15,000 |
| Checkout | 650 starts | Order, 62% | 1,100 |
Research: tickets often ask whether a pan works on induction, and recordings show buyers checking the "Works on" line under the price before adding to basket. The new copy moves it into a tab.
A bundle of changes with no clean control
A relaunch is a one-way door whose total hides gains and losses that cancel. Make it smaller: ship what needs no question, test what the traffic can answer, and stage the rest against a concurrent comparison group.
Task
- Sort each change into ship now, test before launch or relaunch-only, with the reason.
- Work out how long each candidate test would take; say which run, and in what order.
- List what to record before the build, including what the brief must keep.
- Plan the rollout: comparison group, gates, prediction and rollback rules.
- Read the results below against the season, novelty and the tracking change, and decide at the gate.
Four weeks at a 50/50 staged rollout, read on the agreed date
| Measure | Old site | New site |
|---|---|---|
| Sessions | 31,000 | 31,000 |
| Orders, shop's order system | 470 | 505 |
| Orders, analytics | 471 | 580 |
| Average order value | £68 | £67 |
| Checkout support contacts | 19 | 23 |
| Week | New site against old, returning visitors | New site against old, new visitors |
|---|---|---|
| First | +14% | +6% |
| Fourth | +2% | +5% |
The agency compares the whole site with the four weeks before launch, in analytics: 1,051 orders on 62,000 sessions against 730 on 52,000, "a 21% lift". Last year the same weeks rose about a fifth on the weeks before. The gate rule: proceed if the new site's orders per session, from the order system, are no more than 5% below the old site's, with guardrails steady.
Worked answer
Sort. Ship now on the current site, without a test: the zoom fix (a bug), the basket errors (a barrier, which the accessibility guide fixes rather than tests) and the delivery charge (a missing cost). The photography is cheap and reversible: ship it with a prediction and a date to check. Test first: the product template and the checkout. Relaunch-only: navigation, category pages and the rest of the design, judged in the staged rollout. The tag set-up is not a design change: install it on the current site first, reconcile it with the order system and log it, so both arms are measured alike.
Weeks each candidate test needs
| Test | Arithmetic | Weeks |
|---|---|---|
| Product template, all products | 22,000 × 2 ÷ 16,000 | about 3 |
| Flagship page alone | 15,000 × 2 ÷ 1,400 | about 21 |
| Checkout | 1,100 × 2 ÷ 650 | about 3.4 |
Each row divides the visitors the sample-size calculation asks for, across both arms, by the visitors that page or step receives each week; the flagship page alone would need about five months of its own traffic.
Test the template across all products; the flagship page alone could not finish before the peak. Run the template test, then the checkout test, because both reach the same visitors. The template A/B TestControlled comparison of two variants. reads add to basket with orders as a Guardrail MetricA metric an experiment must not worsen beyond an agreed bound, watched beside the primary metric that decides the test.; the checkout test reads orders directly. Both arms pass a keyboard check first.
Baseline. Orders per session by device and source over full weeks beside the order system's counts; screenshots and proof elements; form fields; campaign destinations; speed; the conversion event. The brief keeps the "Works on" line under the price: research shows it persuading, and a tab would hide it.
Rollout. Random, sticky assignment, identical tracking, widening at gates with a named decision-maker; the prediction and reading date written before launch. Faults are fixed at once; a result below the agreed range restores the specific elements research points to, template by template, not the whole site.
Reading. Tracking first: analytics exceeds the order system in the new arm only, so the purchase event fires twice on the new confirmation page. A fault: fix it, annotate the gap in the measurement change log and read from the order system.
Reading Vellerhoe's four weeks
- Agency's before-and-after: 1,051 ÷ 62,000 ≈ 1.70% against 730 ÷ 52,000 ≈ 1.40%. Inflated by the double count, and last year's same weeks show the season alone moves the site by about a fifth.
- Concurrent comparison, order system: 505 ÷ 31,000 ≈ 1.63% against 470 ÷ 31,000 ≈ 1.52%, about 7% higher, on a few hundred orders per arm, so a range running from a small loss to a moderate gain.
- Returning visitors' lead shrinks, a novelty signature; new visitors, the better guide, hold steadier.
- Average order value and support contacts barely move.
The concurrent groups shared the season, so it cancels out. The new site's CVRShare of visits or clicks that convert. is inside the agreed range with steady guardrails: proceed to the next gate, reported as consistent with a small gain, not a lift of a fifth.
Common mistakes
- Accepting the one-day launch. Nothing can be attributed.
- Queuing bug and barrier fixes for the relaunch. They ship now, on the current site.
- Trusting the before-and-after. It mixes the season and a counting fault into the design.
- Judging on the first weeks. Novelty fades; read new visitors and wait for the date.
- Rolling everything back. Restore the specific elements that were lost.
The checkpoint on conversion work without a split test follows.
