Why redesigns are hard to judge
Most redesigns are judged by comparing the weeks after launch with the weeks before. The experimentation hub explains why that comparison is weak for any change: everything else that moved in the same period, a promotion, a new traffic source, a tracking fix, the weather, is mixed into the result. A redesign makes the problem worse in four ways.
It changes many things at once. A product page redesign may move the reviews, shorten the description, restyle the button and rewrite the delivery message in one release. If conversion holds steady, a real improvement in one element may be cancelling a real loss in another, and nobody can tell which. The site migrations guide lists changing URLs, templates and content in one release among its common mistakes for the same reason: the losses cannot be attributed.
It often changes the measurement. New templates mean new buttons, forms and confirmation pages, and tags that relied on the old ones can stop firing or start counting something else. The diagnosis for a conversion rate that dropped after a redesign puts broken or redefined tracking first among its causes, because a reported fall in the conversion rate (CVRShare of visits or clicks that convert.) can happen while orders in the business's own system hold.
Returning visitors react to the change itself. The testing pitfalls guide describes novelty effects, where returning visitors notice something new and engage with it for a while, and primacy effects, where they resist a layout they have to relearn. Both fade, so the first weeks after a relaunch say more about familiarity than about the design.
It can move search traffic at the same time. If the relaunch changes addresses, the number of visits from search can shift while the engine recrawls and reprocesses the site. That is a separate problem from the share of visitors who convert, with separate causes, and the two have to be read apart.
None of this means redesigns cannot be judged. It means the evaluation has to be designed before the build, not improvised after launch.
Evolve or relaunch
The first decision is whether a redesign needs to happen all at once. The one-way and two-way doors framework sorts decisions by how hard they are to undo. A full relaunch is a one-way door: costly to reverse, it changes what returning customers expect, and it can disturb search rankings and the tracking history. Each element of a redesign taken separately, such as a new product gallery, a shorter form or a clearer delivery message, is usually a two-way door that can be tested or shipped and watched on its own.
| Approach | When it fits | How it is evaluated | Main risk |
|---|---|---|---|
| Evolve in steps | The current platform can carry the new design; the goals are about conversion | Each change tested or shipped with a prediction, one at a time | Slow; the site can feel inconsistent while steps are under way |
| Relaunch in phases | A platform or brand change is forced, but templates can move one at a time | Key templates tested first; the rest rolled out to a comparison group | Two systems to run and keep consistent for a period |
| Relaunch at once | The old platform cannot be kept, or a rebrand has a fixed public date | Before-and-after with matched weeks, a written prediction and rollback rules | Every change arrives together and cannot be attributed |
Evolve where possible, and relaunch only where a platform or brand change forces it. When a relaunch is forced, make the one-way door smaller. The site migrations guide notes that Google's documentation on site moves, listed in this page's sources, treats moves with URL changes separately from moves without, and its practical advice follows the same line: keep URLs where they can be kept, and if they must change, change nothing else in the same release. Applied to a redesign, that means moving platform first with the old design as close to intact as possible, then changing the design in measured steps, or the reverse, rather than both together. The academy exercise on evaluating a website relaunch for a cookware maker practises sorting an agency's relaunch proposal into changes to ship, test or keep for the relaunch.
A relaunch also benefits from decision points. The phase-gate reviews framework divides a large piece of work into phases with a go, change, hold or stop decision between each, judged against criteria written before the phase began. For a redesign the natural gates are the end of the baseline recording, the end of the template tests, the end of the staged rollout and the full switch-over. Each gate has a named decision-maker and criteria agreed in advance, so that a disappointing result leads to a decision rather than to moved goalposts. When the redesign is part of a new name or identity, the rebrand rollout tutorial sequences the channels; this guide covers the evaluation.
What to record before the build
A redesign cannot be judged against a baseline nobody recorded. The prevention routine in the redesign diagnosis sets out most of it, and it is worth doing before the design work starts, because it also tells the designers what they must not lose.
- What converts now. The pages that carry most conversions, with their rates by source and device over comparable full weeks, and the counts of orders, leads or bookings from the business's own system for the same weeks.
- What sits on those pages. Screenshots on a phone and a desktop, the proof elements on each page, such as reviews, guarantees, credentials and delivery terms, and every form field with its validation.
- Every campaign destination. An export of the final URLs in ad accounts, links in active emails and partner links, so that none of them ends on a redirect to the homepage after launch.
- Current speed. Field data for loading and responsiveness on the main templates, so the new ones can be compared.
- The measurement definitions. Which Key EventA GA4 event marked as important to the business, the property's measure of success. counts as the conversion on each template, and where it fires.
- The search baseline, if addresses will change. The migrations guide lists rankings, organic traffic by page, Search Console performance and the pages with external links.
Then do the research that finds what is quietly doing the persuading. The conversion research methods guide describes recordings that show where visitors look before they decide, polls that ask what nearly stopped them, and support tickets that reveal the questions the page answers. A delivery line beside the basket button or a review count under the product title can be doing work that nobody designed it to do. Findings like these go into the design brief as elements to keep or to improve deliberately, not to tidy away.
Testing key templates before launch
The templates that carry most conversions, often the product page, the plan or pricing page, the basket and the main lead form, are worth testing on their own before the rest of the site moves. The test hub's table of designs names the right one: a split-URL test compares two separately built pages with traffic redirected between them, and it suits redesigns and structural changes too large for in-page edits.
The same table names the two things to watch. Redirect latency adds a delay to whichever arm is redirected, so that arm is tested with a handicap the launched version will not carry. Tracking the redirect needs care, so that the assignment, the landing and the conversion are all recorded for both arms. The testing pitfalls guide adds the most common failure: a redirect that fails on some browsers sends those visitors back to the control, producing a sample ratio mismatch that makes the arms incomparable. Check the split daily, by device and browser.
Run the test as the A/B testing guide describes: a written plan with the primary metric, guardrails, the sample and the reading date; QA of both versions on real devices; the tracking walked in each arm before the test counts; and a single read at the planned sample. Test the new template against the old one as a whole, because that is the decision the business faces. If the new template loses, the research and the recordings for each arm are cheaper places to find out why than a launched site.
Where a new flow has no traffic yet, or the template serves too few visitors for a test, the methods guide's usability sessions on a prototype or a staging version are the fallback. They will not size the effect, but they find the places where people cannot complete the task.
Rolling out against a comparison group
Once the key templates have been tested, the rest of the redesign can move in a way that still leaves a comparison. Three designs are available, in order of strength.
| Design | What is compared | Strength | Weakness |
|---|---|---|---|
| Staged rollout to a random share | Visitors assigned at random to the new or old site, both running at the same time | Controlled: the season and the traffic mix hit both groups equally | Two sites to run and keep consistent; assignment must be sticky |
| Holdout after launch | A small random group kept on the old experience for a fixed period | Measures the cumulative effect of everything that changed | The holdout group forgoes any benefit; costly to keep the old site alive |
| Before-and-after with matched weeks | The weeks after launch against comparable weeks before | Possible on any platform | Cannot separate the redesign from everything else that changed |
The test hub is precise about the first: a staged rollout is not a controlled comparison unless the groups are random and concurrent. A rollout that sends the new site to mobile visitors first, or to one country first, or that switches everyone over on a fixed day, is a sequence of before-and-after comparisons, not a test. To make it a comparison, assign visitors at random, keep each visitor in the same group on every visit, measure both groups with identical tracking, and widen the new site's share only at the gates agreed in advance. A Holdout GroupA randomly chosen part of an audience kept from exposure to measure what exposure caused. works on the same principle after the switch-over, and the hub's table names its costs: a long duration and a group that does without the benefit.
Where the platform cannot serve two sites at once, which is common when the addresses change, the fallback is a before-and-after comparison read with its limits stated. Choose matched weeks, aligned by weekday and by the holidays that move, and the same weeks a year earlier where the business is seasonal. Compare rates by source and device rather than the overall figure, because a relaunch is often timed with a campaign that changes the mix. Write the prediction, including the range of results that would count as success, before launch.
When addresses do change, the move follows the site migrations guide, which draws on Google's documentation on site moves: every old URL mapped one to one to its equivalent with permanent redirectsA permanent server redirect telling browsers and search engines a URL has moved for good. and no chains, the old sitemap kept available for a period so that the old URLs are recrawled and their redirects discovered, and the redirects kept in place for at least a year, the period the guide says Google documents for signals to transfer. The website migration checklist turns that into a tickable list. The paid side has its own checks, which the guide to Google Ads during a website migration sets out: final URLs updated, the conversion tags and triggers moved to the new templates, and the click identifier tested through every redirect pattern so that conversions can still be credited to the click.
Reading the result fairly
Reading starts with the measurement, not the result. Work through the tracking verification checklist on the live site, confirm that every key event fires once per real action on the new templates, and reconcile a closed day of conversions with the business's own system. A dated entry in the measurement change log marks the launch, with the expected effect written down, so that every later report crossing the date carries the caveat that destinations, templates and possibly definitions changed that day.
Then account for novelty. The testing pitfalls guide's check applies to a relaunch as much as to a test: plot the daily difference across the reading period, split new from returning visitors, and look for the signature of a novelty effect, a strong early lead among returning visitors that shrinks steadily, or of a primacy effect, the same shape reversed. New visitors have no memory of the old site, so their results are the better guide to the long-run effect, and the reading period should run long enough for the returning visitors' trend to flatten.
Then read once, on the date fixed in advance. Evan Miller's essay on how not to run an A/B test, which the testing guides cite for the rule that the sample is fixed in advance and read once, shows how checking repeatedly and stopping on the first good result inflates false positives. A relaunch watched daily by a team that wants it to succeed is under the same pressure, and the defence is the same: decide when to read before there is anything to look at.
Read the guardrails beside the primary metric: order value, returns, support contacts and speed. Read segments only where the plan named them. And keep search separate. If organic visits fell after launch while the share of visitors converting held, the problem is redirects, indexing or lost content, which the traffic drop after migration diagnosis covers. If the share converting fell, the conversion diagnosis applies. A relaunch often causes a little of each, and they need different fixes.
Rollback rules agreed in advance
A rollback decided in the week after launch is decided under pressure, by whoever is most alarmed. Agreed before launch, it is a rule the team can apply calmly. The migrations guide's pre-launch checks already ask for a written rollback plan for when something critical breaks; for a redesign, the plan also covers what happens when the design itself disappoints.
Separate faults from verdicts. A fault, such as a key event that no longer fires, a form that fails on one browser, a campaign URL that redirects to the homepage or a payment step that errors, is fixed at once and never waits for the reading date. A verdict on the design waits for the reading date and the criteria agreed at the gate.
| Trigger | Action | Decided by |
|---|---|---|
| Tracking broken or counting differently | Fix immediately; annotate the gap; restart the reading period if needed | Measurement owner |
| A journey step failing for some visitors | Fix immediately as a bug; no test | Web lead |
| Business figures below the agreed range on the reading date, tracking verified | Restore the specific elements research points to, or roll the affected templates back | Named decision-maker at the gate |
| Results inside the agreed range | Continue the rollout to the next gate | Named decision-maker at the gate |
| Results above the agreed range | Continue, and record which elements to keep in future templates | Named decision-maker at the gate |
The redesign diagnosis answers the question most teams ask first: a full rollback is rarely the right response, because it discards whatever the redesign improved and usually reintroduces the problems it was meant to solve. Comparing old and new versions of the pages that carry most conversions, and restoring the specific elements that were lost, is usually faster. Write the rollback rule that way, by template and by element, with the old versions kept available until the final gate has passed.
A tea retailer measures its relaunch (illustrative)
Steepwhorl Teas evaluates a platform move
Steepwhorl Teas, an invented loose-leaf tea retailer, has to leave its old shop platform by the end of the year. Before the build, the team records its best-converting pages: the product pages for its black tea range carry 41% of orders, at a session conversion rate of 2.9% over the previous eight full weeks, against 2.1% for the site as a whole. Recordings and a poll on those pages show that the brewing-guide panel under the price is read by most buyers, so it goes into the brief as an element to keep. The new product template is split-tested against the old one on the black tea range for four weeks, with visitors assigned at random and the split checked daily by browser. It reaches its planned 18,000 visitors per arm, and orders per visitor come out 2.9% for the old template against 3.1% for the new, with an interval running from a small loss to a moderate gain. The team records the result as inconclusive but not harmful and proceeds to the next gate. The rest of the site is rolled out to a random half of visitors for six weeks, with both versions tracked by the same key event and reconciled daily with the order system. Overall orders per visitor are 2.2% on the new site and 2.1% on the old. Returning visitors on the new site start 12% ahead and fall back to level by the fourth week; new visitors are 4% ahead throughout. The decision-maker approves the switch-over, the change log records the date, and the brewing-guide panel is carried into every product template.
Common mistakes
- Changing platform, URLs, templates and content in one release. Nothing can be attributed; make the one-way door smaller.
- Starting the design before recording the baseline. Without screenshots, proof inventories and business counts, there is nothing fair to compare with.
- Calling a phased launch a test. A rollout by device, country or date is a sequence of before-and-after comparisons unless assignment is random and concurrent.
- Reading the first weeks as the verdict. Returning visitors' novelty and primacy effects fade; read new visitors separately and wait for the reading date.
- Judging conversion before checking tracking. A rebuilt site that counts differently can show a fall or a rise that never happened.
- Comparing a launch month with a different season. Match weeks and holidays, and use last year's weeks where the business is seasonal.
- Mixing search traffic losses with conversion losses. Read visits from search and the share converting separately; their causes differ.
- Redirecting old pages to the homepage. Map every old address and campaign destination to its equivalent.
- Deciding rollback in the moment. Write the triggers, actions and decision-makers down before launch.
- Rolling back everything. Restore the specific elements that were lost, template by template.
