Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Conversion Rate Optimisation › Experimentation

Evaluating a Website Redesign

How to judge a website redesign: evolve or relaunch, record what already works, test key templates, roll out against a comparison group and read it fairly.
Guideadvanced15 min readUpdated

Why redesigns are hard to judge

Most redesigns are judged by comparing the weeks after launch with the weeks before. The experimentation hub explains why that comparison is weak for any change: everything else that moved in the same period, a promotion, a new traffic source, a tracking fix, the weather, is mixed into the result. A redesign makes the problem worse in four ways.

It changes many things at once. A product page redesign may move the reviews, shorten the description, restyle the button and rewrite the delivery message in one release. If conversion holds steady, a real improvement in one element may be cancelling a real loss in another, and nobody can tell which. The site migrations guide lists changing URLs, templates and content in one release among its common mistakes for the same reason: the losses cannot be attributed.

It often changes the measurement. New templates mean new buttons, forms and confirmation pages, and tags that relied on the old ones can stop firing or start counting something else. The diagnosis for a conversion rate that dropped after a redesign puts broken or redefined tracking first among its causes, because a reported fall in the conversion rate (CVR) can happen while orders in the business's own system hold.

Returning visitors react to the change itself. The testing pitfalls guide describes novelty effects, where returning visitors notice something new and engage with it for a while, and primacy effects, where they resist a layout they have to relearn. Both fade, so the first weeks after a relaunch say more about familiarity than about the design.

It can move search traffic at the same time. If the relaunch changes addresses, the number of visits from search can shift while the engine recrawls and reprocesses the site. That is a separate problem from the share of visitors who convert, with separate causes, and the two have to be read apart.

None of this means redesigns cannot be judged. It means the evaluation has to be designed before the build, not improvised after launch.

Evolve or relaunch

The first decision is whether a redesign needs to happen all at once. The one-way and two-way doors framework sorts decisions by how hard they are to undo. A full relaunch is a one-way door: costly to reverse, it changes what returning customers expect, and it can disturb search rankings and the tracking history. Each element of a redesign taken separately, such as a new product gallery, a shorter form or a clearer delivery message, is usually a two-way door that can be tested or shipped and watched on its own.

ApproachWhen it fitsHow it is evaluatedMain risk
Evolve in stepsThe current platform can carry the new design; the goals are about conversionEach change tested or shipped with a prediction, one at a timeSlow; the site can feel inconsistent while steps are under way
Relaunch in phasesA platform or brand change is forced, but templates can move one at a timeKey templates tested first; the rest rolled out to a comparison groupTwo systems to run and keep consistent for a period
Relaunch at onceThe old platform cannot be kept, or a rebrand has a fixed public dateBefore-and-after with matched weeks, a written prediction and rollback rulesEvery change arrives together and cannot be attributed

Evolve where possible, and relaunch only where a platform or brand change forces it. When a relaunch is forced, make the one-way door smaller. The site migrations guide notes that Google's documentation on site moves, listed in this page's sources, treats moves with URL changes separately from moves without, and its practical advice follows the same line: keep URLs where they can be kept, and if they must change, change nothing else in the same release. Applied to a redesign, that means moving platform first with the old design as close to intact as possible, then changing the design in measured steps, or the reverse, rather than both together. The academy exercise on evaluating a website relaunch for a cookware maker practises sorting an agency's relaunch proposal into changes to ship, test or keep for the relaunch.

A relaunch also benefits from decision points. The phase-gate reviews framework divides a large piece of work into phases with a go, change, hold or stop decision between each, judged against criteria written before the phase began. For a redesign the natural gates are the end of the baseline recording, the end of the template tests, the end of the staged rollout and the full switch-over. Each gate has a named decision-maker and criteria agreed in advance, so that a disappointing result leads to a decision rather than to moved goalposts. When the redesign is part of a new name or identity, the rebrand rollout tutorial sequences the channels; this guide covers the evaluation.

What to record before the build

A redesign cannot be judged against a baseline nobody recorded. The prevention routine in the redesign diagnosis sets out most of it, and it is worth doing before the design work starts, because it also tells the designers what they must not lose.

  • What converts now. The pages that carry most conversions, with their rates by source and device over comparable full weeks, and the counts of orders, leads or bookings from the business's own system for the same weeks.
  • What sits on those pages. Screenshots on a phone and a desktop, the proof elements on each page, such as reviews, guarantees, credentials and delivery terms, and every form field with its validation.
  • Every campaign destination. An export of the final URLs in ad accounts, links in active emails and partner links, so that none of them ends on a redirect to the homepage after launch.
  • Current speed. Field data for loading and responsiveness on the main templates, so the new ones can be compared.
  • The measurement definitions. Which Key Event counts as the conversion on each template, and where it fires.
  • The search baseline, if addresses will change. The migrations guide lists rankings, organic traffic by page, Search Console performance and the pages with external links.

Then do the research that finds what is quietly doing the persuading. The conversion research methods guide describes recordings that show where visitors look before they decide, polls that ask what nearly stopped them, and support tickets that reveal the questions the page answers. A delivery line beside the basket button or a review count under the product title can be doing work that nobody designed it to do. Findings like these go into the design brief as elements to keep or to improve deliberately, not to tidy away.

Testing key templates before launch

The templates that carry most conversions, often the product page, the plan or pricing page, the basket and the main lead form, are worth testing on their own before the rest of the site moves. The test hub's table of designs names the right one: a split-URL test compares two separately built pages with traffic redirected between them, and it suits redesigns and structural changes too large for in-page edits.

The same table names the two things to watch. Redirect latency adds a delay to whichever arm is redirected, so that arm is tested with a handicap the launched version will not carry. Tracking the redirect needs care, so that the assignment, the landing and the conversion are all recorded for both arms. The testing pitfalls guide adds the most common failure: a redirect that fails on some browsers sends those visitors back to the control, producing a sample ratio mismatch that makes the arms incomparable. Check the split daily, by device and browser.

Run the test as the A/B testing guide describes: a written plan with the primary metric, guardrails, the sample and the reading date; QA of both versions on real devices; the tracking walked in each arm before the test counts; and a single read at the planned sample. Test the new template against the old one as a whole, because that is the decision the business faces. If the new template loses, the research and the recordings for each arm are cheaper places to find out why than a launched site.

Where a new flow has no traffic yet, or the template serves too few visitors for a test, the methods guide's usability sessions on a prototype or a staging version are the fallback. They will not size the effect, but they find the places where people cannot complete the task.

Rolling out against a comparison group

Once the key templates have been tested, the rest of the redesign can move in a way that still leaves a comparison. Three designs are available, in order of strength.

DesignWhat is comparedStrengthWeakness
Staged rollout to a random shareVisitors assigned at random to the new or old site, both running at the same timeControlled: the season and the traffic mix hit both groups equallyTwo sites to run and keep consistent; assignment must be sticky
Holdout after launchA small random group kept on the old experience for a fixed periodMeasures the cumulative effect of everything that changedThe holdout group forgoes any benefit; costly to keep the old site alive
Before-and-after with matched weeksThe weeks after launch against comparable weeks beforePossible on any platformCannot separate the redesign from everything else that changed

The test hub is precise about the first: a staged rollout is not a controlled comparison unless the groups are random and concurrent. A rollout that sends the new site to mobile visitors first, or to one country first, or that switches everyone over on a fixed day, is a sequence of before-and-after comparisons, not a test. To make it a comparison, assign visitors at random, keep each visitor in the same group on every visit, measure both groups with identical tracking, and widen the new site's share only at the gates agreed in advance. A Holdout Group works on the same principle after the switch-over, and the hub's table names its costs: a long duration and a group that does without the benefit.

Where the platform cannot serve two sites at once, which is common when the addresses change, the fallback is a before-and-after comparison read with its limits stated. Choose matched weeks, aligned by weekday and by the holidays that move, and the same weeks a year earlier where the business is seasonal. Compare rates by source and device rather than the overall figure, because a relaunch is often timed with a campaign that changes the mix. Write the prediction, including the range of results that would count as success, before launch.

When addresses do change, the move follows the site migrations guide, which draws on Google's documentation on site moves: every old URL mapped one to one to its equivalent with permanent redirects and no chains, the old sitemap kept available for a period so that the old URLs are recrawled and their redirects discovered, and the redirects kept in place for at least a year, the period the guide says Google documents for signals to transfer. The website migration checklist turns that into a tickable list. The paid side has its own checks, which the guide to Google Ads during a website migration sets out: final URLs updated, the conversion tags and triggers moved to the new templates, and the click identifier tested through every redirect pattern so that conversions can still be credited to the click.

Reading the result fairly

Reading starts with the measurement, not the result. Work through the tracking verification checklist on the live site, confirm that every key event fires once per real action on the new templates, and reconcile a closed day of conversions with the business's own system. A dated entry in the measurement change log marks the launch, with the expected effect written down, so that every later report crossing the date carries the caveat that destinations, templates and possibly definitions changed that day.

Then account for novelty. The testing pitfalls guide's check applies to a relaunch as much as to a test: plot the daily difference across the reading period, split new from returning visitors, and look for the signature of a novelty effect, a strong early lead among returning visitors that shrinks steadily, or of a primacy effect, the same shape reversed. New visitors have no memory of the old site, so their results are the better guide to the long-run effect, and the reading period should run long enough for the returning visitors' trend to flatten.

Then read once, on the date fixed in advance. Evan Miller's essay on how not to run an A/B test, which the testing guides cite for the rule that the sample is fixed in advance and read once, shows how checking repeatedly and stopping on the first good result inflates false positives. A relaunch watched daily by a team that wants it to succeed is under the same pressure, and the defence is the same: decide when to read before there is anything to look at.

Read the guardrails beside the primary metric: order value, returns, support contacts and speed. Read segments only where the plan named them. And keep search separate. If organic visits fell after launch while the share of visitors converting held, the problem is redirects, indexing or lost content, which the traffic drop after migration diagnosis covers. If the share converting fell, the conversion diagnosis applies. A relaunch often causes a little of each, and they need different fixes.

Rollback rules agreed in advance

A rollback decided in the week after launch is decided under pressure, by whoever is most alarmed. Agreed before launch, it is a rule the team can apply calmly. The migrations guide's pre-launch checks already ask for a written rollback plan for when something critical breaks; for a redesign, the plan also covers what happens when the design itself disappoints.

Separate faults from verdicts. A fault, such as a key event that no longer fires, a form that fails on one browser, a campaign URL that redirects to the homepage or a payment step that errors, is fixed at once and never waits for the reading date. A verdict on the design waits for the reading date and the criteria agreed at the gate.

TriggerActionDecided by
Tracking broken or counting differentlyFix immediately; annotate the gap; restart the reading period if neededMeasurement owner
A journey step failing for some visitorsFix immediately as a bug; no testWeb lead
Business figures below the agreed range on the reading date, tracking verifiedRestore the specific elements research points to, or roll the affected templates backNamed decision-maker at the gate
Results inside the agreed rangeContinue the rollout to the next gateNamed decision-maker at the gate
Results above the agreed rangeContinue, and record which elements to keep in future templatesNamed decision-maker at the gate

The redesign diagnosis answers the question most teams ask first: a full rollback is rarely the right response, because it discards whatever the redesign improved and usually reintroduces the problems it was meant to solve. Comparing old and new versions of the pages that carry most conversions, and restoring the specific elements that were lost, is usually faster. Write the rollback rule that way, by template and by element, with the old versions kept available until the final gate has passed.

A tea retailer measures its relaunch (illustrative)

Illustrative example — not a real result

Steepwhorl Teas evaluates a platform move

Steepwhorl Teas, an invented loose-leaf tea retailer, has to leave its old shop platform by the end of the year. Before the build, the team records its best-converting pages: the product pages for its black tea range carry 41% of orders, at a session conversion rate of 2.9% over the previous eight full weeks, against 2.1% for the site as a whole. Recordings and a poll on those pages show that the brewing-guide panel under the price is read by most buyers, so it goes into the brief as an element to keep. The new product template is split-tested against the old one on the black tea range for four weeks, with visitors assigned at random and the split checked daily by browser. It reaches its planned 18,000 visitors per arm, and orders per visitor come out 2.9% for the old template against 3.1% for the new, with an interval running from a small loss to a moderate gain. The team records the result as inconclusive but not harmful and proceeds to the next gate. The rest of the site is rolled out to a random half of visitors for six weeks, with both versions tracked by the same key event and reconciled daily with the order system. Overall orders per visitor are 2.2% on the new site and 2.1% on the old. Returning visitors on the new site start 12% ahead and fall back to level by the fourth week; new visitors are 4% ahead throughout. The decision-maker approves the switch-over, the change log records the date, and the brewing-guide panel is carried into every product template.

Common mistakes

  • Changing platform, URLs, templates and content in one release. Nothing can be attributed; make the one-way door smaller.
  • Starting the design before recording the baseline. Without screenshots, proof inventories and business counts, there is nothing fair to compare with.
  • Calling a phased launch a test. A rollout by device, country or date is a sequence of before-and-after comparisons unless assignment is random and concurrent.
  • Reading the first weeks as the verdict. Returning visitors' novelty and primacy effects fade; read new visitors separately and wait for the reading date.
  • Judging conversion before checking tracking. A rebuilt site that counts differently can show a fall or a rise that never happened.
  • Comparing a launch month with a different season. Match weeks and holidays, and use last year's weeks where the business is seasonal.
  • Mixing search traffic losses with conversion losses. Read visits from search and the share converting separately; their causes differ.
  • Redirecting old pages to the homepage. Map every old address and campaign destination to its equivalent.
  • Deciding rollback in the moment. Write the triggers, actions and decision-makers down before launch.
  • Rolling back everything. Restore the specific elements that were lost, template by template.

Frequently asked questions

Can you A/B test a whole website redesign?

Partly. The templates that carry most conversions can be tested as split-URL tests, with visitors assigned at random to the old or the new version, and the rest of the site can be rolled out to a random share of visitors while the old site keeps serving the others. Both designs measure the net effect of a bundle of changes, not which change made the difference, and both need the same tracking on old and new versions before any result counts. Where the platform cannot serve two versions at once, the fallback is a before-and-after comparison with matched weeks and a prediction written in advance.

How long should we wait before judging a redesign?

Set the reading date before launch and keep to it. Faults such as broken tracking, missing redirects and failing forms are checked on launch day and fixed at once, because they are not a verdict on the design. The design itself is read after full weeks with a stable traffic mix, long enough for returning visitors' reaction to novelty to fade, which the daily trend for returning visitors shows. Reading early, on the first good or bad week, is the same mistake as peeking at a running test.

Why did the new design win in testing and then disappoint after launch?

Usually for one of four reasons. The test was read during a novelty spike among returning visitors; the test covered one template while the launch changed everything else as well; tracking on the new site counts differently from tracking on the old one; or the launch landed in a different season or traffic mix from the test. Check the tracking first, then compare new visitors' results with the test, then look for the parts of the launch that were never tested.

Terms used on this page

Frameworks

Where this page is used

Part of these learning paths

Also related

Sources

  1. How Not To Run an A/B Test — Evan Miller
  2. Site moves with URL changes — Google Search Central

Related