Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

SEO › SEO Measurement

Measuring the Effect of an SEO Change

How to judge whether an SEO change already made worked: comparison pages left unchanged, matched time windows, confounders, and stating the result honestly.
Tutorialadvanced9 min readUpdated

Why simple before-and-after comparisons mislead

A before-and-after comparison attributes every movement in the period to the change. In organic search, plenty of other things move clicks over the same weeks.

What moves organic clicksWhy it distorts a before-and-after comparison
SeasonalityDemand for the queries rises or falls with the time of year, whatever the site does
Search updates and results page changesPositions and click-through shift across the site at once
Demand shocksNews, weather, a competitor's campaign or a trend change search volumes
Other releasesTemplate changes, internal links, redirects and content published in the same weeks
Tracking and reporting changesFilters, property settings and analytics configuration change the numbers, not the traffic
Brand activityAdvertising and press change brand searches and sometimes non-brand behaviour

The fix is a comparison group: pages similar to the changed ones that were not changed. Anything that affects both groups, such as a season, an update or a demand shift, appears in both, and the difference between the groups' changes isolates what is left. When pages were assigned to the groups at random before the change, that is a planned experiment, as in the SEO title tag test design. When the change has already shipped, the comparison group has to be chosen afterwards, which makes the evidence weaker but still far better than a before-and-after line. The SEO measurement hub explains why correlation with annotated causes is the usual standard in SEO reporting and why a comparison between groups is stronger. The academy exercise on testing title changes across a tyre retailer's vehicle pages practises judging a title rewrite that has already shipped against unchanged comparison pages, then designing the randomised test that should have come first.

Before you start

  • Access to Search Console (GSC) for the property, with the ability to filter the performance report by page and to export. The Search Console tutorial covers both.
  • The exact change: what was changed, on which URLs, and the date it went live. The release notes, content management history or ticket are the sources.
  • The annotations or change log for the site, so other releases and events in the period can be found.
  • A list of similar pages of the same type that were not changed, ideally with similar traffic.
  • A spreadsheet or notebook to hold weekly data for both groups and the calculations.
Change typeTypical measureComparison pages to look for
Title or description rewriteClick-through and clicks per pageSame template, another range or section
Content refreshImpressions, position and clicks per pageSimilar articles not refreshed
Internal linking changeImpressions and position for linked pagesSimilar pages that did not gain links
Consolidation of overlapping pagesClicks to the surviving page against the combined clicks beforeSimilar topics where pages were not merged
Technical fix on a templateImpressions and clicks for the templateA similar template not yet fixed

The steps

  1. Record the change, the pages and the date

    Write down what changed, the full list of changed URLs and the date and time each went live. If the change rolled out over several days, record the date the last page changed. Then confirm that the search engine has seen it: URL Inspection in Search Console shows, for a single URL, whether it is indexed and when it was last crawled. Inspect a sample of changed pages and note the earliest date by which they had all been crawled after the change. The effect cannot start before that date, so the post-change window starts from it, not from the release.

  2. Choose the measure

    Pick one primary measure that the change should move, and decide it before looking at the results. A title rewrite should move click-through rate (CTR) and clicks at a similar position. A content refresh or internal linking change should move impressions and position first. Add one or two guardrails: for a title change, average position, to check that click-through, not ranking, changed; for any change, conversions from the pages, to check that extra clicks are useful. Use clicks per page or totals per group rather than individual queries, which are too noisy.

  3. Pick comparison pages left unchanged

    Choose pages of the same type, serving similar intent and with similar traffic, that received no changes in the period. They act as a Holdout Group, although one chosen afterwards rather than assigned at random. Check that the two groups moved together before the change: plot their weekly clicks over the pre-change weeks, and if the lines do not move in parallel, find a better comparison or note the weakness. Exclude any comparison page that changed for another reason, gained or lost internal links from the changed pages, or shares queries with them.

  4. Set matched windows that avoid known events

    Choose a pre-change window and a post-change window of the same length, starting on the same day of the week, so that weekday patterns cancel. Several weeks each is usual for pages with steady traffic; fewer pages or less traffic need longer. Leave out the weeks between the release and the crawl date. Check the change log and known search updates, and move or shorten the windows to avoid major events, especially those that might affect the two groups differently. If the windows cross a seasonal shift (Seasonality), confirm that the comparison pages share the same seasonal pattern.

  5. Pull the data per page group

    In the Search Console performance report, filter by page to each group, using a URL pattern if the groups follow one or a list of URLs otherwise, and export clicks, impressions, click-through and average position by date for both windows. Keep the country and device filters identical for both groups, and exclude brand queries if brand searches land on these pages. Aggregate the daily rows to weeks for each group. The page dimension aggregates data by page, which is the right view for page groups; keep the same view throughout.

  6. Compare changed pages with unchanged ones

    For each group, calculate the change in the measure from the pre-change window to the post-change window. The estimate of the effect is the difference between the two changes: how much more, or less, the changed group moved than the comparison group. Equivalently, compute the ratio of the changed group's measure to the comparison group's each week, and compare the ratio after the change with the ratio before. Read the guardrails in the same way. If position moved in the changed group but not in the comparison group, a title change may have affected ranking as well as click-through, and the result should say so.

  7. Check for confounders

    Go through the list in the first section and look for anything that affected one group and not the other during the windows: a release touching only one range, a promotion on one product line, a stock problem, a news event, a tracking change. Check whether the changed pages actually show the change in search results: for titles, sample the results, because the title rewrite troubleshooting page explains that the title shown is not always the one written. Check each post-change week separately; an effect concentrated in one week often has a cause other than the change.

  8. State the result and its uncertainty

    Measure how much the ratio between the groups wandered in the pre-change weeks, when nothing was changed. That range is the noise. If the post-change ratio sits outside it, week after week, the change probably had an effect, and its size lies somewhere in a range set by that noise, not at a single figure. If it sits inside, the honest result is that no effect could be distinguished, which is not the same as no effect. A useful check is to run the same calculation on a date in the pre-change period when nothing happened: if it shows an "effect" as large as the real one, the method cannot separate signal from noise here. Write the result in one sentence with the range, the comparison used and the main caveat.

  9. Record it in the change log

    Add the result to the change log beside the original change entry: the measure, the groups, the windows, the estimate and range, the confounders checked and the decision taken, whether to roll out further, revert or test properly. The SEO KPIs and reporting guide treats annotations as part of every report; a measured result is the most useful annotation there is. Over time the log becomes the site's own evidence of which kinds of change work, which feeds the next round of the experimentation loop.

When the next change is still being planned, design it as a test instead: split the pages at random and hold a group back. The content refresh test and the internal linking test set out those designs, and the incrementality testing guide explains the logic of measuring against a group left untouched, which applies here too.

A title rewrite measured (illustrative)

Illustrative example — not a real result

A craft-supplies store measures a title rewrite on one product range

Crumbquill Craft Supplies, an invented online store, rewrote the title tags on its 140 knitting yarn product pages to lead with fibre and weight. Nobody planned a test. URL Inspection on 20 sampled pages showed all had been crawled within 12 days of the release, so the post-change window starts on day 14. The comparison group is the 120 embroidery thread product pages: same template, similar traffic, untouched in the period, and their weekly clicks moved in parallel with the yarn pages over the previous 12 weeks. Both windows are six weeks, Monday to Sunday, non-brand queries, all devices, in the store's main country.

Group and windowImpressionsClicksClick-throughAverage position
Yarn (changed), before410,0009,8402.4%7.9
Yarn (changed), after395,00011,0602.8%7.8
Thread (unchanged), before300,0007,5002.5%8.3
Thread (unchanged), after288,0007,0602.45%8.4

Yarn clicks rose by about 12% while thread clicks fell by about 6%, as impressions fell in both with the season. The ratio of yarn to thread clicks was 1.31 before and 1.57 after, about 19% higher. In the 12 pre-change weeks the weekly ratio ranged from 1.22 to 1.40; after the change every week sat between 1.50 and 1.62. Against that noise, the effect lies roughly between 12% and 28%. Average position barely moved in either group. Checks: a site-wide sale in post-change week three affected both groups equally; an announced search update fell in week five, and that week's ratio matched the others. Sampling the results found the new title shown for 16 of 20 yarn pages. The log entry reads: "Yarn title rewrite: clicks about 12% to 28% higher than the thread range would predict, position unchanged; roll the pattern out to the thread range and measure it against the needlework kits range."

Common mistakes

  • Comparing before with after and nothing else. Seasonality, updates and demand get credited to the change.
  • Starting the clock at the release date. The effect begins after the pages are crawled; check with URL Inspection.
  • Choosing the measure after seeing the data. Some measure will always have moved; decide first.
  • A comparison group that did not move with the changed one before the change. The difference after the change then means little.
  • Windows of different lengths or weekdays. Weekday patterns become a false effect.
  • Reading single queries. Query-level clicks are too noisy; use page groups.
  • Reporting one number. State the range implied by the pre-change noise and the main caveat.
  • Not recording the result. The next debate about the same kind of change starts from nothing.

Frequently asked questions

How long should I wait before measuring an SEO change?

Until the search engine has crawled the changed pages and the effect has had time to settle, and then for a post-change window long enough to cover normal week-to-week variation. Check the last crawl date of a sample of changed pages with URL Inspection before starting the clock. The window should match the pre-change window in length and in days of the week, and should not run into a seasonal shift that affects the two page groups differently.

What if there are no comparable pages that were left unchanged?

Then the evidence is weaker, and the result should say so. Options, in order of preference: use pages of the same type in another section or market that were not changed; use the same pages' history over the same weeks of the previous year as a seasonal comparison; or compare against queries that the change could not plausibly affect. If none exists, report the before-and-after movement with every confounder listed, and plan the next change as a test with a group held back.

Is this the same as an SEO split test?

No. A split test assigns similar pages at random to a changed group and an unchanged group before the change, which makes the groups comparable by design. Measuring after the fact uses whatever unchanged pages exist, so the comparison group is chosen rather than randomised, and the result is less certain. The method here is for changes already shipped; the SEO experiment designs are for changes still to come.

Terms used on this page

Frameworks

Where this page is used

Part of these learning paths

Sources

  1. Performance report (Search results) — Google Search Console Help
  2. URL Inspection Tool — Google Search Console Help

Related