Why simple before-and-after comparisons mislead
A before-and-after comparisonA measure read before and after a change with no comparison group, so the season and everything else that moved in those weeks is credited to the change. attributes every movement in the period to the change. In organic search, plenty of other things move clicks over the same weeks.
| What moves organic clicks | Why it distorts a before-and-after comparison |
|---|---|
| Seasonality | Demand for the queries rises or falls with the time of year, whatever the site does |
| Search updates and results page changes | Positions and click-through shift across the site at once |
| Demand shocks | News, weather, a competitor's campaign or a trend change search volumes |
| Other releases | Template changes, internal links, redirects and content published in the same weeks |
| Tracking and reporting changes | Filters, property settings and analytics configuration change the numbers, not the traffic |
| Brand activity | Advertising and press change brand searches and sometimes non-brand behaviour |
The fix is a comparison group: pages similar to the changed ones that were not changed. Anything that affects both groups, such as a season, an update or a demand shift, appears in both, and the difference between the groups' changes isolates what is left. When pages were assigned to the groups at random before the change, that is a planned experiment, as in the SEO title tag test design. When the change has already shipped, the comparison group has to be chosen afterwards, which makes the evidence weaker but still far better than a before-and-after line. The SEO measurement hub explains why correlation with annotated causes is the usual standard in SEO reporting and why a comparison between groups is stronger. The academy exercise on testing title changes across a tyre retailer's vehicle pages practises judging a title rewrite that has already shipped against unchanged comparison pages, then designing the randomised test that should have come first.
Before you start
- Access to Search Console (GSCGoogle's search performance and indexing tool.) for the property, with the ability to filter the performance report by page and to export. The Search Console tutorial covers both.
- The exact change: what was changed, on which URLs, and the date it went live. The release notes, content management history or ticket are the sources.
- The annotations or change log for the site, so other releases and events in the period can be found.
- A list of similar pages of the same type that were not changed, ideally with similar traffic.
- A spreadsheet or notebook to hold weekly data for both groups and the calculations.
| Change type | Typical measure | Comparison pages to look for |
|---|---|---|
| Title or description rewrite | Click-through and clicks per page | Same template, another range or section |
| Content refresh | Impressions, position and clicks per page | Similar articles not refreshed |
| Internal linking change | Impressions and position for linked pages | Similar pages that did not gain links |
| Consolidation of overlapping pages | Clicks to the surviving page against the combined clicks before | Similar topics where pages were not merged |
| Technical fix on a template | Impressions and clicks for the template | A similar template not yet fixed |
The steps
Record the change, the pages and the date
Write down what changed, the full list of changed URLs and the date and time each went live. If the change rolled out over several days, record the date the last page changed. Then confirm that the search engine has seen it: URL Inspection in Search Console shows, for a single URL, whether it is indexed and when it was last crawled. Inspect a sample of changed pages and note the earliest date by which they had all been crawled after the change. The effect cannot start before that date, so the post-change window starts from it, not from the release.
Choose the measure
Pick one primary measure that the change should move, and decide it before looking at the results. A title rewrite should move click-through rate (CTRShare of impressions that result in a click.) and clicks at a similar position. A content refresh or internal linking change should move impressions and position first. Add one or two guardrails: for a title change, average positionMean ranking of a site's topmost result across the impressions it received., to check that click-through, not ranking, changed; for any change, conversions from the pages, to check that extra clicks are useful. Use clicks per page or totals per group rather than individual queries, which are too noisy.
Pick comparison pages left unchanged
Choose pages of the same type, serving similar intent and with similar traffic, that received no changes in the period. They act as a Holdout GroupA randomly chosen part of an audience kept from exposure to measure what exposure caused., although one chosen afterwards rather than assigned at random. Check that the two groups moved together before the change: plot their weekly clicks over the pre-change weeks, and if the lines do not move in parallel, find a better comparison or note the weakness. Exclude any comparison page that changed for another reason, gained or lost internal links from the changed pages, or shares queries with them.
Set matched windows that avoid known events
Choose a pre-change window and a post-change window of the same length, starting on the same day of the week, so that weekday patterns cancel. Several weeks each is usual for pages with steady traffic; fewer pages or less traffic need longer. Leave out the weeks between the release and the crawl date. Check the change log and known search updates, and move or shorten the windows to avoid major events, especially those that might affect the two groups differently. If the windows cross a seasonal shift (SeasonalityPredictable, repeating changes in demand tied to the calendar, such as months, holidays, weekdays or hours.), confirm that the comparison pages share the same seasonal pattern.
Pull the data per page group
In the Search Console performance report, filter by page to each group, using a URL pattern if the groups follow one or a list of URLs otherwise, and export clicks, impressions, click-through and average position by date for both windows. Keep the country and device filters identical for both groups, and exclude brand queries if brand searches land on these pages. Aggregate the daily rows to weeks for each group. The page dimension aggregates data by page, which is the right view for page groups; keep the same view throughout.
Compare changed pages with unchanged ones
For each group, calculate the change in the measure from the pre-change window to the post-change window. The estimate of the effect is the difference between the two changes: how much more, or less, the changed group moved than the comparison group. Equivalently, compute the ratio of the changed group's measure to the comparison group's each week, and compare the ratio after the change with the ratio before. Read the guardrails in the same way. If position moved in the changed group but not in the comparison group, a title change may have affected ranking as well as click-through, and the result should say so.
Check for confounders
Go through the list in the first section and look for anything that affected one group and not the other during the windows: a release touching only one range, a promotion on one product line, a stock problem, a news event, a tracking change. Check whether the changed pages actually show the change in search results: for titles, sample the results, because the title rewrite troubleshooting page explains that the title shown is not always the one written. Check each post-change week separately; an effect concentrated in one week often has a cause other than the change.
State the result and its uncertainty
Measure how much the ratio between the groups wandered in the pre-change weeks, when nothing was changed. That range is the noise. If the post-change ratio sits outside it, week after week, the change probably had an effect, and its size lies somewhere in a range set by that noise, not at a single figure. If it sits inside, the honest result is that no effect could be distinguished, which is not the same as no effect. A useful check is to run the same calculation on a date in the pre-change period when nothing happened: if it shows an "effect" as large as the real one, the method cannot separate signal from noise here. Write the result in one sentence with the range, the comparison used and the main caveat.
Record it in the change log
Add the result to the change log beside the original change entry: the measure, the groups, the windows, the estimate and range, the confounders checked and the decision taken, whether to roll out further, revert or test properly. The SEO KPIs and reporting guide treats annotations as part of every report; a measured result is the most useful annotation there is. Over time the log becomes the site's own evidence of which kinds of change work, which feeds the next round of the experimentation loop.
When the next change is still being planned, design it as a test instead: split the pages at random and hold a group back. The content refresh test and the internal linking test set out those designs, and the incrementality testing guide explains the logic of measuring against a group left untouched, which applies here too.
A title rewrite measured (illustrative)
A craft-supplies store measures a title rewrite on one product range
Crumbquill Craft Supplies, an invented online store, rewrote the title tags on its 140 knitting yarn product pages to lead with fibre and weight. Nobody planned a test. URL Inspection on 20 sampled pages showed all had been crawled within 12 days of the release, so the post-change window starts on day 14. The comparison group is the 120 embroidery thread product pages: same template, similar traffic, untouched in the period, and their weekly clicks moved in parallel with the yarn pages over the previous 12 weeks. Both windows are six weeks, Monday to Sunday, non-brand queries, all devices, in the store's main country.
| Group and window | Impressions | Clicks | Click-through | Average position |
|---|---|---|---|---|
| Yarn (changed), before | 410,000 | 9,840 | 2.4% | 7.9 |
| Yarn (changed), after | 395,000 | 11,060 | 2.8% | 7.8 |
| Thread (unchanged), before | 300,000 | 7,500 | 2.5% | 8.3 |
| Thread (unchanged), after | 288,000 | 7,060 | 2.45% | 8.4 |
Yarn clicks rose by about 12% while thread clicks fell by about 6%, as impressions fell in both with the season. The ratio of yarn to thread clicks was 1.31 before and 1.57 after, about 19% higher. In the 12 pre-change weeks the weekly ratio ranged from 1.22 to 1.40; after the change every week sat between 1.50 and 1.62. Against that noise, the effect lies roughly between 12% and 28%. Average position barely moved in either group. Checks: a site-wide sale in post-change week three affected both groups equally; an announced search update fell in week five, and that week's ratio matched the others. Sampling the results found the new title shown for 16 of 20 yarn pages. The log entry reads: "Yarn title rewrite: clicks about 12% to 28% higher than the thread range would predict, position unchanged; roll the pattern out to the thread range and measure it against the needlework kits range."
Common mistakes
- Comparing before with after and nothing else. Seasonality, updates and demand get credited to the change.
- Starting the clock at the release date. The effect begins after the pages are crawled; check with URL Inspection.
- Choosing the measure after seeing the data. Some measure will always have moved; decide first.
- A comparison group that did not move with the changed one before the change. The difference after the change then means little.
- Windows of different lengths or weekdays. Weekday patterns become a false effect.
- Reading single queries. Query-level clicks are too noisy; use page groups.
- Reporting one number. State the range implied by the pre-change noise and the main caveat.
- Not recording the result. The next debate about the same kind of change starts from nothing.
