Definition
A before-and-after comparison takes one measure, such as conversions, organic clicks, time per task or orders, reads it over a period before a change and a period after it, and treats the difference as the effect of the change. No comparison group exists. Every page, visitor, region or customer saw the change, so nothing shows what the measure would have done without it. The guides also call it a before-and-after reading.
The guides use it, for example, when traffic cannot support a test or the change is cheap to reverse (the conversion work guide), when a feature is switched on and read afterwards (the enhanced conversions guide) or when a channel is being stopped (the channel stop guide). They rank it rather than forbid it, and ask what it can honestly claim.
What it can and cannot show
It shows that a measure moved around the time of a change. It cannot show that the change caused the movement: whatever else happened in those weeks is counted as the change's work. The tutorial on measuring the effect of an SEO change tabulates what else moves organic clicks in the same weeks. A promotion, a price change or a tracking fix can move the same measure.
In the hierarchy of evidence it sits above a platform's own attributed figure and below a matched comparison, a holdout or a randomised test. So the guides log a result beside the tests with its strength marked, not as proof.
Reading one with care
Where a comparison has to be before and after, the guides want the reading fixed ahead of launch. The conversion work guide lists what to write down first, such as the prediction and the reading date, and how to choose the comparison period. Units that did not receive the change, such as pages of the same type, shops or regions, become a comparison group for the same period: stronger than a plain line, weaker than groups assigned at random.
In practice: SEO changes, AI tools, tracking features and channel pauses
The SEO tutorial opens with the failure: a plain before-and-after line credits the change with everything that moved in the period, so it measures against comparable pages left alone. The guide to AI's impact on team output calls it the weakest design, since the rest of the business was changing too, and prefers splitting the work or holding a group out.
The enhanced conversions guide makes it the measurement check: count and rate are noted for a steady stretch before the feature is switched on, then compared once it has settled for about as long, allowing for anything else that changed. The guide to stopping a paid channel reads a pause the same way and adds that the holdout run before the exit remains stronger evidence, because a reading after the stop leaves the season and any other change of those weeks as rival explanations.
An online ceramics seller checks a redesign against pages it did not change
Manotul Ceramics, an invented online ceramics seller, ships a new layout to 60 of its 100 product pages, the ones on its newer template. In the 2 weeks after launch, conversion on those pages rises from 2.1% to 2.6%, and the team is ready to call the redesign a win. A colleague notes that a newsletter went out and a gift-buying season began in the same weeks. The other 40 pages kept the old layout, and their conversion rose from 2.0% to 2.4%. So 0.4 of the 0.5-point rise happened without the redesign, leaving about 0.1 points, a fifth of the gain, for it. Because the pages were not chosen at random, the team records the result as consistent with a small gain, names the newsletter and the season as rival explanations, and plans a randomised split for the next layout change.
Related terms
A BaselineAn estimate of what a metric would have done without a given change, campaign or promotion, against which the effect is measured. is what the measure would have done without the change; a before-and-after comparison stands in for it with the period before. SeasonalityPredictable, repeating changes in demand tied to the calendar, such as months, holidays, weekdays or hours. is one of the things mixed into the result. A Holdout GroupA randomly chosen part of an audience kept from exposure to measure what exposure caused. is the group kept away from the change that a plain before-and-after comparison lacks. An A/B TestControlled comparison of two variants. splits visitors at random so that both versions run in the same weeks, and a Staged RolloutReleasing a change to a growing share of visitors, pages or regions in steps and reading each before the next; a test only if groups are random and concurrent. sits between the two: it is a controlled comparison only when its groups are random and run side by side.
Common confusions
- A rise after launch means the change worked. It may be the season, a promotion or a tracking change. The defensible claim is modest: the result fits the prediction, and the rival explanations are named beside it.
- A rollout by country or device is a test. A launch to everyone on a fixed date is a before-and-after reading; a launch to one country or one device type is at best a comparison with a group that was not assigned at random.
