Why this matters
The advanced SEO level forecast the value of SEO work and set the operating rhythm that delivers it. Sooner or later that rhythm produces a claim that something worked, and the claim decides where the next quarter's effort goes. The expert's job is the evidence standard. The tutorial on measuring an SEO change judges a change already shipped against comparable pages left alone; the SEO title tag test designs the next change so the groups are comparable by design. One chooses its comparison afterwards, the other randomises it beforehand, and a team that knows which it is doing knows how much weight its result can bear.
The tyre retailer and its title rewrite
Veskaran Tyres is an invented online tyre retailer that fits tyres at a partner garage or at the customer's address. Every vehicle model has a page built from one template, grouped as cars, vans, and pickups and off-roaders. Last spring the marketing team rewrote the van page titles to lead with fitting at the customer's address. The head of marketing now wants the same idea on every car page, on the strength of the report below.
Veskaran Tyres: the van rewrite, the report and the exports
Pages: 180 van models, 140 pickup and off-roader models, 2,400 car models.
The report: "Van page clicks up 31% after the new titles", from all queries: 16,900 clicks in the eight weeks before the release against 22,140 in the nine weeks after it.
Release notes, same day: van titles changed from "[Make] [Model] Van Tyres | Veskaran Tyres" to "[Make] [Model] Van Tyres Fitted at Your Address | Veskaran"; a homepage "Van tyres" banner added, linking to the 20 most visited van pages. A radio campaign for the address fitting service ran for four weeks from the week after the release.
URL Inspection on 20 sampled van pages: all recrawled within 11 days of the release. Sampled in search results afterwards: 13 showed the new title, 7 a rewritten one.
Search Console exports, non-brand queries, both groups, six weeks before the release and six weeks starting on day 14 after it:
| Group and window | Clicks | Impressions | Click-through | Average position |
|---|---|---|---|---|
| Vans (changed), before | 12,600 | 504,000 | 2.50% | 8.6 |
| Vans (changed), after | 15,400 | 579,600 | 2.66% | 7.9 |
| Pickups and off-roaders (unchanged), before | 9,800 | 392,000 | 2.50% | 8.4 |
| Pickups and off-roaders (unchanged), after | 11,270 | 450,800 | 2.50% | 8.4 |
Weekly ratio of van clicks to pickup and off-roader clicks over the 12 weeks before the release: between 1.18 and 1.39, the two groups moving together. After the change: between 1.31 and 1.42. The 20 banner-linked van pages account for 2,100 of the van group's extra clicks between the matched windows.
For the car pages: 1,640 have steady organic traffic, 60 of them due for redesign this summer. Car page clicks rise every October as winter tyre searches arrive. The proposed pattern: "[Make] [Model] Tyres: Prices, Sizes and Fitting at Home | Veskaran".
Comparison pages and page groups
A comparison group earns its place only if it moved with the changed pages before the change. Pickups and off-roaders share the template, the commercial buyer and the spring rise in demand, so SeasonalityPredictable, repeating changes in demand tied to the calendar, such as months, holidays, weekdays or hours. appears in both groups and cancels out of the difference. Chosen afterwards, they act as a Holdout GroupA randomly chosen part of an audience kept from exposure to measure what exposure caused. nobody planned, which makes the evidence weaker than a random split, and the result has to say so. Matched windows start from the recrawl, not the release, and brand queries are excluded because the radio campaign moved them. The noise is how far the ratio between groups wandered before anything changed. For the car pages none of this has to be improvised: pages are split at random before the change, so the groups are comparable by design.
Task
- Say what the report actually measured and list what is wrong with it.
- Check the rebuilt comparison: windows, start date, query filter and whether the comparison group is fit for use.
- Estimate the effect, compare it with the noise and read the position and rewrite guardrails.
- Name the confounders and write the result in one sentence for the change log.
- Design the car page test: pages in scope, randomisation, metric, guardrails, duration, what is frozen and the decision rule, recorded in the experiment hypothesis template.
- Decide where the question raised by the homepage banner goes next, and why it is not tested on the same pages at the same time.
Worked answer
What the report measured. Spring demand, a radio campaign and a homepage banner, as much as titles. Its windows differ in length, it starts at the release rather than the recrawl, it counts brand queries in the weeks the radio campaign ran and it has no comparison. Per week, its own figures show a far smaller rise than the headline.
The rebuilt comparison. The exports fix each of those faults. The groups moved together throughout the weeks before the release, so the comparison is usable, with the caveat that it was chosen rather than randomised.
Veskaran Tyres: the arithmetic
The report per week: 2,113 clicks before, 2,460 after, a rise of 16%, not 31%.
Matched windows: vans up 22% (12,600 to 15,400), pickups and off-roaders up 15% (9,800 to 11,270). Ratio of vans to pickups and off-roaders: 1.29 before, 1.37 after, about 6% higher. The post-change weekly ratios of 1.31 to 1.42 sit mostly inside the 1.18 to 1.39 the ratio wandered before the change. The same calculation on a date six weeks before the release, when nothing changed, shows an "effect" of 5%.
Guardrails: van position improved from 8.6 to 7.9 while the comparison held at 8.4; 7 of 20 sampled van titles were rewritten. Of the 2,800 extra van clicks, 2,100 went to the 20 pages the banner linked.
Confounders and the result. Position improved for vans and not for the comparison, so the Title TagThe HTML title element that names a page and usually supplies its title link in search results. change, or something else in the same release, moved ranking as well as click-through, and the click gain cannot be read as a title effect; most of the gain sits on the pages the banner linked, so new internal links are the likelier cause. On a fair share of sampled pages, searchers saw a rewritten title rather than the pattern, as the page on titles rewritten in search results explains. Brand queries are excluded, but the radio campaign may have nudged non-brand behaviour too. The change log entry reads: "Van title rewrite: van clicks rose slightly more than pickups and off-roaders, within the range the ratio wandered before the change and about as large as the effect a placebo date shows; position improved for vans only, and most of the gain is on pages the same release linked from the homepage. No title effect can be distinguished. Test the pattern on car pages with a random split before any rollout."
The car page test. The design follows the title tag test, with Veskaran's numbers fixed before launch:
Veskaran Tyres: the car page title test plan
| Element | Choice |
|---|---|
| Pages in scope | 1,580 car pages with steady traffic; the 60 due for redesign excluded |
| Randomisation | By page, within four traffic bands: 790 variant, 790 control |
| Pre-period | Eight weeks, both groups unchanged; the weekly ratio between them wandered from 0.97 to 1.03 |
| Primary metric | Non-brand organic clicks per page, variant change relative to control |
| Guardrails | Average position; share of sampled variant titles rewritten; tyre orders and fitting bookings per page |
| Smallest change worth rolling out | 4% more clicks per page, set with the CTR calculator at the template's average position |
| Duration | Post-period starts when 18 of 20 sampled variant pages show the new title, runs six weeks and ends before October |
| Frozen on both groups | Content, internal links, banners, structured data and other template work |
| Result at the planned read | Decision |
|---|---|
| Variant clicks at least 4% above control and outside the pre-period noise; position and orders hold | Roll out to every car page |
| Clicks rise but position moved too | Inconclusive; find out what changed ranking first |
| More than half of sampled variant titles rewritten | Put the old titles back on the variant pages and find out why the search engine replaced them |
| Variant clicks below control beyond the pre-period noise | Return the variant pages to the current pattern |
| Variant and control within the pre-period noise | Leave car titles as they are; the next test tries a stronger benefit in the title |
The smallest change worth acting on comes from the CTR calculator for organic search: the CTRShare of impressions that result in a click. gain at the template's position, turned into clicks and then orders, that would justify the rollout. In the pre-period the ratio between the groups wandered by less than the smallest change worth acting on, so a change of that size could stand out from it, though only just; had it not, the answer would be a bolder pattern or more pages.
The banner question. The van evidence suggests internal links did the work, which is a hypothesis, not a result. It goes to the internal linking test, with its own random target groups, on the van template once the car test has been read. Run on car pages during the title test, new links would land in both groups unevenly and the title read would measure the bundle.
Common mistakes
- Comparing windows of different lengths. Extra weeks become a false effect.
- Starting the clock at the release. The new titles were not in the index yet.
- Counting brand queries during a campaign. The radio advertising is credited to the titles.
- Reading clicks without position. If the changed pages also moved up, the extra clicks may have nothing to do with the title.
- Reporting one number. The pre-period noise sets the range; a placebo date tests the method.
- Testing two changes on the same pages at once. Neither result can then be read.
The module checkpoint puts these calls to a guitar shop and adds an internal linking design with source-page guardrails.
