Why this matters
Free filtering is good for shoppers and ruinous for crawling: every filter multiplies the URL count until the crawler spends its time on near-identical lists. Blocking every filter is just as costly, because it hides the refinements people search for. The faceted navigation guide sets out the three tiers; this exercise makes you draw the lines on real-looking data and turn them into rules rather than a list of URLs.
The shop and its filter combinations
Bobbinworth Fabrics is an invented online fabric and haberdashery shop with dressmaking, upholstery and quilting categories. Shoppers combine filters for fibre, colour, weight and pattern, and every selection creates a crawlable link.
Bobbinworth Fabrics: crawl symptoms and sample combinations
Crawl stats: about 78% of crawl requests go to filter URLs; the crawler knows about 410,000 URLs for a catalogue of about 3,800 products; new products take around three weeks to be crawled.
| Combination | Example query | Monthly searches | Products | Note |
|---|---|---|---|---|
| Linen (fibre) | linen fabric | 9,900 | 140 | Distinct range |
| Linen + heavy weight | heavyweight linen fabric | 880 | 22 | Distinct sub-range |
| Cotton + floral | floral cotton fabric | 2,400 | 310 | Distinct |
| Quilting + gingham | gingham fabric | 1,900 | 18 | Distinct |
| Viscose + floral | floral viscose fabric | 1,000 | 120 | Distinct |
| Linen + navy | navy linen fabric | 260 | 9 | Small subset of linen |
| Green (colour only) | green fabric | 320 | 600 | Mixed fibres and uses |
| Price under £10 a metre | cheap fabric | 1,600 | 450 | Changes with every price update |
| Silk + floral + light weight | none | 0 | 4 | Deep combination |
| Any filter + sort by price or 96 per page | none | 0 | varies | View options |
Index, canonicalise or keep out of the crawl
Ask four questions of each combination: is there demand for it on the SERPThe page of results a search engine shows., does it hold enough products to be a useful page, is it distinct from its parent, and can the template give it its own title, heading and introduction? A view that passes all four is a landing page. A view that fails on demand but is a fair subset of a landing page can point to it with a Canonical TagTells search engines the preferred URL., which is a hint the engine may ignore if the content differs too much. Everything else should not be crawled at all. Remember the trap: a robots rule stops a crawler from ever seeing a noindex or canonical on the blocked URL, so pick one control per tier rather than stacking them. The category and product pages guide supplies what an indexable landing page must contain.
Task
- Classify each combination in the table as a landing page, a canonicalised view or kept out of the crawl, with the reason.
- Turn the classifications into general rules, with thresholds, that would also cover combinations not in the table.
- Specify URL format, canonical, robots, internal linking and sitemap treatment for each tier.
- Decide what happens to the colour-only and price views, which do not fit the tiers neatly.
- Write the rollout plan: tests before launch, the first category to change, and what to monitor.
Worked answer
Bobbinworth: the rules for the developer
Landing pages: a single fibre or pattern within a category, and fibre plus one weight or pattern where the combination has at least 500 searches a month and at least 15 products. From the table: linen, heavyweight linen, floral cotton, gingham and floral viscose.
Canonicalised views: any other single or two-filter combination, such as navy linen, kept as parameters in a fixed alphabetical order and canonicalised to the nearest landing page.
Kept out of the crawl: three or more filters, sort, per-page and price parameters, rendered as buttons rather than links, with robots rules for those parameter patterns.
Exceptions: colour-only views canonicalise to the category; a curated "fabrics under £10" collection page replaces the price filter if the shop wants that query.
Landing pages get a clean path such as a fibre folder under the category, a self-referencing canonical, a written introduction, a title naming the range as shoppers do, a place in the sitemap and links from the category page and navigation. Their pagination stays self-canonical with plain links, so products on later pages are still found.
Canonicalised views stay usable for shoppers, but their links are not promoted in navigation and they never appear in the sitemap. Navy linen is a fair subset of linen, so the canonical is honest; if the products differed sharply, a noindex would be the more truthful control.
Kept out of the crawl. Deep combinations and view options are not links the crawler can follow, so it never finds them; the robots rules catch any that leak through external links. Empty results return a noindex message rather than a soft not-found.
The awkward cases. Colour alone has some demand but mixes every fibre and use, so it is not distinct enough to be a page. The price query has real demand, but a filter whose contents change with every price update is a poor page; a curated collection that the team maintains is better, or nothing.
Rollout. List the landing-page URLs and confirm each stays indexable and allowed under the new rules. Crawl staging and compare the indexable set with the intended set; the difference is the bug list. Change dressmaking first, then watch the indexing report by reason, crawl requests by URL pattern and the time from publishing a product to its first crawl. Keep the old rules ready to restore. Success means Crawl BudgetHow much a search engine crawls a site. moving from parameters to products, and the new landing pages appearing for their queries.
The same tiering logic returns at scale elsewhere: language editions governed by the hreflang guide, and hundreds of near-identical branch pages covered by the guide to location pages and listings at scale.
Common mistakes
- Indexing every value of every filter. Thousands of thin pages compete with the categories.
- Blocking every filter. The refinements people search for never get a page.
- Robots rules on URLs that carry a noindex. The crawler never sees the noindex.
- Canonicals that lie. A view with different products pointed at the category is ignored.
- Rolling out everywhere at once. A rule that catches the landing pages removes revenue pages for weeks.
Test your judgement in the indexing at scale checkpoint, or revisit the intermediate SEO level for the crawling and indexing controls.
