Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

SEO › Advanced

Deciding Which Filter Pages to Index for a Fabric Shop

An index-control exercise for an invented fabric shop whose free filter combinations are consuming the crawl. You sort combinations into landing pages, canonicalised views and URLs kept out of the crawl, using demand, product counts and distinctness, then write rules a developer can build and a staged rollout that can be reversed.
ExerciseAdvanced levelModule 2: Indexing at scale5 minUpdated

Why this matters

Free filtering is good for shoppers and ruinous for crawling: every filter multiplies the URL count until the crawler spends its time on near-identical lists. Blocking every filter is just as costly, because it hides the refinements people search for. The faceted navigation guide sets out the three tiers; this exercise makes you draw the lines on real-looking data and turn them into rules rather than a list of URLs.

The shop and its filter combinations

Bobbinworth Fabrics is an invented online fabric and haberdashery shop with dressmaking, upholstery and quilting categories. Shoppers combine filters for fibre, colour, weight and pattern, and every selection creates a crawlable link.

Illustrative example — not a real result

Bobbinworth Fabrics: crawl symptoms and sample combinations

Crawl stats: about 78% of crawl requests go to filter URLs; the crawler knows about 410,000 URLs for a catalogue of about 3,800 products; new products take around three weeks to be crawled.

CombinationExample queryMonthly searchesProductsNote
Linen (fibre)linen fabric9,900140Distinct range
Linen + heavy weightheavyweight linen fabric88022Distinct sub-range
Cotton + floralfloral cotton fabric2,400310Distinct
Quilting + ginghamgingham fabric1,90018Distinct
Viscose + floralfloral viscose fabric1,000120Distinct
Linen + navynavy linen fabric2609Small subset of linen
Green (colour only)green fabric320600Mixed fibres and uses
Price under £10 a metrecheap fabric1,600450Changes with every price update
Silk + floral + light weightnone04Deep combination
Any filter + sort by price or 96 per pagenone0variesView options

Index, canonicalise or keep out of the crawl

Ask four questions of each combination: is there demand for it on the SERP, does it hold enough products to be a useful page, is it distinct from its parent, and can the template give it its own title, heading and introduction? A view that passes all four is a landing page. A view that fails on demand but is a fair subset of a landing page can point to it with a Canonical Tag, which is a hint the engine may ignore if the content differs too much. Everything else should not be crawled at all. Remember the trap: a robots rule stops a crawler from ever seeing a noindex or canonical on the blocked URL, so pick one control per tier rather than stacking them. The category and product pages guide supplies what an indexable landing page must contain.

Task

  1. Classify each combination in the table as a landing page, a canonicalised view or kept out of the crawl, with the reason.
  2. Turn the classifications into general rules, with thresholds, that would also cover combinations not in the table.
  3. Specify URL format, canonical, robots, internal linking and sitemap treatment for each tier.
  4. Decide what happens to the colour-only and price views, which do not fit the tiers neatly.
  5. Write the rollout plan: tests before launch, the first category to change, and what to monitor.

Worked answer

Illustrative example — not a real result

Bobbinworth: the rules for the developer

Landing pages: a single fibre or pattern within a category, and fibre plus one weight or pattern where the combination has at least 500 searches a month and at least 15 products. From the table: linen, heavyweight linen, floral cotton, gingham and floral viscose.

Canonicalised views: any other single or two-filter combination, such as navy linen, kept as parameters in a fixed alphabetical order and canonicalised to the nearest landing page.

Kept out of the crawl: three or more filters, sort, per-page and price parameters, rendered as buttons rather than links, with robots rules for those parameter patterns.

Exceptions: colour-only views canonicalise to the category; a curated "fabrics under £10" collection page replaces the price filter if the shop wants that query.

Landing pages get a clean path such as a fibre folder under the category, a self-referencing canonical, a written introduction, a title naming the range as shoppers do, a place in the sitemap and links from the category page and navigation. Their pagination stays self-canonical with plain links, so products on later pages are still found.

Canonicalised views stay usable for shoppers, but their links are not promoted in navigation and they never appear in the sitemap. Navy linen is a fair subset of linen, so the canonical is honest; if the products differed sharply, a noindex would be the more truthful control.

Kept out of the crawl. Deep combinations and view options are not links the crawler can follow, so it never finds them; the robots rules catch any that leak through external links. Empty results return a noindex message rather than a soft not-found.

The awkward cases. Colour alone has some demand but mixes every fibre and use, so it is not distinct enough to be a page. The price query has real demand, but a filter whose contents change with every price update is a poor page; a curated collection that the team maintains is better, or nothing.

Rollout. List the landing-page URLs and confirm each stays indexable and allowed under the new rules. Crawl staging and compare the indexable set with the intended set; the difference is the bug list. Change dressmaking first, then watch the indexing report by reason, crawl requests by URL pattern and the time from publishing a product to its first crawl. Keep the old rules ready to restore. Success means Crawl Budget moving from parameters to products, and the new landing pages appearing for their queries.

The same tiering logic returns at scale elsewhere: language editions governed by the hreflang guide, and hundreds of near-identical branch pages covered by the guide to location pages and listings at scale.

Common mistakes

  • Indexing every value of every filter. Thousands of thin pages compete with the categories.
  • Blocking every filter. The refinements people search for never get a page.
  • Robots rules on URLs that carry a noindex. The crawler never sees the noindex.
  • Canonicals that lie. A view with different products pointed at the category is ignored.
  • Rolling out everywhere at once. A rule that catches the landing pages removes revenue pages for weeks.

Test your judgement in the indexing at scale checkpoint, or revisit the intermediate SEO level for the crawling and indexing controls.

Pages this lesson applies

Terms used on this page

Also related

Related