Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Glossary term

Indexability

Whether a URL is eligible to be added to a search engine's index: reachable, returning success, not excluded by noindex and not canonicalised elsewhere.
SEOconceptUpdated

Definition

Indexability is the state of a URL that meets every technical condition for a search engine to add it to its index. It is a property the site controls, which is what separates it from indexing, the engine's decision about whether an eligible page is worth keeping. A non-indexable page will not be indexed however good it is; an indexable page may still be left out on quality or duplication grounds.

The term matters because eligibility is assembled from signals that live in different places: the robots.txt file, the server's response, the page's HTML or HTTP headers, and the output of its scripts. The crawling and indexing guide adds an important qualifier: a page is indexable only after rendering, so content or directives that scripts change are judged on the rendered version.

The conditions: status, directives, canonical and rendering

For a URL to be indexable, all of the following have to hold:

  • It can be crawled. A URL disallowed in robots.txt cannot be fetched, and the crawler then never sees any other signal on it. Google's documentation notes the conflict this creates: a page blocked from crawling cannot show its noindex rule, so it can still appear in results if other pages link to it.
  • It returns a success status. A redirect sends the engine on to the target, which is then the URL judged, and not-found or gone responses take the page out.
  • It carries no noindex rule. Google's noindex documentation describes the rule as a meta tag or an HTTP response header, and explains that the page is dropped when it is next crawled and the rule is seen. A noindex is therefore a deliberate exclusion, not a temporary hint.
  • Its canonical points at itself. A page whose Canonical Tag names another URL is telling the engine that the other URL is the one to index. Google's guide to consolidating duplicate URLs treats the canonical as a hint rather than a directive, and Google may choose a different canonical when the signals disagree.
  • Its content survives Rendering. If the content exists only after scripts run and the render fails, the page may be indexed as near-empty or treated as a soft not-found page.

In practice: deciding indexability per page type and template

Indexability is best decided for page types rather than page by page. The tutorial on writing SEO requirements for developers asks every new template to state, for each URL type it creates, whether it is indexable and where its canonical points, so the robots rules, canonical, sitemap and internal links agree. On stores the decision is sharpest in filtered listings: the guide to faceted navigation keeps a small indexable tier of filter combinations with real demand and canonicalises or noindexes the rest. Pages that are built to end, such as listings and events, need the decision made before they expire, which the guide to expired pages sets out per page type.

Because a single template change can switch eligibility for thousands of URLs, indexability is also a release check. The SEO release checks compare robots rules, directives and canonicals on sample URLs of each template before and after a deployment, and keep the sitemap to canonical, indexable URLs.

Illustrative example — not a real result

A furniture shop with eligibility the wrong way round

Tansyrook Furniture, an online furniture shop, launched a new range of 64 sofas in March. Four weeks later none of the new product pages appeared in Search Console's indexed pages, while 2,300 colour-filter URLs did. A crawl showed both problems at once: the new product template had shipped with the noindex rule used on the staging site, and the colour filters had lost their canonical tags in the same release, making every filtered view indexable. The team removed the noindex, restored the canonicals to the parent categories and added a release check that compares the directive and canonical on one sample URL per template. Within six weeks 58 of the sofa pages were indexed.

The noindex rule is the explicit way to make a page non-indexable, and robots.txt controls crawling rather than indexing, which is why the two should not be used on the same URL. The Canonical Tag decides which of several duplicate URLs is the indexable one. Rendering determines whether the content and directives the engine judges are the ones the developer intended.

Common confusions

  • Indexable means indexed. Eligibility is the site's side of the bargain; the engine still chooses.
  • Blocking in robots.txt removes a page from the index. It stops crawling; the URL can still appear from links, and any noindex on it can never be read.
  • A canonical tag is an instruction. It is a hint that the engine can override when other signals disagree.
  • Every page should be indexable. Filter combinations, internal search results and thin duplicates are usually better left out, so that the pages that matter carry the signals.
  • The page source settles it. Directives changed by scripts are read from the rendered page, which can differ from the source.

More terms to look up

Where this term is used

Sources

  1. Block Search indexing with noindex — Google Search Central
  2. Consolidate duplicate URLs — Google Search Central

Related