Definition
Indexability is the state of a URL that meets every technical condition for a search engine to add it to its index. It is a property the site controls, which is what separates it from indexing, the engine's decision about whether an eligible page is worth keeping. A non-indexable page will not be indexed however good it is; an indexable page may still be left out on quality or duplication grounds.
The term matters because eligibility is assembled from signals that live in different places: the robots.txt file, the server's response, the page's HTML or HTTP headers, and the output of its scripts. The crawling and indexing guide adds an important qualifier: a page is indexable only after rendering, so content or directives that scripts change are judged on the rendered version.
The conditions: status, directives, canonical and rendering
For a URL to be indexable, all of the following have to hold:
- It can be crawled. A URL disallowed in robots.txtA file at a site's root telling crawlers which URLs they may and may not fetch. cannot be fetched, and the crawler then never sees any other signal on it. Google's documentation notes the conflict this creates: a page blocked from crawling cannot show its noindex rule, so it can still appear in results if other pages link to it.
- It returns a success status. A redirect sends the engine on to the target, which is then the URL judged, and not-found or gone responses take the page out.
- It carries no noindex rule. Google's noindex documentation describes the rule as a meta tag or an HTTP response header, and explains that the page is dropped when it is next crawled and the rule is seen. A noindexA robots directive telling search engines not to show a page in their results. is therefore a deliberate exclusion, not a temporary hint.
- Its canonical points at itself. A page whose Canonical TagTells search engines the preferred URL. names another URL is telling the engine that the other URL is the one to index. Google's guide to consolidating duplicate URLs treats the canonical as a hint rather than a directive, and Google may choose a different canonical when the signals disagree.
- Its content survives RenderingThe step in which a search engine runs a page's HTML, CSS and JavaScript to build the content a browser would display before indexing it.. If the content exists only after scripts run and the render fails, the page may be indexed as near-empty or treated as a soft not-found page.
In practice: deciding indexability per page type and template
Indexability is best decided for page types rather than page by page. The tutorial on writing SEO requirements for developers asks every new template to state, for each URL type it creates, whether it is indexable and where its canonical points, so the robots rules, canonical, sitemap and internal links agree. On stores the decision is sharpest in filtered listings: the guide to faceted navigation keeps a small indexable tier of filter combinations with real demand and canonicalises or noindexes the rest. Pages that are built to end, such as listings and events, need the decision made before they expire, which the guide to expired pages sets out per page type.
Because a single template change can switch eligibility for thousands of URLs, indexability is also a release check. The SEO release checks compare robots rules, directives and canonicals on sample URLs of each template before and after a deployment, and keep the sitemap to canonical, indexable URLs.
A furniture shop with eligibility the wrong way round
Tansyrook Furniture, an online furniture shop, launched a new range of 64 sofas in March. Four weeks later none of the new product pages appeared in Search Console's indexed pages, while 2,300 colour-filter URLs did. A crawl showed both problems at once: the new product template had shipped with the noindex rule used on the staging site, and the colour filters had lost their canonical tags in the same release, making every filtered view indexable. The team removed the noindex, restored the canonicals to the parent categories and added a release check that compares the directive and canonical on one sample URL per template. Within six weeks 58 of the sofa pages were indexed.
Related terms
The noindexA robots directive telling search engines not to show a page in their results. rule is the explicit way to make a page non-indexable, and robots.txtA file at a site's root telling crawlers which URLs they may and may not fetch. controls crawling rather than indexing, which is why the two should not be used on the same URL. The Canonical TagTells search engines the preferred URL. decides which of several duplicate URLs is the indexable one. RenderingThe step in which a search engine runs a page's HTML, CSS and JavaScript to build the content a browser would display before indexing it. determines whether the content and directives the engine judges are the ones the developer intended.
Common confusions
- Indexable means indexed. Eligibility is the site's side of the bargain; the engine still chooses.
- Blocking in robots.txt removes a page from the index. It stops crawling; the URL can still appear from links, and any noindex on it can never be read.
- A canonical tag is an instruction. It is a hint that the engine can override when other signals disagree.
- Every page should be indexable. Filter combinations, internal search results and thin duplicates are usually better left out, so that the pages that matter carry the signals.
- The page source settles it. Directives changed by scripts are read from the rendered page, which can differ from the source.
