noindex directive and robots.txt disallow at a glance
noindex is an instruction about indexing. It sits on the page itself, as a robots meta tag in the head or as an X-Robots-Tag HTTP header, and tells search engines that support it not to include the page in results. Google's documentation on blocking indexing is explicit that the crawler has to be able to fetch the page to see the directive: if the page is blocked from crawling, the noindex is never read. The page remains reachable for anyone who visits it, and the crawler can still follow its links. The crawling and indexing guide covers where this stage sits in the pipeline from discovery to the index.
A robots.txt disallow is an instruction about crawling. It lives in a single file at the root of the host and tells compliant crawlers which URL paths or patterns they should not request. Google's robots.txt guide describes the file as a way to manage crawler traffic and says plainly that it is not a mechanism for keeping a page out of Google. A disallowed URL that is linked from elsewhere can still appear in results, usually without a description, because Google knows the URL exists but could not read it. The troubleshooting page for indexed, though blocked by robots.txt covers how to fix that state when it appears.
The distinction is the whole comparison. noindex controls what is shown. A disallow controls what is fetched. Each is the right tool for its own goal and the wrong tool for the other's.
Compare table
noindex directive vs robots.txt disallow
| Row label | noindex directive | robots.txt disallow |
|---|---|---|
| What it controls | Whether a page can appear in search results | Whether compliant crawlers fetch matching URLs |
| Where it lives | On each page, as a meta tag or an HTTP header | In one file at the root of the host |
| Whether the page is crawled | Yes, and it must be, or the directive is never seen | No, for crawlers that respect the file |
| Whether it can be indexed | No, once the directive has been read | Yes, if the URL is linked from elsewhere, usually without a description |
| Effect on links | The crawler can still read the page and follow its links | Links on the page are not read, and any canonical or noindex on it is invisible |
| Scope | One page or one response at a time, usually set per template | Whole paths or URL patterns across the site in one rule |
| Speed of effect | When each page is next crawled | When the crawler next fetches the robots.txt file |
| Best-fit cases | Thin, private-ish or utility pages that may stay reachable but should not rank | Infinite URL spaces, internal search results, sort and filter parameters that waste crawling |
The rows on whether the page is crawled and whether it can be indexed explain every confusion between the two. noindex needs crawling to work; a disallow prevents crawling; so a disallow cannot deliver what noindex does.
When to choose the noindex directive
Choose noindex when a page should be reachable but should not appear in search. Thank-you pages, account and basket pages, thin tag archives, internal campaign pages and staging copies that must be accessible to some people all fit. Visitors can still use them; search simply leaves them out.
Choose it when pages are already indexed and need to come out. Because noindex is read on the next crawl, it is the documented way to remove pages from results, and it works only if crawling is allowed. If a disallow is already blocking those pages, it has to be lifted for the noindex to be seen.
Choose it for individual pages or templates rather than for large URL spaces. Every noindexed URL still has to be crawled for the directive to work, so applying it to millions of parameter combinations spends the crawl effort the site was trying to save. For pages that duplicate another indexable page, a Canonical TagTells search engines the preferred URL. usually fits better than noindex, because it consolidates signals instead of discarding them; the comparison of canonical tags and 301 redirects covers that choice.
Remember that noindex is not access control. A noindexed page can still be visited by anyone with the link. Content that must not be seen at all belongs behind a login.
Files such as PDFs cannot carry a meta tag, so they take the noindex rule as an HTTP response header, which the guide to PDFs and other files in search explains with the matching canonical header.
When to choose a robots.txt disallow
Choose a disallow when the problem is crawling, not indexing. Internal search result pages, endless calendar URLs, sort orders and parameter combinations can generate far more URLs than a site has real pages. Blocking those patterns in robots.txt stops crawlers spending Crawl BudgetHow much a search engine crawls a site. on them, so they reach the pages that matter sooner. The faceted navigation guide shows how filter URLs are tiered and which tier robots.txt suits.
Choose it for areas that should never be fetched by crawlers for operational reasons, such as resource-heavy endpoints or infinite spaces a bot could get lost in. One rule in one file covers every matching URL, which no page-level directive can match for scale.
Choose it only for URLs that do not need removing from the index. If the blocked URLs are linked from anywhere, some may appear in results as indexed but blocked. For low-value parameter URLs that is usually tolerable; for anything sensitive or embarrassing it is not, and those pages need noindex or a login instead.
Do not block the resources pages need to render. Disallowing script or stylesheet files can leave the crawler seeing an empty or broken page, which hurts the pages you want indexed.
A job board separates crawling from indexing
Larkspire Jobs finds that its internal search pages generate about 180,000 crawled URLs a month, and that 3,400 of them are indexed and appearing in results. Its first instinct is to disallow the /search/ path. Instead it adds noindex to the search results template and leaves crawling open. Over the next six weeks the indexed count falls to under 200. Only then does it add the robots.txt disallow for /search/, which cuts crawler requests to that path to almost nothing. The few dozen search URLs that still show as indexed but blocked come from external links and have no descriptions, which the team accepts. Separately, expired job listings, around 9,000 a month, stay live for applicants who saved the link but carry noindex, so they leave results without being blocked from crawling.
Decision table
| Goal | Choose | Why |
|---|---|---|
| Keep a reachable page out of search results | noindex | It is read on crawl and removes the page from results |
| Remove pages that are already indexed | noindex, with crawling allowed | The crawler must fetch the page to see the directive |
| Stop crawlers wasting effort on an infinite URL space | robots.txt disallow | One rule blocks every matching URL without fetching any |
| Hide sensitive content from everyone | Neither: put it behind a login | noindex hides from search only; robots.txt hides nothing from visitors |
| Consolidate duplicates onto a preferred URL | Neither: a canonical tag or a redirect | noindex discards signals; a disallow hides the canonical |
| Clear indexed parameter URLs, then save crawl effort | noindex first, then a disallow once they have dropped out | Sequencing lets the directive be seen before crawling stops |
Why combining them backfires
The most common mistake with these two controls is applying both to the same pages to be doubly sure. The disallow tells the crawler not to fetch the page, so the noindex on that page is never read. If the URL is linked from anywhere, inside or outside the site, Google can still index it from the link alone, and now the site has no way to tell Google to drop it. The page that was meant to disappear shows up in results as indexed but blocked, with no description.
The same logic hides other page-level signals. A canonical tag on a disallowed page cannot be seen, so it cannot consolidate anything. Links on a disallowed page are not read, so pages reachable only through it may never be discovered.
The fix is to sequence the controls rather than stack them. To remove pages and then save crawl effort, apply noindex with crawling allowed, wait until URL inspection and the page indexing report show the pages excluded, and only then add the disallow. When Search Console reports a pile of blocked-but-indexed URLs, the Search Console error triage SOP puts that row into the right queue, and the indexed-though-blocked troubleshooting page decides for each group of URLs whether to unblock and noindex, unblock and keep, or stop linking. Review robots.txt changes as carefully as code, because a single rule can hide noindex and canonical tags across thousands of pages.
Both controls are also the settings most often carried from staging into production by accident, which is why the guide to SEO checks for staging and releases protects staging with access control instead and compares both before each deployment.
Common mistakes
- Disallowing pages in robots.txt to remove them from search, which leaves them indexable from links with no way to drop them.
- Adding noindex and a disallow to the same pages, so the directive is never read.
- Applying noindex to millions of parameter URLs, which spends the crawl effort a disallow would have saved.
- Treating noindex as security. Anyone with the link can still open the page.
- Leaving a staging noindex or a blanket disallow on the live site after a launch.
- Blocking scripts and stylesheets that pages need to render, so indexed pages look broken to the crawler.
- Using noindex on duplicates that should be canonicalised, discarding signals that could have been consolidated.
