Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Comparisons

noindex vs robots.txt Disallow

The noindex directive and robots.txt disallow compared: one stops indexing, the other stops crawling, why combining them backfires and which fits each case.
Comparisonintermediate8 min readUpdated

noindex directive and robots.txt disallow at a glance

noindex is an instruction about indexing. It sits on the page itself, as a robots meta tag in the head or as an X-Robots-Tag HTTP header, and tells search engines that support it not to include the page in results. Google's documentation on blocking indexing is explicit that the crawler has to be able to fetch the page to see the directive: if the page is blocked from crawling, the noindex is never read. The page remains reachable for anyone who visits it, and the crawler can still follow its links. The crawling and indexing guide covers where this stage sits in the pipeline from discovery to the index.

A robots.txt disallow is an instruction about crawling. It lives in a single file at the root of the host and tells compliant crawlers which URL paths or patterns they should not request. Google's robots.txt guide describes the file as a way to manage crawler traffic and says plainly that it is not a mechanism for keeping a page out of Google. A disallowed URL that is linked from elsewhere can still appear in results, usually without a description, because Google knows the URL exists but could not read it. The troubleshooting page for indexed, though blocked by robots.txt covers how to fix that state when it appears.

The distinction is the whole comparison. noindex controls what is shown. A disallow controls what is fetched. Each is the right tool for its own goal and the wrong tool for the other's.

Compare table

noindex directive vs robots.txt disallow

Row labelnoindex directiverobots.txt disallow
What it controlsWhether a page can appear in search resultsWhether compliant crawlers fetch matching URLs
Where it livesOn each page, as a meta tag or an HTTP headerIn one file at the root of the host
Whether the page is crawledYes, and it must be, or the directive is never seenNo, for crawlers that respect the file
Whether it can be indexedNo, once the directive has been readYes, if the URL is linked from elsewhere, usually without a description
Effect on linksThe crawler can still read the page and follow its linksLinks on the page are not read, and any canonical or noindex on it is invisible
ScopeOne page or one response at a time, usually set per templateWhole paths or URL patterns across the site in one rule
Speed of effectWhen each page is next crawledWhen the crawler next fetches the robots.txt file
Best-fit casesThin, private-ish or utility pages that may stay reachable but should not rankInfinite URL spaces, internal search results, sort and filter parameters that waste crawling

The rows on whether the page is crawled and whether it can be indexed explain every confusion between the two. noindex needs crawling to work; a disallow prevents crawling; so a disallow cannot deliver what noindex does.

When to choose the noindex directive

Choose noindex when a page should be reachable but should not appear in search. Thank-you pages, account and basket pages, thin tag archives, internal campaign pages and staging copies that must be accessible to some people all fit. Visitors can still use them; search simply leaves them out.

Choose it when pages are already indexed and need to come out. Because noindex is read on the next crawl, it is the documented way to remove pages from results, and it works only if crawling is allowed. If a disallow is already blocking those pages, it has to be lifted for the noindex to be seen.

Choose it for individual pages or templates rather than for large URL spaces. Every noindexed URL still has to be crawled for the directive to work, so applying it to millions of parameter combinations spends the crawl effort the site was trying to save. For pages that duplicate another indexable page, a Canonical Tag usually fits better than noindex, because it consolidates signals instead of discarding them; the comparison of canonical tags and 301 redirects covers that choice.

Remember that noindex is not access control. A noindexed page can still be visited by anyone with the link. Content that must not be seen at all belongs behind a login.

Files such as PDFs cannot carry a meta tag, so they take the noindex rule as an HTTP response header, which the guide to PDFs and other files in search explains with the matching canonical header.

When to choose a robots.txt disallow

Choose a disallow when the problem is crawling, not indexing. Internal search result pages, endless calendar URLs, sort orders and parameter combinations can generate far more URLs than a site has real pages. Blocking those patterns in robots.txt stops crawlers spending Crawl Budget on them, so they reach the pages that matter sooner. The faceted navigation guide shows how filter URLs are tiered and which tier robots.txt suits.

Choose it for areas that should never be fetched by crawlers for operational reasons, such as resource-heavy endpoints or infinite spaces a bot could get lost in. One rule in one file covers every matching URL, which no page-level directive can match for scale.

Choose it only for URLs that do not need removing from the index. If the blocked URLs are linked from anywhere, some may appear in results as indexed but blocked. For low-value parameter URLs that is usually tolerable; for anything sensitive or embarrassing it is not, and those pages need noindex or a login instead.

Do not block the resources pages need to render. Disallowing script or stylesheet files can leave the crawler seeing an empty or broken page, which hurts the pages you want indexed.

Illustrative example — not a real result

A job board separates crawling from indexing

Larkspire Jobs finds that its internal search pages generate about 180,000 crawled URLs a month, and that 3,400 of them are indexed and appearing in results. Its first instinct is to disallow the /search/ path. Instead it adds noindex to the search results template and leaves crawling open. Over the next six weeks the indexed count falls to under 200. Only then does it add the robots.txt disallow for /search/, which cuts crawler requests to that path to almost nothing. The few dozen search URLs that still show as indexed but blocked come from external links and have no descriptions, which the team accepts. Separately, expired job listings, around 9,000 a month, stay live for applicants who saved the link but carry noindex, so they leave results without being blocked from crawling.

Decision table

GoalChooseWhy
Keep a reachable page out of search resultsnoindexIt is read on crawl and removes the page from results
Remove pages that are already indexednoindex, with crawling allowedThe crawler must fetch the page to see the directive
Stop crawlers wasting effort on an infinite URL spacerobots.txt disallowOne rule blocks every matching URL without fetching any
Hide sensitive content from everyoneNeither: put it behind a loginnoindex hides from search only; robots.txt hides nothing from visitors
Consolidate duplicates onto a preferred URLNeither: a canonical tag or a redirectnoindex discards signals; a disallow hides the canonical
Clear indexed parameter URLs, then save crawl effortnoindex first, then a disallow once they have dropped outSequencing lets the directive be seen before crawling stops

Why combining them backfires

The most common mistake with these two controls is applying both to the same pages to be doubly sure. The disallow tells the crawler not to fetch the page, so the noindex on that page is never read. If the URL is linked from anywhere, inside or outside the site, Google can still index it from the link alone, and now the site has no way to tell Google to drop it. The page that was meant to disappear shows up in results as indexed but blocked, with no description.

The same logic hides other page-level signals. A canonical tag on a disallowed page cannot be seen, so it cannot consolidate anything. Links on a disallowed page are not read, so pages reachable only through it may never be discovered.

The fix is to sequence the controls rather than stack them. To remove pages and then save crawl effort, apply noindex with crawling allowed, wait until URL inspection and the page indexing report show the pages excluded, and only then add the disallow. When Search Console reports a pile of blocked-but-indexed URLs, the Search Console error triage SOP puts that row into the right queue, and the indexed-though-blocked troubleshooting page decides for each group of URLs whether to unblock and noindex, unblock and keep, or stop linking. Review robots.txt changes as carefully as code, because a single rule can hide noindex and canonical tags across thousands of pages.

Both controls are also the settings most often carried from staging into production by accident, which is why the guide to SEO checks for staging and releases protects staging with access control instead and compares both before each deployment.

Common mistakes

  • Disallowing pages in robots.txt to remove them from search, which leaves them indexable from links with no way to drop them.
  • Adding noindex and a disallow to the same pages, so the directive is never read.
  • Applying noindex to millions of parameter URLs, which spends the crawl effort a disallow would have saved.
  • Treating noindex as security. Anyone with the link can still open the page.
  • Leaving a staging noindex or a blanket disallow on the live site after a launch.
  • Blocking scripts and stylesheets that pages need to render, so indexed pages look broken to the crawler.
  • Using noindex on duplicates that should be canonicalised, discarding signals that could have been consolidated.

Frequently asked questions

Does a robots.txt disallow remove a page from Google?

No. Google's robots.txt guide says the file is mainly for managing crawler traffic and is not a mechanism for keeping a page out of Google. A disallowed URL can still be indexed if other pages link to it, usually shown without a description because Google could not read the page. To keep a page out of results, let it be crawled and use noindex, or protect it with a login if it must not be seen at all.

Should I add both noindex and a robots.txt disallow to be safe?

No, that is the combination that fails. The disallow stops the crawler fetching the page, so it never sees the noindex, and the URL can stay indexed from links. If the goal is removal from results, allow crawling and use noindex. Once the pages have dropped out and you also want to save crawl effort, you can then add the disallow, accepting that any new links to those URLs could bring them back as indexed but blocked.

How quickly does noindex take effect?

Only when the page is next crawled, because Google has to fetch it to read the directive. Frequently crawled pages drop out sooner and rarely crawled ones later, and there is no fixed timetable. URL inspection in Search Console shows whether Google has seen the directive, and requesting a recrawl of important URLs can speed up the process for those pages.

Terms used on this page

Where this page is used

Also related

Sources

  1. Robots.txt introduction and guide — Google Search Central
  2. Block Search indexing with noindex — Google Search Central

Related