First, decide whether each blocked URL should be indexed at all
Open the Page Indexing report in Search Console (GSCGoogle's search performance and indexing tool.), select the "Indexed, though blocked by robots.txt" reason and export the examples. The list is a sample, but it is enough to see patterns. Sort the URLs by path and parameter until they fall into a few groups: a folder, a template, a parameter, a staging path. Then give every group one of three verdicts.
The first verdict is should not be indexed: account pages, internal search results, thank-you pages, print versions. These need to leave the index, which requires the crawler to read a noindex, which requires the block to go. The second is should be indexed: pages meant to rank that a rule catches by mistake. These need the block removed and nothing else. The third is should not exist as a URL for crawlers at all: endless filter combinations and session parameters. These are better kept blocked, with the links that expose them removed, so the index stops learning about them. Each verdict leads to a different fix, and the crawling and indexing guide explains why the controls behave as they do. Then inspect two or three URLs from each group with URL Inspection to confirm the block and find the referring page, and use the table to name the cause.
Diagnosis table
Diagnosis
| Symptom | Check | Likely cause | Fix |
|---|---|---|---|
| Private or utility pages appear in results with no description | Whether the robots rule was added to keep them out of search | Disallow rule used where noindex was intended | Remove the disallow, add noindex, leave the URL crawlable until it drops |
| Blocked URLs keep reappearing although the rule is correct | Internal links, sitemaps and external links pointing at the blocked paths | Blocked URLs still linked internally or from other sites | Remove internal links and sitemap entries; decide on noindex if the URL must stay reachable |
| Filter or parameter URLs listed, many of them | When the parameter rule was added compared with when the URLs were first indexed | Parameter or faceted URLs blocked after they were indexed | Unblock briefly with noindex or canonical, or keep blocked and stop linking |
| Pages meant to rank show in the list | The rule that matches them, tested against the live robots.txt | Overbroad disallow pattern catching pages meant to rank | Narrow the pattern so it matches only the intended paths |
| A whole section or the site appears, soon after a launch | The live robots.txt against the staging version and the launch date | Staging or migration rules left in production robots.txt | Replace the production file with the intended rules and recheck |
Fix, verify and prevent each cause
A block used as a way to hide pages
The most common origin is a reasonable wish expressed with the wrong tool: someone wanted login, basket or thank-you pages out of search and added a disallow. Google's robots.txt introduction is explicit that the file manages crawler traffic and is not a mechanism for keeping a page out of results.
Fix: remove the disallow for those paths, add a noindex meta tag or response header, and keep the pages crawlable until Google has fetched them and seen the directive, as the noindex documentation requires. The controls compared in the crawling and indexing guide show which tool does which job.
Verify: URL Inspection on a sample shows the page as crawled and excluded by noindex, and the group moves from this status to the noindex reason in the report.
Prevent: keep a written rule that robots.txt is for crawl control only, and route every "hide this page" request to noindex.
Links that keep feeding the index
A correct block can still produce this status if the site keeps linking to the blocked URLs: a header link to the basket, an old navigation item, a sitemap that was never updated, or external sites that cite a blocked path. Each link is a reason for Google to keep the URL.
Fix: remove internal links and sitemap entries pointing at blocked paths, using a crawl to find them as the internal linking guide describes; where a link must stay for users, render it in a way crawlers do not follow, or switch the URL to noindex instead of a block. External links cannot be removed, so for URLs that others cite, the noindex route is the one that settles the question.
Verify: a fresh crawl finds no internal links to the blocked paths, and the sitemap lists only indexable URLs.
Prevent: add a check to the crawl routine that flags any internal link or sitemap entry pointing at a disallowed path.
Facets blocked after the damage was done
Filter and parameter URLs often get indexed first and blocked later, once someone notices the crawl waste. The block stops new crawling but freezes the URLs already in the index, because Google can no longer fetch them to see a canonical or a noindex.
Fix: decide per filter type using the tiers in the faceted navigation guide. For combinations that should consolidate into a category, lift the block for a period so the canonical or noindex can be read, then restore it if crawl volume demands. For deep combinations nobody should land on, keep the block and stop exposing them through crawlable links.
Verify: the count for the parameter group falls in the report over the following weeks, and crawl stats show less time spent on parameter URLs, a saving of Crawl BudgetHow much a search engine crawls a site. rather than of pages.
Prevent: set the robots, canonical and linking treatment for a new filter before it is released, not after its URLs have spread.
A pattern wider than intended
Robots rules match by prefix and wildcard, and a short pattern catches more than its author meant: a rule for a private folder that also matches a public folder starting with the same letters, or a wildcard intended for one parameter that matches a product slug. The symptom is pages you want to rank appearing in this list, indexed from links but without their content.
Fix: test the rule against example URLs from both the intended and the accidental groups, then narrow it so it matches only the intended paths, following the robots.txt section of the crawling and indexing guide.
Verify: URL Inspection's live test shows the important pages as allowed, and they move into the indexed and crawled count once recrawled.
Prevent: keep a set of must-allow and must-block example URLs and test every robots change against both lists before release.
Staging rules carried into production
A staging site is often blocked with a sweeping disallow, and a launch that copies the staging configuration carries that rule live. Google keeps the URLs it already knew, now unreadable, and new pages never get crawled. The site migrations guide lists checking robots.txt among its pre-launch checks for exactly this reason.
Fix: replace the production robots.txt with the intended rules at once, confirm the robots.txt report in Search Console shows the new file, and request indexing for the key pages.
Verify: the live test shows important URLs as allowed, and the number of URLs under this reason falls as Google recrawls.
Prevent: protect staging with authentication rather than robots rules where possible, and make the robots file a named item in every release checklist.
Prevention: robots.txt changes reviewed like code
The robots file is short, powerful and rarely reviewed, which is why it causes outsized problems. Keep it in version control with the rest of the site, so each change has an author, a date and a reason. Require a second person to review any edit, and give the reviewer the must-allow and must-block URL lists to test against. Decide the purpose of each rule in a comment beside it: crawl control for an infinite space, a private path, or an endpoint that must not be fetched. Anything whose purpose is "keep out of Google" is a noindex job.
After each release, fetch the live robots.txt and compare it with the reviewed version, then glance at the Page Indexing report for new entries under this reason over the following weeks. Monthly, review the Search Console reports for the trend in each indexing reason, not just the total, because a growing group under this status usually means a new link or a new pattern. Good SEOImproving visibility in unpaid search results. practice treats the file as part of the site's architecture, owned by someone who knows why every line is there.
The robots file is one of several settings worth comparing before every deployment, and the guide to SEO release checks covers the rest, from staging protection to alerts after go-live.
A travel site's blocked booking pages
A travel site sees 2,400 URLs under "Indexed, though blocked by robots.txt". Sorting the export gives three groups. About 1,900 are search-result URLs with a date parameter, blocked a year earlier after they had been crawled. Around 450 are booking-step pages, blocked because the team wanted them out of search, and linked from every hotel page's "Book" button. The last 50 are guide pages under a folder whose name begins with the same letters as a blocked "/bookings" rule. The team narrows the rule so the guides are allowed, switches the booking steps from a block to noindex, and keeps the search parameter blocked while replacing its crawlable links with a form. Within eight weeks the guide pages are indexed with snippets, the booking-step group moves to "Excluded by noindex", and the remaining count drops to about 600.
Before choosing between unblocking and blocking, the comparison of noindex and robots.txt disallow explains which control keeps pages out of results and which only stops crawling.
Common mistakes
Adding noindex to pages that robots.txt still blocks. Blocking a page to remove it from search. Leaving blocked URLs in the sitemap. Writing short prefix rules without testing them. Treating a large count of blocked filter URLs as an emergency when it may only be leftover history. Other indexing statuses are routed from the SEO troubleshooting hub.
