Definition
The file lives at the root of a host, at the path /robots.txt, and applies only to that host and
protocol; a subdomain needs its own. It is made of groups. Each group opens with one or more user-agent
lines naming the crawlers it addresses, or an asterisk for all of them, followed by disallow and allow
rules written as path prefixes. A crawler reads the group that names it most specifically and ignores the
others. Where an allow and a disallow rule both match a URL, Google documents how it chooses between
them, broadly in favour of the more specific rule.
Google's guide is plain about the file's limits. It describes robots.txt as a way to manage crawler traffic, says it is not a mechanism for keeping a page out of Google, and notes that the rules are instructions that reputable crawlers follow rather than something the file can enforce. Content that must stay private needs authentication, not a disallow line.
In practice
The legitimate uses are about where crawlers spend their time: internal search results, endless calendar or filter combinations, staging paths and resources that genuinely must not be fetched. On large stores the faceted navigation guide reserves robots rules for deep filter combinations, sorts and parameters, and uses canonical or noindex for the filter pages crawlers will find through links anyway.
What never belongs in the file is anything a page needs in order to be understood. Scripts, stylesheets and data endpoints blocked by a disallow rule stop search engines rendering the page as a visitor sees it; the JavaScript SEO guide treats blocked script and API hosts as a leading cause of empty renders. Images are the same: the image SEO guide lists an image path or CDN blocked in robots.txt among the reasons pictures never reach image search.
robots.txt versus noindex
The two are confused because both affect what appears in search, but they act at different stages. The crawling and indexing guide sets out the sequence: robots rules decide what is fetched; noindex decides what is kept. That ordering produces the two classic mistakes.
- Blocking a page to remove it. The crawler obeys and stops fetching, but the URL is still known from links, so it can stay listed, typically with no description because the engine could not read the page.
- Blocking a page that carries noindex. The noindex sits inside the page the crawler is no longer allowed to fetch, so it is never seen. To remove a page, leave it crawlable until the noindex has been read.
Google also documents that a noindex rule written inside robots.txt is not supported.
Testing and the sitemap line
A robots file may carry one or more sitemap lines giving the full address of the site's sitemaps, which
is a convenient way for any crawler to find them. The file should never block a URL that the sitemap lists,
because the two then contradict each other.
Changes are tested before they ship. Search Console's URL inspection shows whether a given URL is allowed, and a site crawl configured to respect robots.txt shows which sections it cannot reach. The technical SEO audit checks that every priority page is allowed by the file, and the site migrations guide adds a launch check that the new site's robots.txt allows what should be crawled and carries none of the staging rules.
A staging rule that went live
Hallowmere Wedding Venue relaunched its website in March. The developer had built it on a staging server whose robots.txt held a single group for all crawlers with "Disallow: /", and the file was copied to the live server with the rest of the site. Within three weeks organic visits fell from about 950 a week to 310, and enquiries from search dropped from 14 a week to 4. URL inspection on the venue's main pages reported them blocked by robots.txt. The fix was one line: the disallow rule was replaced with rules covering only the booking system's internal paths. The sitemap was resubmitted the same day, the main pages were inspected again, and weekly organic visits were back above 900 within five weeks. The launch checklist now includes reading the live robots.txt.
Related terms
noindexA robots directive telling search engines not to show a page in their results. is the control for keeping a crawlable page out of results, and robots.txt must leave such a page fetchable for it to work. An XML SitemapA file listing the URLs a site wants search engines to discover and crawl. lists the URLs a site wants crawled, so it and the robots file should agree. On very large sites robots rules are the main lever on Crawl BudgetHow much a search engine crawls a site., because they stop fetches rather than merely discouraging them. A Canonical TagTells search engines the preferred URL. on a blocked URL is never read, so blocking duplicates prevents the consolidation the canonical was meant to achieve.
Common confusions
- Disallow means removed from search. It means not fetched; the URL can still be indexed from links.
- robots.txt keeps content private. Anyone can read the file, and it advertises the paths it names.
- Every crawler obeys it. Reputable crawlers do; the file cannot enforce anything.
- One file covers every subdomain. Each host needs its own.
- Blocking resources saves crawl without cost. Blocked scripts and styles can leave pages unrenderable.
