Definition
Google's large-site crawl budget guide defines crawl budget as the amount of time and resources Google devotes to crawling a site, set by two factors. Crawl capacity limit, sometimes called crawl rate, is the maximum number of simultaneous connections and the delay between fetches that Googlebot will use, which it raises when the site responds quickly and lowers when it slows or returns server errors. Crawl demand is how much Google wants to crawl, driven by how popular the URLs are, how stale Google's copy has become, and site-wide events such as a migration. The registry's one-line summary captures the practical point: crawl budget is wasted on duplicates and traps, and it matters most for large sites.
How it works
Googlebot keeps a list of known URLs for the site, ranks them by demand, and fetches as many as the capacity limit allows. Every fetch counts, whether or not the URL is worth indexing: a parameterised duplicate of a category page, a calendar that generates a new month forever, an internal search results page, a soft error page that returns success with no content, or a redirect chain all consume budget that could have gone to a new product page. Google's guide lists these as the common sources of waste and names the remedies: consolidate duplicate content, block crawling of URLs that should never be indexed, return a proper not-found status for removed pages, keep sitemaps accurate, avoid long redirect chains, and make pages load efficiently so the capacity limit rises. The guide also states which sites should care, giving documented size and change-frequency thresholds; below them, Google says, the site is crawled efficiently anyway.
| Source of waste | Why it costs budget | Documented remedy |
|---|---|---|
| Faceted and parameter URLs | Thousands of near-identical pages per category | Consolidate; block crawling of combinations that should not be indexed |
| Soft error pages | Return success, so they are recrawled like real pages | Return a not-found or gone status |
| Infinite spaces | Calendars, session identifiers, endless pagination | Block or remove the generating pattern |
| Redirect chains | Each hop is a fetch | Point links and sitemaps at the final URL |
| Slow or erroring servers | Capacity limit is lowered | Improve response time and reliability |
In practice
The first question is whether the site is large enough to care; Google's guide gives the thresholds, and a site of a few thousand stable pages is not. For sites that are, the Crawl Stats report in Search Console (GSCGoogle's search performance and indexing tool.) shows how many requests Googlebot made, what response codes it received and which file types it fetched, which is where waste becomes visible. The crawling and indexing guide explains the controls, from robots rules to status codes, and the site architecture guide covers the internal linking that steers crawl demand toward important pages. On stores, the faceted navigation guide is where most crawl waste is created and resolved.
A store with more filter URLs than products
A retailer has 40,000 products and a category system with filters for size, colour, brand and price. The Crawl Stats report shows about 900,000 requests a month, and log analysis finds that 80 percent of them hit filter combinations, of which fewer than 2,000 have any search demand. New products take three to four weeks to be indexed. The team keeps the combinations with demand as real, self-canonical pages, blocks crawling of the rest with a robots rule on the parameter pattern, and removes filter links from the HTML for blocked combinations. Two months later crawl requests are down to roughly 300,000 a month, most of them to product and category pages, and new products are typically indexed within a week.
What it is not
Crawl budget is not an indexing guarantee: a crawled page can still be left out of the index on quality grounds, and the guide separates the two. It is not a ranking factor; crawling more often does not rank a page higher, it only keeps Google's copy fresh. It is not controlled by the canonical tag (Canonical TagTells search engines the preferred URL.), which consolidates signals after a fetch but does not prevent the fetch; only robots rules do that, and a URL blocked in robots cannot pass canonical signals. And it is not a small-site problem: for most sites, optimising it changes nothing, and the time is better spent on content and links, or on the indexing problems the same reports reveal, which are far more common.
Related terms
Search engine optimisation (SEOImproving visibility in unpaid search results.) treats crawl budget as one of the technical conditions for a page to rank at all. The canonical tag (Canonical TagTells search engines the preferred URL.) is the main consolidation tool for the duplicates that waste budget, and Search Console (GSCGoogle's search performance and indexing tool.) is where the Crawl Stats and indexing reports make the problem measurable. Related mechanisms covered in the crawling and indexing guide include robots rules, the noindex directive, XML sitemaps and the status codes that tell a crawler whether to come back; each acts at a different point in the discovery, crawl and index sequence, and confusing them is the usual cause of a wasted budget or a lost page, which is why the guide treats them together as one system.
Common mistakes
- Optimising crawl budget on a small site where Google documents that it is not a constraint, at the cost of work that would have moved rankings.
- Using a canonical tag to stop crawling, which it cannot do; the duplicate is still fetched and the budget is still spent.
- Blocking URLs in robots that should be consolidated, so their signals are lost rather than merged into the page that should rank.
- Leaving soft error pages returning success, so they keep being recrawled as if they were content and clutter the index with empty pages.
- Not reading the Crawl Stats report before and after changes, so the effect is never measured and the next decision is a guess rather than a reading.
