Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

SEO › Technical SEO

Crawling and Indexing

How Google discovers, crawls, renders and indexes pages, what robots.txt, sitemaps, canonicals and noindex control, and how to fix indexing issues.
Guideintermediate10 min readUpdated

Discovery: how URLs get on the list

A crawler cannot fetch what it does not know exists. URLs enter the queue in three main ways:

  • Links from pages already crawled — internal links above all, which is why site architecture is a crawling topic, and external links.
  • XML sitemaps submitted in Search Console or referenced from robots.txt. A sitemap is a list of URLs the site considers canonical and worth indexing; it does not force crawling, but it is the most direct way to say "these pages exist and these are their preferred URLs". Google documents that it uses the lastmod value when it is accurate and consistently maintained, and ignores it where it is not.
  • Manual requests through the URL inspection tool, for individual URLs.

A URL that is in none of these — an orphan page reachable only by typing its address — is effectively invisible. Audits find orphans by comparing the crawl with the sitemap and with the server's own list of pages.

Crawling: fetching and capacity

Googlebot fetches URLs from its queue, respecting robots.txt and pacing itself so it does not overload the server. Google describes crawl budget as the combination of crawl capacity limit (how fast the site can be fetched without harm, inferred from response times and errors) and crawl demand (how much the engine wants to fetch, driven by popularity and staleness). Google's documentation is explicit that crawl budget is a concern for large sites — on the order of a million or more unique pages with content that changes often, or tens of thousands of pages changing daily — and that most sites never need to think about it. The Crawl Budget glossary entry covers the term; the practical point for smaller sites is that "improving crawl budget" is rarely the lever, while for large catalogues it is, which is why faceted navigation gets its own guide.

Two things reliably waste crawl on any size of site: endless URL spaces (parameters, calendars, session IDs, filter combinations) and large numbers of redirects and errors. Both are visible in server logs and in Search Console's crawl stats report; the log file analysis tutorial shows how to count them template by template from the logs.

Robots.txt

A plain-text file at the site root that tells crawlers which paths they may fetch. Documented behaviour worth stating plainly:

  • It controls crawling, not indexing. A disallowed URL can still appear in results if other pages link to it; Google shows it with no description because it could not read the page.
  • A noindex rule inside robots.txt is not supported by Google and has not been since 2019.
  • Disallowing a page also prevents the crawler from seeing any noindex or canonical tag on it — the two controls conflict, and the robots block wins.
  • Blocking CSS and JavaScript resources prevents rendering, which can make a page look empty to the indexer.

Use robots.txt for what it is for: keeping crawlers out of infinite spaces, internal search results, staging paths and resources that genuinely must not be fetched.

Status codes and redirects

The server's response decides what happens next. A 200 is fetched and considered; a 301 or 308 passes the URL's signals to the destination and the destination is indexed instead; a 302 or 307 is treated as temporary and may keep the original URL indexed for a while, though Google documents that long-lived temporary redirects are eventually treated as permanent; a 404 or 410 drops the URL from the index after it is re-confirmed; a 5xx or a timeout slows crawling and, if persistent, drops the page. Soft 404s — pages that return 200 but say "not found" — are detected and treated as errors. Redirect chains (A to B to C) waste fetches and dilute signals; the migrations guide covers mapping them cleanly.

Rendering: seeing what the browser sees

After fetching the HTML, Google queues the page for rendering with a headless Chromium, executes its JavaScript, and indexes the rendered result. Rendering happens after the initial crawl and can be delayed, so content that exists only after script execution is indexed later and less reliably than content in the initial HTML. The JavaScript SEO guide covers the consequences; for this guide the point is that "indexable" means indexable after rendering, and that blocked scripts or failed rendering produce a page the indexer reads as nearly empty.

Indexing: what gets kept

Fetching a page does not mean storing it. The indexer decides whether the page is worth keeping, which URL is its canonical, and what it is about. Three controls and one judgement are involved.

The noindex directive

A <meta name="robots" content="noindex"> tag in the HTML head, or an X-Robots-Tag: noindex HTTP header for non-HTML files, tells the engine not to index the page. It is the correct tool for pages that must exist but must not appear in search: internal search results, thank-you pages, filtered views, thin tag pages. The page must be crawlable for the directive to be seen. Google documents that a long-standing noindex page is also eventually crawled less and that its links may stop being followed, so noindex is not a way to preserve link flow through a page that is kept out of the index.

Canonicalisation

When several URLs show the same or very similar content — http and https, with and without www, with tracking parameters, with and without a trailing slash, printer versions, filter combinations — the engine groups them and picks one to index and show. The rel="canonical" link element (Canonical Tag) is the site's statement of which it should be. Google documents the canonical tag as a strong hint, not a directive: it is weighed with redirects, sitemap inclusion, internal linking, HTTPS preference and the content itself, and Google will choose a different canonical when the signals disagree. Search Console's URL inspection shows both the user-declared and the Google-selected canonical, which is where canonical disputes are diagnosed.

Practical rules that follow from the documented behaviour:

  • Every indexable page carries a self-referencing canonical, so duplicates created by parameters point back to it.
  • The canonical URL is the one that is linked internally, listed in the sitemap and returns 200 — a canonical pointing to a redirect or a noindexed page sends contradictory signals.
  • Canonical and noindex are not combined on the same page; one says "index that one instead", the other says "index nothing", and Google documents the combination as contradictory.
  • Cross-domain canonicals are supported for syndicated content but are honoured only when the pages really are duplicates.

Quality judgement

Pages that are crawlable and not blocked can still be left out. Search Console reports these as "Crawled — currently not indexed" (fetched, judged not worth storing now) and "Discovered — currently not indexed" (known but not yet fetched, often because the engine deprioritised the crawl). Google describes both as signals that the pages, or the site overall, did not meet its quality bar or crawl priority rather than as errors to fix with a directive. The usual remedies are to improve or consolidate the pages concerned, strengthen internal links to the ones that matter, and remove the ones that should not exist.

Diagnosing indexing problems

The audit tutorial covers the full process; the indexing-specific sequence is short:

  1. Is the URL known? Check internal links and the sitemap. If it is in neither, the fix is architecture.
  2. Can it be fetched? Test the URL against robots.txt and check the status code. A block or an error ends the investigation here.
  3. Is it allowed to be indexed? Look for noindex in the HTML and the headers — including ones injected by plugins or staging configurations that were never removed.
  4. Which canonical did Google choose? Inspect the URL in Search Console (GSC). If the selected canonical differs from the declared one, find the stronger signal (a redirect, a sitemap entry, internal links to the other URL) and align them.
  5. Was it judged worth indexing? If none of the above explains the gap, the page is in the quality category: compare it with similar indexed pages, consolidate near-duplicates, and strengthen links to it.

The SEO page checker reads the robots directives, canonical and status of a single live URL, which answers steps two to four in one pass.

Illustrative example — not a real result

A catalogue losing category pages

An online retailer notices that half its category pages have disappeared from search. Search Console shows thousands of "Duplicate, Google chose different canonical than user" URLs. The crawl reveals why: every category page is reachable at four URLs (with and without a trailing slash, with a sort parameter, and through a legacy path), each with a self-referencing canonical, and the navigation links to the version with the parameter. Google chose the parameter version as canonical for some categories and the clean version for others, splitting signals between them. The fix was a redirect from every variant to the clean URL, a canonical on the clean URL only, and navigation links updated to match. No content changed; the pages returned as the redirects were recrawled.

To catch problems like these before they spread, the technical SEO maintenance playbook sets a monthly review of crawling and indexing alongside release checks and weekly Search Console triage.

Controls compared

GoalRight controlWrong control and why
Keep crawlers out of an infinite URL spacerobots.txt disallownoindex on every URL — they would all have to be fetched first
Keep a page out of results but let crawlers read itmeta robots noindex (page must be crawlable)robots.txt — the URL can still be indexed from links
Consolidate duplicate URLs onto oneredirect where possible; canonical where the variants must stay reachablenoindex on the duplicates — throws their signals away instead of consolidating them
Remove a page permanently410 or 404, plus removal of internal linksredirect to the homepage — treated as a soft 404
Tell the engine which pages exist and which URL is preferredXML sitemap of canonical, indexable, 200 URLs onlya sitemap containing redirects, noindexed or parameter URLs — contradictory signals

When Search Console reports pages as crawled but not kept, the troubleshooting page on fixing crawled, currently not indexed works through the quality, duplication, linking and rendering causes one at a time.

When URLs sit in the queue without ever being fetched, the diagnosis on fixing discovered, currently not indexed separates server capacity, URL bloat and weak linking.

When Google indexes a different URL because no canonical was declared, follow the steps on resolving duplicate without user-selected canonical.

Launch pages need to be discoverable on the day of the announcement without leaking early; SEO strategy for a product launch covers which pages to publish ahead and how to get the rest indexed quickly.

When Search Console lists URLs as indexed even though robots.txt blocks them, the page on fixing indexed, though blocked by robots.txt decides per URL group whether to unblock and noindex, unblock and keep, or stop linking.

Which of the two controls fits a given goal, and why stacking them on the same page fails, is set out in the comparison of noindex and robots.txt disallow.

For documents that cannot carry a meta tag, the guide to PDFs and other files in search shows how indexing rules, canonicals and language alternates are sent in response headers.

Common mistakes

  • Blocking in robots.txt to prevent indexing. The URL stays indexable from links and the noindex can never be seen.
  • Leaving a staging noindex on the live site. The most common cause of a site vanishing after launch.
  • Canonicals pointing at redirects, noindexed pages or the wrong protocol.
  • Sitemaps as a dump of every URL rather than a curated list of canonicals.
  • Fixing "crawled, currently not indexed" with directives instead of with page quality and consolidation.
  • Blocking CSS and JavaScript, so rendered pages look empty.
  • Treating crawl budget as a small-site problem. For most sites, the lever is quality and links, not crawl allocation.

The site architecture guide covers the link structure that drives discovery and priority; the JavaScript SEO guide covers the rendering stage in detail; and the author's technical SEO expertise page shows how these checks fit a working audit routine.

Frequently asked questions

Does blocking a URL in robots.txt remove it from search results?

No. Robots.txt controls fetching, not indexing. A blocked URL can still be indexed from links pointing to it, shown with no snippet. To keep a page out of the index, allow crawling and use a noindex directive; to remove it quickly, use Search Console's removals tool as well.

Why does Search Console say 'Crawled, currently not indexed'?

Google fetched the page and decided not to store it, usually because it judged the content thin, duplicate or low value relative to similar pages it already has. It is a quality signal about the page or its template, not a technical block.

How often does Google recrawl a page?

There is no fixed schedule. Crawl frequency follows how often the page appears to change, how important it seems from links and sitemaps, and how much crawl capacity the site is allocated. Pages that rarely change are recrawled rarely; that is expected, not a problem.

Terms used on this page

Tools to try

Frameworks

Where this page is used

Part of these learning paths

Sources

  1. In-depth guide to how Google Search works — Google Search Central
  2. Robots.txt introduction and guide — Google Search Central
  3. Block Search indexing with noindex — Google Search Central
  4. Large site owner's guide to managing your crawl budget — Google Search Central

Related