What a site crawl can and cannot show
A crawler copies one part of a search engine's work. It starts from the addresses you give it, reads each response, follows the links it finds and records what came back: the status, the headers, the title, the directives, the links in and out. It does this from your machine or your provider's servers, under your settings, on the day you run it. That makes it a model of a search engine, not a search engine. It cannot say what Google indexed or why, which is Search Console's job, and it cannot say what search engine crawlers actually requested, which the server's own records show in full. The log file analysis tutorial sets those views side by side; a crawl answers a narrower question, what can be reached and what each page says about itself, and answers it well only when the settings match the question.
The settings matter because each one changes what the crawl finds. A crawl that runs scripts finds links that a plain crawl misses. A crawl that ignores robots rules reports pages search engines are told to skip. A crawl that drops query strings never meets the parameter versions of a page, and a crawl that keeps all of them may never finish. None of these is wrong in itself; each is right for some questions and misleading for others.
| What the crawl must answer | Where it starts | Settings that decide the result | What it produces |
|---|---|---|---|
| A technical audit of the whole site | The home page and the sitemap, plus URL lists | Robots rules obeyed, rendering on templates that need it, every field | One of the views the audit reconciles |
| Whether a release changed anything | Staging and production, the same sample addresses | Access to staging, identical settings on both hosts | A difference between two crawls |
| A migration inventory | Every old address: links, sitemaps, logs, Search Console | Lists loaded alongside link-following, no depth cap | The source list for the redirect map |
| A parameter register | Listing templates with their filters and sorts | Query strings kept, a limit for each template | Every parameter with example addresses |
| Broken citations on your own pages | The pages that cite sources | Outbound links requested and their status recorded | A list of dead or redirected references |
Where a step below goes beyond what the linked guides state, such as writing the question first, recording the boundaries and fences, a separate crawl that ignores robots rules, the crawler's own name, pausing when errors rise, a template marker or a test on a sample, it is general practice rather than search engine documentation.
Before you start
- The question, in one sentence, and the name of the person who will act on the answer. Settings chosen without a question default to everything, which is slow and still answers nothing in particular.
- Permission and a contact. The site owner's agreement to crawl, and a named person who runs the server, the hosting or the protection layer, because a crawl adds load and may trip security rules.
- A crawler that can do what the question needs: render pages, read response headers, crawl a list of addresses, log in to a protected host and extract custom fields. Desktop and cloud tools differ in these, so check before you start rather than halfway through.
- The address of every sitemap, Search Console access for the property, the analytics landing page report and, where they exist, server logs filtered to search engine crawlers.
- A template list: every page type the site produces, the URL pattern for each and one sample address per template.
- The previous crawl's settings and results, if there was one. Without them, a comparison measures the change in settings as much as the change in the site.
The steps
Write down what the crawl has to answer
State the question before opening the tool: whether the new product template keeps its links and canonicals, which old addresses a migration must redirect, which parameters a listing page generates, whether the site's citations still work. An audit's crawl is one of the three views the technical SEO audit compares, so it takes the broadest settings; a release crawl takes the narrowest settings that cover the templates the release touches. Write down the output the question needs, a list, a difference between two crawls or a register, and what the crawl will not answer, so nobody reads ranking or indexing conclusions into a crawler's export.
Set the start points and the boundaries
Start from the home page and the XML SitemapA file listing the URLs a site wants search engines to discover and crawl., and add the main section pages where the navigation is thin or built by scripts. Decide which hosts count as the site: the main host, subdomains such as a help centre or a shop, and the variants with and without
wwwor on the old protocol. Give the variants as start points so their redirects are tested, but keep the crawler from wandering into hosts that are not yours. Set include and exclude patterns for paths the question does not need, and on a large site split the crawl by section, each section run with the same settings, rather than one crawl that never completes. Write every boundary into the crawl's notes, because a reader of the results has to know what was left out before counting what is there.Decide how robots rules and directives are treated
The main crawl obeys the site's robots.txtA file at a site's root telling crawlers which URLs they may and may not fetch., because its job is to show what search engines may fetch. Google's robots documentation, as the crawling and indexing guide sets out, makes the file a control on crawling, not on indexing, and a blocked page's own noindex or canonical cannot be read. Where you need to see what sits behind a rule, such as a section disallowed by mistake, run a second crawl that ignores the file, on your own site only, label it, and never add its counts to the main one. For page-level directives, set the crawler to record noindex, nofollow and canonicals but still fetch and follow the pages that carry them, so that links from noindexed pages and the addresses canonicalised away both appear. Read directives from response headers as well as the HTML: Google's noindex documentation allows the rule in an HTTP response header, and the guide to PDFs and other files relies on headers for files that have no HTML head.
Agree speed, timing and identity with whoever runs the server
Every request a crawler makes is load on the server. Google paces its own crawling to what the site can take, as its crawl budget guide describes; your crawler has to be paced by hand. The diagnosis for server errors in Search Console shows the two ways a crawl goes wrong at this point: heavy templates that fail under crawling load, and protection layers that challenge visitors they do not recognise. Agree a request rate and a time window away from the site's busiest hours, let the crawler announce itself under its own name, and ask for it to be let through the protection layer so the crawl measures the site rather than the firewall. Watch error responses while the crawl runs; if they rise, pause, slow the crawl and mark the errors already recorded as possibly caused by the crawl itself until a slower rerun confirms them. Borrowing a search engine's name proves nothing about what that engine receives; as the log file analysis tutorial notes, a user agent is a claim, not proof.
Choose plain HTML, rendered, or both
A plain crawl reads the HTML the server sends; a rendered crawl runs the page's scripts first, as a browser would, and is slower and heavier on the server. Google renders pages after fetching them, later and with limits, so content and links that exist only after RenderingThe step in which a search engine runs a page's HTML, CSS and JavaScript to build the content a browser would display before indexing it. are the ones at risk; the JavaScript SEO guide explains the failure modes and compares crawls in both modes. Crawl a sample of each template both ways and compare link counts, headings and body text. Where the two agree, crawl that template plain; where they differ, crawl it rendered in full and report the difference itself as a finding. Let the crawler fetch the scripts and styles a page needs, and record any resource that robots rules block, because a blocked script produces an emptier page for a search engine as well as for your tool.
Keep parameters, but fence off endless URL spaces
Do not let the crawler strip query strings by default: tracking tags, sorts, filters and session identifiers are part of what search engines meet, and the URL parameter audit cannot run without them. Filters, sorts and session identifiers can also produce more addresses than any crawl can finish, which is why the faceted navigation guide treats them as a crawl problem in their own right. Put a fence round each URL ParameterA key and value appended to a URL after a question mark, such as a filter, sort order, session or tracking tag, that may or may not change the page. or template: a limit on how many variations are followed, exclusions for patterns that are known to repeat content, and a separate small crawl when the parameters themselves are the question. Record every limit. A crawl that stops at its own fence shows where the fence was, not that the space beyond it is small.
Give the crawler access to protected hosts
A Staging EnvironmentA private copy of a website or tool where changes are tested before release, kept behind access control and out of search and production analytics. belongs behind authentication, as the release checks guide recommends, so the crawler needs credentials or an allowed address, never a gap in the protection opened for the crawl and forgotten afterwards. Staging can serve a different robots file, different headers and a different host name in canonicals from the ones production will use, so note which settings come from the environment and compare them against the version that will be deployed, not the version staging happens to serve. Crawl staging and production with the same settings and the same start points, so the difference between the two crawls is the release and nothing else.
Add the URL lists that links cannot reach
A crawler that only follows links finds only what is linked, which leaves out exactly the addresses an audit or a migration cannot do without: orphaned pages, old addresses other sites still link to, parameter versions found elsewhere. Load lists alongside the link-following crawl: the sitemap's addresses, the example addresses in Search Console's Page Indexing report, the pages in its performance report, the analytics landing pages, the addresses from the logs and, before a move, the whole of the old site, which the website migration checklist inventories. Keep a column recording where each address came from. That column is what later separates pages nobody links to from pages that should not exist; the comparison itself belongs to the audit, not to the crawl settings.
Choose the fields to extract and label each template
Keep the standard fields on: the status codeThe numeric code a web server returns with every response, saying whether the request succeeded, was redirected or failed and why., the redirect target and the number of hops, directives from the HTML and the headers, the canonicalTells search engines the preferred URL., the title, the meta description, the main heading, a word count, internal links in and out with their anchors, the click depth the site architecture guide reads, hreflang annotations, structured data types, response time and content type. Add custom extraction for what the question needs and the defaults miss: a price or stock element, a published date, an author, or a template marker such as a class or data attribute that developers can add to each page type. Then label every address with its template, from the marker or from URL patterns, so the results are read template by template rather than as one long list. If the question includes citations, have the crawler request outbound links and record their status, as the outbound links guide suggests for its scheduled checks.
Test on a sample before the full crawl
Run the settings on a few addresses from each template and check the output against two other views. Open the same pages in a browser and confirm that the title, headings, links and body text the crawler recorded are the ones a visitor sees. Then run the live test in Search Console's URL Inspection, which the Search Console tutorial describes, and compare its rendered HTML, its directives and its canonical with your crawler's. The SEO page checker gives a quick third reading of status, directives and canonical for a single address. Where the crawler disagrees with both, the settings are wrong: a script it was not allowed to fetch, a consent or cookie notice hiding the content, a header it did not read, an extraction rule pointing at the wrong element. Fix the settings and rerun the sample before committing to the whole site.
Save the settings with the results and compare like with like
Export the configuration and store it with the results: the date, the start points, the boundaries, the speed, the identity, the rendering choice, the parameter fences, the lists loaded and the extraction rules. The technical SEO audit scorecard asks for the same crawler settings when the site is rescored, for the reason that applies to every comparison: different settings produce different counts. When you compare two crawls, before and after a release or audit to audit, use the same saved settings and the same start points, compare template by template, and treat any difference you cannot trace to a change on the site as a question about the crawl before it becomes a finding. The technical SEO maintenance playbook sets the rhythm in which those crawls recur.
Where the results go next
A crawl is an input, not a conclusion. Each kind of finding has an owner elsewhere on the site, and the crawl's job is to hand it over in a form the owner can use: addresses grouped by template, with the setting that produced them stated.
| What the crawl found | Where it goes next |
|---|---|
| Pages Google may not be able to reach or index | The audit's indexability and duplication checks, then a ticket per template |
| Links and content present only after rendering | The JavaScript failure modes and the developers' acceptance criteria |
| Parameter versions and endless spaces | The parameter register and its rule for each class |
| Old addresses and where they now lead | The redirect map for the move |
| A template fault repeated on every page built from it | SEO requirements written for developers, so the next template is built right |
| Differences between staging and production | The release decision, before anything ships |
Findings that need Google's own view, such as whether a page is indexed or which canonical Google chose, go to Search Console before they are reported; the crawl shows what the page declares, and only Google's tools show what Google did with it.
A mineral dealer crawls before a template release (illustrative)
Xelvrudo Minerals checks a new specimen template
Xelvrudo Minerals, an invented online seller of mineral and fossil specimens, is replacing the template for its 4,800 specimen pages with one that loads a panel of related specimens by script. Its catalogue also has 140 category pages with filters for colour, size and price. The question is written first: does the new template keep each specimen's links, canonical and description visible to a crawler, on staging as it will be in production?
The team agrees with its host a rate of 2 requests a second between 22:00 and 06:00, and has the crawler let through the protection layer under its own name; a first test at 4 requests a second drew challenge pages on about 6% of requests, which disappear at the agreed rate. Both hosts are crawled from the home page and the sitemap with robots rules obeyed, query strings kept but capped at 50 variations per category, and custom extraction for a template marker the developers add to every page type. Staging stays behind its password; the crawler is given credentials.
The sample of 20 addresses per template shows the problem before the full crawl does. In plain mode, staging specimen pages carry 3 internal links each against 11 in rendered mode; production shows 11 in both, because the old template wrote the related panel into the HTML. The full staging crawl confirms it across all 4,800 specimen pages, with self-referencing canonicals intact. The developers move the related panel into the server response, a rerun with the saved settings shows 11 links per page in both modes, and the release goes ahead. The settings file, dated and named for the release, is used again for the crawl of production a week later, which matches the staging result.
The technical SEO hub places site crawls among the other jobs that keep a site reachable, renderable and indexable.
Common mistakes
- Crawling without a question. Every setting defaults to everything, the crawl runs far longer than it needs to and the export answers nothing in particular.
- Comparing crawls run with different settings. The difference measures the tool's configuration as much as the site; save the settings and reuse them.
- Ignoring robots rules in the main crawl. The results then include pages search engines are told not to fetch, and the counts no longer describe what an engine can reach.
- Reading crawler-caused errors as the site's errors. A crawl that is too fast can create the very failures it reports; slow down and rerun before raising them.
- Posing as a search engine's crawler. It proves nothing about what the engine receives, and the results still come from your tool rather than from the engine.
- Crawling a script-built site in plain mode only. Links and content that exist only after rendering are missing from the results, and the gap is invisible unless both modes are compared.
- Letting parameters run unfenced, or stripping them all. One crawl never finishes; the other never meets the addresses the parameter audit needs.
- Opening staging to make the crawl easier. Give the crawler credentials instead; a gap in the protection lets crawlers reach staging pages.
- Relying on links alone. Orphaned pages, old addresses and parameter versions found elsewhere only appear when lists are loaded too.
- Reading totals instead of templates. A fault in one template looks small in a site-wide count and is fixed once in the template.
