What server logs show that other sources do not
Google's description of how Search works explains that crawling is algorithmic: Googlebot decides which sites to crawl, how often and how many pages to fetch from each. Its guide for large sites describes the limit on that activity as Crawl BudgetHow much a search engine crawls a site., set by how much the server can take and how much Google wants to crawl, and states that for most sites it is not a constraint. The crawling and indexing guide covers those mechanics and the crawl budget entry lists what wastes it. Logs are how you see the result for your own site, request by request.
| Source | What it tells you | What it cannot tell you |
|---|---|---|
| Server or CDN access logs | Every request: URL, time, status code, user agent, bytes, response time | Whether a page was indexed or why |
| Search Console Crawl Stats | Google's own requests in aggregate, by response code and file type, with examples | Per-URL counts for the whole site |
| Search Console page indexing | What Google indexed and the stated reason for what it did not | How often each URL is requested |
| A site crawl | What can be reached by following links today | What crawlers actually fetch, including unlinked URLs |
The value of logs is in the gaps between those rows: URLs crawled heavily that no link on the site points to any more, important pages requested rarely or never, and status codes the crawler receives that a browser check would not reveal. The academy exercise on reading crawl logs and URL parameters for a job board practises finding those gaps in a week of logs labelled by template, then turning them into parameter rules and a requirement developers can build.
Before you start
- Access, agreed in writing. Logs often sit with a hosting provider, a CDN or an operations team. Ask for the access logs, not the error logs, for every host name that serves the site.
- Personal data handled properly. Access logs record IP addresses and sometimes query strings that contain personal details. Agree with whoever owns data protection where the extract is stored, who can see it and when it is deleted, and drop the fields you do not need.
- The fields you need (below). If any is missing, ask before the extract is made rather than after.
- A list of templates. Every URL pattern on the site mapped to its page type: product, category, article, filter, internal search, and so on. The technical SEO audit produces one if you do not have it.
- The XML sitemap (XML SitemapA file listing the URLs a site wants search engines to discover and crawl.) and a recent site crawl for the same period.
- A tool that can handle the volume. A spreadsheet copes with a small site; anything larger needs a database, a command-line toolkit or a log analysis tool.
| Field | Why it is needed |
|---|---|
| Timestamp | Spreads requests across days and catches spikes after releases |
| Requested URL, including the query string | The unit of analysis; parameters are where most waste hides |
| Status code | Separates successes, redirects, client errors and server errors |
| User agent | Identifies requests claiming to be search engine crawlers |
| Client IP address | Lets you isolate requests that borrow a crawler's name without being from it |
| Host name | Separates subdomains and protocol variants served by the same logs |
| Response time or bytes | Shows slow templates, which matter for how much crawling the server can take |
The steps
Agree access and the fields
Confirm with the log owner which hosts are covered, which fields each line contains, how far back logs are kept and in what format the extract will arrive. Retention is often short, so request the extract early; if the period you need has already been deleted, start collecting now and run the analysis later.
Collect enough days
Take at least a week for an active site and several weeks for a large one, so that sections crawled rarely show up at all. Avoid windows that contain a migration, an outage or a major release unless that event is what you are studying. Note the dates; the second measurement must use a window of the same length.
Isolate search engine crawler requests
Filter the lines to the user agents of the search engine crawlers you care about. Crawler names change and one search engine may use several, so build the filter from the names actually present in your logs, check them against the search engine's own documentation, and keep the list in one place. Treat the user agent as a claim, not proof: scrapers and monitoring tools often borrow crawler names.
Check the filter against Search Console
Compare your daily totals for Google's crawler with the Crawl Stats report in Search Console (GSCGoogle's search performance and indexing tool.), which counts Google's own requests. The two will not match exactly, because the report covers the whole property and your logs may not, but a large excess in your logs means impostors are inside the filter. Find the addresses responsible for the excess, confirm with the log owner that they are not the search engine's, and exclude them before going further.
Label each URL with its template
Normalise the URLs (lower-case host, consistent trailing slash, query strings kept but parameters noted) and attach a template label from your pattern list. Add the click depth from the site crawl and a flag for whether the URL is in the sitemap. Unlabelled URLs are a finding in themselves: old patterns, staging paths or parameter variants nobody planned for.
Count requests by template, status code and depth
Build three summaries: requests per template, requests per status code within each template, and requests per click depth. The first shows where crawling goes; the second shows what the crawler receives; the third shows whether deep pages are reached at all. Look at requests per URL as well as totals, because a template with many URLs and few requests is under-crawled even if its total looks healthy.
Compare crawled URLs with the sitemap and a site crawl
Join three lists: URLs crawlers requested, URLs in the sitemap, and URLs the site crawl found. URLs in the sitemap but never requested in the window are candidates for neglect. URLs requested but absent from both the sitemap and the crawl are orphans, legacy addresses or variants generated somewhere you did not intend. URLs blocked by robots.txtA file at a site's root telling crawlers which URLs they may and may not fetch. rules should be absent from the logs; if they appear, the rule is not doing what you think. Remember that Google's robots documentation is clear that blocking crawling does not by itself keep a URL out of the index, which the noindex versus robots.txt comparison explains.
List the waste: parameters, redirects and errors
Waste is crawling spent on URLs that should not be crawled or should not exist: sort, filter and tracking parameters; session identifiers; internal search results; redirect hops, especially chains where a 301 RedirectA permanent server redirect telling browsers and search engines a URL has moved for good. points to another redirect; not-found responses for URLs still linked somewhere; and pages that return success but say nothing, which are soft 404sA page that returns a success status code but shows an error, empty or not-found message.. Google's large-site guide names the same sources of waste and their remedies. Record each with its share of requests and an example URL.
List the neglect: important pages rarely requested
Neglect is the other side: products, articles or categories that matter commercially but were requested rarely or not at all. Sort important templates by days since last request and look for patterns: new pages linked only from deep pagination, sections reachable only through script, pages missing from the sitemap. The discovered, currently not indexed page covers the same pattern as Search Console reports it.
Fix, then measure again
Turn each finding into a ticket with example URLs and the expected change in the logs: stop linking parameter variants and handle the rest as the faceted navigation guide describes, point internal links and the sitemap at final URLs, return honest status codes, and link neglected pages from where crawlers already go. After the fixes have been live for a few weeks, repeat steps three to nine on a window of the same length and compare the summaries.
A week of logs, read (illustrative)
An outdoor-gear store finds where the crawler goes
Ferngrotto Outdoor Supply, an invented online retailer, sells about 9,000 products in 600 categories. After filtering and checking against Crawl Stats, a seven-day extract holds 180,000 requests from Google's crawler. By template: 58% went to category URLs carrying sort or filter parameters, 21% to product pages, 9% to redirects (mostly old product URLs passing through two hops), 4% to not-found URLs still linked from discontinued-range pages, and the rest to categories, articles and resources. Of the 9,000 product URLs, 3,100 were not requested at all in the week; most had been added in the previous two months and were linked only from the fourth page of their category onwards. The team removed crawlable links to sort orders, applied the store's filter rules, updated internal links and the sitemap to final URLs, fixed the links to removed products, and added a new-arrivals block to the first page of each category. Four weeks later, a second seven-day window shows parameter URLs at 19% of requests, product pages at 52%, and 700 products not requested in the week.
Making log analysis a routine
A one-off analysis finds the big problems; a routine keeps them from coming back. Three habits are enough for most teams. Keep a monthly summary of crawler requests by template and status code, so a shift is visible against the previous month rather than discovered in an audit. Watch for spikes in server errors or redirects returned to crawlers after each release, because a deployment that changes URL handling shows up in the logs days before it shows up anywhere else. When those errors also appear in Search Console, the page on server errors in Search Console uses the same counts by template and hour to find the cause. And re-run the full comparison with the sitemap and a crawl at each technical audit, so the template list and the filter stay current. The technical SEO maintenance playbook schedules this log review beside the release checks and Search Console triage, and sends what it finds into a single fix queue. The academy checkpoint on server logs, parameters and developer requirements tests verifying crawlers, labelling URLs by template and turning what the logs show into rules developers can test.
Common mistakes
- Trusting the user agent. Requests that borrow a crawler's name inflate every count; check against Search Console and the addresses before drawing conclusions.
- Reading totals instead of rates. A template with a large total can still be under-crawled per URL; divide by the number of URLs in the template.
- Analysing a window that contains a migration or outage and treating the result as normal.
- Ignoring personal data in the extract, or keeping it longer than the analysis needs.
- Reporting findings without example URLs. Developers fix what they can reproduce.
- Expecting more crawling to mean better rankings. Logs show fetching, not indexing or ranking; a page requested often can still be left out of the index, as the crawling guide explains.
- Measuring once. Without a second window of the same length, there is no evidence the fixes worked.
