Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

SEO › Technical SEO

Log File Analysis for SEO

How to use server logs for SEO: get the files, isolate search engine crawlers, label URLs by template, and find crawl waste and pages crawlers neglect.
Tutorialadvanced9 min readUpdated

What server logs show that other sources do not

Google's description of how Search works explains that crawling is algorithmic: Googlebot decides which sites to crawl, how often and how many pages to fetch from each. Its guide for large sites describes the limit on that activity as Crawl Budget, set by how much the server can take and how much Google wants to crawl, and states that for most sites it is not a constraint. The crawling and indexing guide covers those mechanics and the crawl budget entry lists what wastes it. Logs are how you see the result for your own site, request by request.

SourceWhat it tells youWhat it cannot tell you
Server or CDN access logsEvery request: URL, time, status code, user agent, bytes, response timeWhether a page was indexed or why
Search Console Crawl StatsGoogle's own requests in aggregate, by response code and file type, with examplesPer-URL counts for the whole site
Search Console page indexingWhat Google indexed and the stated reason for what it did notHow often each URL is requested
A site crawlWhat can be reached by following links todayWhat crawlers actually fetch, including unlinked URLs

The value of logs is in the gaps between those rows: URLs crawled heavily that no link on the site points to any more, important pages requested rarely or never, and status codes the crawler receives that a browser check would not reveal. The academy exercise on reading crawl logs and URL parameters for a job board practises finding those gaps in a week of logs labelled by template, then turning them into parameter rules and a requirement developers can build.

Before you start

  • Access, agreed in writing. Logs often sit with a hosting provider, a CDN or an operations team. Ask for the access logs, not the error logs, for every host name that serves the site.
  • Personal data handled properly. Access logs record IP addresses and sometimes query strings that contain personal details. Agree with whoever owns data protection where the extract is stored, who can see it and when it is deleted, and drop the fields you do not need.
  • The fields you need (below). If any is missing, ask before the extract is made rather than after.
  • A list of templates. Every URL pattern on the site mapped to its page type: product, category, article, filter, internal search, and so on. The technical SEO audit produces one if you do not have it.
  • The XML sitemap (XML Sitemap) and a recent site crawl for the same period.
  • A tool that can handle the volume. A spreadsheet copes with a small site; anything larger needs a database, a command-line toolkit or a log analysis tool.
FieldWhy it is needed
TimestampSpreads requests across days and catches spikes after releases
Requested URL, including the query stringThe unit of analysis; parameters are where most waste hides
Status codeSeparates successes, redirects, client errors and server errors
User agentIdentifies requests claiming to be search engine crawlers
Client IP addressLets you isolate requests that borrow a crawler's name without being from it
Host nameSeparates subdomains and protocol variants served by the same logs
Response time or bytesShows slow templates, which matter for how much crawling the server can take

The steps

  1. Agree access and the fields

    Confirm with the log owner which hosts are covered, which fields each line contains, how far back logs are kept and in what format the extract will arrive. Retention is often short, so request the extract early; if the period you need has already been deleted, start collecting now and run the analysis later.

  2. Collect enough days

    Take at least a week for an active site and several weeks for a large one, so that sections crawled rarely show up at all. Avoid windows that contain a migration, an outage or a major release unless that event is what you are studying. Note the dates; the second measurement must use a window of the same length.

  3. Isolate search engine crawler requests

    Filter the lines to the user agents of the search engine crawlers you care about. Crawler names change and one search engine may use several, so build the filter from the names actually present in your logs, check them against the search engine's own documentation, and keep the list in one place. Treat the user agent as a claim, not proof: scrapers and monitoring tools often borrow crawler names.

  4. Check the filter against Search Console

    Compare your daily totals for Google's crawler with the Crawl Stats report in Search Console (GSC), which counts Google's own requests. The two will not match exactly, because the report covers the whole property and your logs may not, but a large excess in your logs means impostors are inside the filter. Find the addresses responsible for the excess, confirm with the log owner that they are not the search engine's, and exclude them before going further.

  5. Label each URL with its template

    Normalise the URLs (lower-case host, consistent trailing slash, query strings kept but parameters noted) and attach a template label from your pattern list. Add the click depth from the site crawl and a flag for whether the URL is in the sitemap. Unlabelled URLs are a finding in themselves: old patterns, staging paths or parameter variants nobody planned for.

  6. Count requests by template, status code and depth

    Build three summaries: requests per template, requests per status code within each template, and requests per click depth. The first shows where crawling goes; the second shows what the crawler receives; the third shows whether deep pages are reached at all. Look at requests per URL as well as totals, because a template with many URLs and few requests is under-crawled even if its total looks healthy.

  7. Compare crawled URLs with the sitemap and a site crawl

    Join three lists: URLs crawlers requested, URLs in the sitemap, and URLs the site crawl found. URLs in the sitemap but never requested in the window are candidates for neglect. URLs requested but absent from both the sitemap and the crawl are orphans, legacy addresses or variants generated somewhere you did not intend. URLs blocked by robots.txt rules should be absent from the logs; if they appear, the rule is not doing what you think. Remember that Google's robots documentation is clear that blocking crawling does not by itself keep a URL out of the index, which the noindex versus robots.txt comparison explains.

  8. List the waste: parameters, redirects and errors

    Waste is crawling spent on URLs that should not be crawled or should not exist: sort, filter and tracking parameters; session identifiers; internal search results; redirect hops, especially chains where a 301 Redirect points to another redirect; not-found responses for URLs still linked somewhere; and pages that return success but say nothing, which are soft 404s. Google's large-site guide names the same sources of waste and their remedies. Record each with its share of requests and an example URL.

  9. List the neglect: important pages rarely requested

    Neglect is the other side: products, articles or categories that matter commercially but were requested rarely or not at all. Sort important templates by days since last request and look for patterns: new pages linked only from deep pagination, sections reachable only through script, pages missing from the sitemap. The discovered, currently not indexed page covers the same pattern as Search Console reports it.

  10. Fix, then measure again

    Turn each finding into a ticket with example URLs and the expected change in the logs: stop linking parameter variants and handle the rest as the faceted navigation guide describes, point internal links and the sitemap at final URLs, return honest status codes, and link neglected pages from where crawlers already go. After the fixes have been live for a few weeks, repeat steps three to nine on a window of the same length and compare the summaries.

A week of logs, read (illustrative)

Illustrative example — not a real result

An outdoor-gear store finds where the crawler goes

Ferngrotto Outdoor Supply, an invented online retailer, sells about 9,000 products in 600 categories. After filtering and checking against Crawl Stats, a seven-day extract holds 180,000 requests from Google's crawler. By template: 58% went to category URLs carrying sort or filter parameters, 21% to product pages, 9% to redirects (mostly old product URLs passing through two hops), 4% to not-found URLs still linked from discontinued-range pages, and the rest to categories, articles and resources. Of the 9,000 product URLs, 3,100 were not requested at all in the week; most had been added in the previous two months and were linked only from the fourth page of their category onwards. The team removed crawlable links to sort orders, applied the store's filter rules, updated internal links and the sitemap to final URLs, fixed the links to removed products, and added a new-arrivals block to the first page of each category. Four weeks later, a second seven-day window shows parameter URLs at 19% of requests, product pages at 52%, and 700 products not requested in the week.

Making log analysis a routine

A one-off analysis finds the big problems; a routine keeps them from coming back. Three habits are enough for most teams. Keep a monthly summary of crawler requests by template and status code, so a shift is visible against the previous month rather than discovered in an audit. Watch for spikes in server errors or redirects returned to crawlers after each release, because a deployment that changes URL handling shows up in the logs days before it shows up anywhere else. When those errors also appear in Search Console, the page on server errors in Search Console uses the same counts by template and hour to find the cause. And re-run the full comparison with the sitemap and a crawl at each technical audit, so the template list and the filter stay current. The technical SEO maintenance playbook schedules this log review beside the release checks and Search Console triage, and sends what it finds into a single fix queue. The academy checkpoint on server logs, parameters and developer requirements tests verifying crawlers, labelling URLs by template and turning what the logs show into rules developers can test.

Common mistakes

  • Trusting the user agent. Requests that borrow a crawler's name inflate every count; check against Search Console and the addresses before drawing conclusions.
  • Reading totals instead of rates. A template with a large total can still be under-crawled per URL; divide by the number of URLs in the template.
  • Analysing a window that contains a migration or outage and treating the result as normal.
  • Ignoring personal data in the extract, or keeping it longer than the analysis needs.
  • Reporting findings without example URLs. Developers fix what they can reproduce.
  • Expecting more crawling to mean better rankings. Logs show fetching, not indexing or ranking; a page requested often can still be left out of the index, as the crawling guide explains.
  • Measuring once. Without a second window of the same length, there is no evidence the fixes worked.

Frequently asked questions

How many days of logs do I need for an SEO log analysis?

Enough to cover the way crawlers move through the site. For an active site a week is a sensible minimum, because daily patterns and release days even out; for a large site, or one where some sections are rarely crawled, take several weeks so that slow sections appear at all. Keep the same window length when you measure again after fixes, otherwise differences in volume are differences in the sample.

Can I analyse crawling without access to the web server's logs?

Often, yes. Sites behind a content delivery network or on a hosted platform usually have request logs at that layer instead, and they serve the same purpose if they include the requested URL, status code, user agent and time. Where no logs exist at all, Search Console's Crawl Stats report is the fallback: it shows Google's own requests, response codes and file types, with example URLs, but not every request for every URL, so it cannot produce the per-template counts this tutorial relies on.

Is a crawl with an SEO crawler tool the same as log analysis?

No. A crawler tool shows what can be reached by following links from a starting point, as that tool sees the site. Logs show what search engine crawlers actually requested, including URLs no longer linked, parameter variants found elsewhere and pages the tool never reached. The two answer different questions, which is why one step of this tutorial compares them: the differences between the lists are where most findings come from.

Terms used on this page

Tools to try

Frameworks

Where this page is used

Part of these learning paths

Sources

  1. Large site owner's guide to managing your crawl budget — Google Search Central
  2. In-depth guide to how Google Search works — Google Search Central
  3. Robots.txt introduction and guide — Google Search Central

Related