First, reproduce the error and find when and where it happens
A server error is a status codeThe numeric code a web server returns with every response, saying whether the request succeeded, was redirected or failed and why. family that says the server, not the visitor, failed. The crawling and indexing guide states the consequence: a server error or a timeout slows crawling and, if it persists, drops the page. So an error that has "gone away" when you check still matters if it keeps coming back.
Follow the reproduce step of the Search Console error triage SOP. Export the example URLs listed under the server error reason in the Page Indexing report, run the live test in URL Inspection on several of them, crawl the same URLs with your own crawler and load them in a browser. If none of them fails now, the SOP's warning applies: the problem may be intermittent, and the server logs have to be read before assuming it is fixed. The Search Console tutorial explains each report; the wording of the reasons lives in the Search Console help centre.
Then give the examples two labels: template and time. Map each URL to its page type, as the log file analysis tutorial does with its template list, and note the first detected date against the release history. In the logs, which that tutorial describes as the only record of every request a crawler actually made, count crawler requests by status code within each template and by hour of the day. The crawl stats report adds Google's own view of response time, host status and responses by type, as the discovered, currently not indexed page describes.
| Pattern in the examples and the logs | What it points to |
|---|---|
| Errors cluster at busy hours or on heavy listing templates, with slow responses just before them | The server cannot keep up with crawling |
| Crawler requests fail while browser requests to the same URL at the same moment succeed | A protection or delivery rule treats the crawler differently |
| One template, or URLs carrying one parameter, fail at any hour | Code that breaks on certain data or values |
| Errors start on a release date, or arrive in short bursts at deployment or job times | Releases, deployments or maintenance |
| Errors sit on addresses nothing on the site links to | Old or junk URLs the server mishandles |
Once a pattern is visible, use five whys to push past the first answer. "The listing page timed out" is a symptom; asking why the query was slow, why it was slow only at night and why a heavy job ran then usually ends at something a developer can change. The SEO page checker reads the status of one live URL, a quick second opinion that proves little about an intermittent fault.
Diagnosis table
Diagnosis
| Symptom | Check | Likely cause | Fix |
|---|---|---|---|
| Errors cluster at busy times on listing or search templates; response times rise first | Logs by template and hour; crawl stats response time | A server that slows or fails under crawling load, especially on heavy listing pages | Remove the bottleneck, cache heavy pages and cut needless crawlable combinations |
| Crawler requests fail while the same URL loads in a browser; live test fails | Protection layer logs; live test against a browser request | Bot protection, a firewall or a delivery network rule answering crawlers with errors | Allow documented crawlers through the provider's own settings and relax rate rules |
| Every error shares a template or a parameter | Group examples by template and parameter; request variants with empty or odd values | One template or parameter pattern that fails on the server | Make the code handle missing data and unexpected values |
| Errors begin on a release date or recur at deployment and job times | Release log against first detected date; logs by minute | Errors during releases, deployments and maintenance windows | Roll back or hotfix; move heavy jobs and keep serving during deployments |
| Errors on old paths, malformed links or junk parameters | Is the URL linked, in the sitemap or needed by anyone? | Errors on old or junk addresses that nobody needs | Answer them honestly: redirect to an equivalent or return not found |
Fix, verify and prevent each cause
A server that slows or fails under crawling load
Google's large-site guide to Crawl BudgetHow much a search engine crawls a site. describes a crawl capacity limit that rises when the site responds quickly and falls when it slows or returns server errors, as the crawl budget entry summarises. The pages that fail first are the expensive ones: category and search listings that run heavy queries for every filter and sort.
Fix: find the bottleneck with the host or developers, usually slow database queries on listing templates or missing caching, and stop exposing filter combinations nobody should crawl, following the tiers in faceted navigation and index control.
Verify: the logs show the listing template answering crawler requests with success and faster responses, the crawl stats report shows fewer server errors, and the live test passes on the old examples.
Prevent: monitor response time and error rates for crawler traffic beside user traffic, and test heavy templates against the catalogue's growth before it happens. If new URLs are also queuing unfetched, the discovered, currently not indexed page covers that side of the same problem.
Bot protection, a firewall or a delivery network rule
Protection layers that challenge unfamiliar visitors or cap request rates can answer crawler requests with errors while staff, who are recognised, see a working site. The pattern is distinctive: the live test in URL Inspection fails while a browser request to the same URL at the same moment succeeds.
Fix: read the protection layer's own log for the rule that matched, use the provider's setting for allowing documented search engine crawlers rather than hand-written exceptions, and relax rate rules that fire on ordinary crawling.
Verify: the live test returns the page, the protection log no longer shows challenges for those requests and the logs show the crawler receiving success.
Prevent: treat protection and delivery rule changes like releases, with a live test on key templates afterwards. When a protection rule stands in front of a brand-new site that has never appeared at all, the page on a website not showing on Google starts from that wider symptom.
A template or parameter pattern that fails
Sometimes every error shares one template or one URL ParameterA key and value appended to a URL after a question mark, such as a filter, sort order, session or tracking tag, that may or may not change the page.: a product page that crashes when a field is empty, a listing that fails on a page number beyond the last page, a parameter value the code never expected. The URL parameter audit asks, for every parameter, whether an empty or nonsense value returns an error or a normal-looking page, which is exactly the test that finds this cause.
Fix: make the template handle missing data, and make unexpected values return an honest not-found or the default view rather than an error; apply the audit's class rules so crawlable links stop creating the failing variants.
Verify: request the failing variants again and confirm each returns success or its intended status, then watch the template's error count fall in the logs.
Prevent: add a register row for every new parameter, as the audit describes, and include empty and odd values in template tests. Do not swing to the opposite fault: an error template that answers with success becomes a soft not-found error instead.
Errors during releases, deployments and maintenance
The SEO checks for staging and releases list changed error handling among the ways a release breaks search, with real pages returning errors, and they ask for the Page Indexing report to be read for new server errors in the week after every release. Deployments that restart servers, overnight imports and backups that lock the database, and maintenance windows all produce bursts that line up with the clock.
Fix: roll back or hotfix a release that broke error handling; move heavy jobs away from the hours when crawling and trading peak; deploy in a way that keeps serving pages while new code starts, which is general practice rather than anything Google prescribes.
Verify: errors in the logs no longer line up with deployment or job times, and the golden URLs return their expected status after each release.
Prevent: compare status handling between the Staging EnvironmentA private copy of a website or tool where changes are tested before release, kept behind access control and out of search and production analytics. and production before each release, keep a dated change log so first detected dates can be matched, and let monitoring alert on any golden URL that returns an error. If a release also caused a sharp fall in traffic, the rankings dropped diagnosis starts from the change log.
Old or junk addresses that nobody needs
Crawlers also request addresses no current link creates: old platform paths, malformed links from other sites, garbled parameters. When the server crashes on them instead of answering not found, they appear as server errors. They rank low in triage, but can hide a routing fault that also affects real pages.
Fix: make the server answer unknown addresses with an honest not-found, redirect old addresses that have a genuine equivalent, and remove any that remain from internal links and sitemaps, as the removal rules in the site migrations guide set out.
Verify: a sample returns not found or a single redirect, and those URLs leave the server error examples; the pages on not-found errors and page with redirect cover the lists they join.
Prevent: inventory old URLs from the logs whenever URL patterns change, so every retired pattern gets a deliberate answer.
Prevention: watching errors before Search Console does
The release checks guide makes the key point: Search Console's indexing reports show a problem only after it has started to affect pages. Prevention means seeing errors in your own data first. Keep a monthly summary of crawler requests by template and status code from the logs, as the log file analysis tutorial recommends, so a rise in errors is visible against the previous month. Run a scheduled check of the golden URLs that alerts when any returns an error, routed to someone who can act on it. And keep server performance for crawlers on the same dashboard as performance for people, so capacity problems are caught as response times climb rather than when requests start failing.
| When | Check | Owner |
|---|---|---|
| Every release | Golden URLs return their expected status; logs show no burst of errors | Developer |
| Weekly | Page Indexing report for new or growing server error examples, triaged by template | SEO lead |
| Monthly | Crawler requests by template and status from the logs; crawl stats response time | SEO lead with operations |
| Any change to protection or delivery rules | Live test on key templates; protection log for challenged crawler requests | Whoever owns the rule |
After a fix, test the examples live in URL Inspection before starting validation, which the triage SOP notes the help centre documents as a recheck that takes time. The technical SEO maintenance playbook places the log review, release checks and weekly triage in one routine with a single fix queue.
A sofa retailer finds what its server tells crawlers (illustrative)
Zembrovy Sofas
Zembrovy Sofas, an invented online sofa retailer, sees the server error row in the Page Indexing report climb from 40 to 1,150 example URLs over 6 weeks, while every page loads normally for staff. Grouping the examples by template shows 92% are filtered listing pages. The server logs explain the timing: 81% of the crawler's error responses fall between 01:00 and 03:30, when the nightly stock import locks the product tables and listing queries time out. A second pattern appears at the weekend sale peak, when a bot protection rule challenges crawler requests above 20 a minute and answers them with errors. The team moves the import to a quieter slot, caches the listing query for 15 minutes, allows documented crawlers through the protection rule in the provider's own settings, and removes crawlable links to sort orders. After checking 25 examples live, it starts validation. Over the next 5 weeks the server error row falls to 12 URLs, all old addresses from a previous platform, which are given redirects or not-found responses.
Common mistakes
- Concluding Google is wrong because the page loads. Your request and the crawler's are different requests at different times; read the logs for the crawler's.
- Validating before checking live. A failed validation restarts the wait.
- Allowing anything that claims to be a crawler. Use the protection provider's own setting for documented crawlers rather than a blanket exception for a name anyone can borrow.
- Counting errors instead of sizing them. Errors on a revenue template matter more than many on junk addresses.
- Turning errors into soft errors. An error page that answers with success hides the fault from the report without fixing it.
