Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

SEO › Troubleshooting

Server Errors in Search Console

Why Search Console reports server errors on pages that load in a browser: capacity under crawling, bot protection, failing templates, releases, old URLs.
Troubleshootingintermediate10 min readUpdated

First, reproduce the error and find when and where it happens

A server error is a status code family that says the server, not the visitor, failed. The crawling and indexing guide states the consequence: a server error or a timeout slows crawling and, if it persists, drops the page. So an error that has "gone away" when you check still matters if it keeps coming back.

Follow the reproduce step of the Search Console error triage SOP. Export the example URLs listed under the server error reason in the Page Indexing report, run the live test in URL Inspection on several of them, crawl the same URLs with your own crawler and load them in a browser. If none of them fails now, the SOP's warning applies: the problem may be intermittent, and the server logs have to be read before assuming it is fixed. The Search Console tutorial explains each report; the wording of the reasons lives in the Search Console help centre.

Then give the examples two labels: template and time. Map each URL to its page type, as the log file analysis tutorial does with its template list, and note the first detected date against the release history. In the logs, which that tutorial describes as the only record of every request a crawler actually made, count crawler requests by status code within each template and by hour of the day. The crawl stats report adds Google's own view of response time, host status and responses by type, as the discovered, currently not indexed page describes.

Pattern in the examples and the logsWhat it points to
Errors cluster at busy hours or on heavy listing templates, with slow responses just before themThe server cannot keep up with crawling
Crawler requests fail while browser requests to the same URL at the same moment succeedA protection or delivery rule treats the crawler differently
One template, or URLs carrying one parameter, fail at any hourCode that breaks on certain data or values
Errors start on a release date, or arrive in short bursts at deployment or job timesReleases, deployments or maintenance
Errors sit on addresses nothing on the site links toOld or junk URLs the server mishandles

Once a pattern is visible, use five whys to push past the first answer. "The listing page timed out" is a symptom; asking why the query was slow, why it was slow only at night and why a heavy job ran then usually ends at something a developer can change. The SEO page checker reads the status of one live URL, a quick second opinion that proves little about an intermittent fault.

Diagnosis table

Diagnosis

SymptomCheckLikely causeFix
Errors cluster at busy times on listing or search templates; response times rise firstLogs by template and hour; crawl stats response timeA server that slows or fails under crawling load, especially on heavy listing pagesRemove the bottleneck, cache heavy pages and cut needless crawlable combinations
Crawler requests fail while the same URL loads in a browser; live test failsProtection layer logs; live test against a browser requestBot protection, a firewall or a delivery network rule answering crawlers with errorsAllow documented crawlers through the provider's own settings and relax rate rules
Every error shares a template or a parameterGroup examples by template and parameter; request variants with empty or odd valuesOne template or parameter pattern that fails on the serverMake the code handle missing data and unexpected values
Errors begin on a release date or recur at deployment and job timesRelease log against first detected date; logs by minuteErrors during releases, deployments and maintenance windowsRoll back or hotfix; move heavy jobs and keep serving during deployments
Errors on old paths, malformed links or junk parametersIs the URL linked, in the sitemap or needed by anyone?Errors on old or junk addresses that nobody needsAnswer them honestly: redirect to an equivalent or return not found

Fix, verify and prevent each cause

A server that slows or fails under crawling load

Google's large-site guide to Crawl Budget describes a crawl capacity limit that rises when the site responds quickly and falls when it slows or returns server errors, as the crawl budget entry summarises. The pages that fail first are the expensive ones: category and search listings that run heavy queries for every filter and sort.

Fix: find the bottleneck with the host or developers, usually slow database queries on listing templates or missing caching, and stop exposing filter combinations nobody should crawl, following the tiers in faceted navigation and index control.

Verify: the logs show the listing template answering crawler requests with success and faster responses, the crawl stats report shows fewer server errors, and the live test passes on the old examples.

Prevent: monitor response time and error rates for crawler traffic beside user traffic, and test heavy templates against the catalogue's growth before it happens. If new URLs are also queuing unfetched, the discovered, currently not indexed page covers that side of the same problem.

Bot protection, a firewall or a delivery network rule

Protection layers that challenge unfamiliar visitors or cap request rates can answer crawler requests with errors while staff, who are recognised, see a working site. The pattern is distinctive: the live test in URL Inspection fails while a browser request to the same URL at the same moment succeeds.

Fix: read the protection layer's own log for the rule that matched, use the provider's setting for allowing documented search engine crawlers rather than hand-written exceptions, and relax rate rules that fire on ordinary crawling.

Verify: the live test returns the page, the protection log no longer shows challenges for those requests and the logs show the crawler receiving success.

Prevent: treat protection and delivery rule changes like releases, with a live test on key templates afterwards. When a protection rule stands in front of a brand-new site that has never appeared at all, the page on a website not showing on Google starts from that wider symptom.

A template or parameter pattern that fails

Sometimes every error shares one template or one URL Parameter: a product page that crashes when a field is empty, a listing that fails on a page number beyond the last page, a parameter value the code never expected. The URL parameter audit asks, for every parameter, whether an empty or nonsense value returns an error or a normal-looking page, which is exactly the test that finds this cause.

Fix: make the template handle missing data, and make unexpected values return an honest not-found or the default view rather than an error; apply the audit's class rules so crawlable links stop creating the failing variants.

Verify: request the failing variants again and confirm each returns success or its intended status, then watch the template's error count fall in the logs.

Prevent: add a register row for every new parameter, as the audit describes, and include empty and odd values in template tests. Do not swing to the opposite fault: an error template that answers with success becomes a soft not-found error instead.

Errors during releases, deployments and maintenance

The SEO checks for staging and releases list changed error handling among the ways a release breaks search, with real pages returning errors, and they ask for the Page Indexing report to be read for new server errors in the week after every release. Deployments that restart servers, overnight imports and backups that lock the database, and maintenance windows all produce bursts that line up with the clock.

Fix: roll back or hotfix a release that broke error handling; move heavy jobs away from the hours when crawling and trading peak; deploy in a way that keeps serving pages while new code starts, which is general practice rather than anything Google prescribes.

Verify: errors in the logs no longer line up with deployment or job times, and the golden URLs return their expected status after each release.

Prevent: compare status handling between the Staging Environment and production before each release, keep a dated change log so first detected dates can be matched, and let monitoring alert on any golden URL that returns an error. If a release also caused a sharp fall in traffic, the rankings dropped diagnosis starts from the change log.

Old or junk addresses that nobody needs

Crawlers also request addresses no current link creates: old platform paths, malformed links from other sites, garbled parameters. When the server crashes on them instead of answering not found, they appear as server errors. They rank low in triage, but can hide a routing fault that also affects real pages.

Fix: make the server answer unknown addresses with an honest not-found, redirect old addresses that have a genuine equivalent, and remove any that remain from internal links and sitemaps, as the removal rules in the site migrations guide set out.

Verify: a sample returns not found or a single redirect, and those URLs leave the server error examples; the pages on not-found errors and page with redirect cover the lists they join.

Prevent: inventory old URLs from the logs whenever URL patterns change, so every retired pattern gets a deliberate answer.

Prevention: watching errors before Search Console does

The release checks guide makes the key point: Search Console's indexing reports show a problem only after it has started to affect pages. Prevention means seeing errors in your own data first. Keep a monthly summary of crawler requests by template and status code from the logs, as the log file analysis tutorial recommends, so a rise in errors is visible against the previous month. Run a scheduled check of the golden URLs that alerts when any returns an error, routed to someone who can act on it. And keep server performance for crawlers on the same dashboard as performance for people, so capacity problems are caught as response times climb rather than when requests start failing.

WhenCheckOwner
Every releaseGolden URLs return their expected status; logs show no burst of errorsDeveloper
WeeklyPage Indexing report for new or growing server error examples, triaged by templateSEO lead
MonthlyCrawler requests by template and status from the logs; crawl stats response timeSEO lead with operations
Any change to protection or delivery rulesLive test on key templates; protection log for challenged crawler requestsWhoever owns the rule

After a fix, test the examples live in URL Inspection before starting validation, which the triage SOP notes the help centre documents as a recheck that takes time. The technical SEO maintenance playbook places the log review, release checks and weekly triage in one routine with a single fix queue.

A sofa retailer finds what its server tells crawlers (illustrative)

Illustrative example — not a real result

Zembrovy Sofas

Zembrovy Sofas, an invented online sofa retailer, sees the server error row in the Page Indexing report climb from 40 to 1,150 example URLs over 6 weeks, while every page loads normally for staff. Grouping the examples by template shows 92% are filtered listing pages. The server logs explain the timing: 81% of the crawler's error responses fall between 01:00 and 03:30, when the nightly stock import locks the product tables and listing queries time out. A second pattern appears at the weekend sale peak, when a bot protection rule challenges crawler requests above 20 a minute and answers them with errors. The team moves the import to a quieter slot, caches the listing query for 15 minutes, allows documented crawlers through the protection rule in the provider's own settings, and removes crawlable links to sort orders. After checking 25 examples live, it starts validation. Over the next 5 weeks the server error row falls to 12 URLs, all old addresses from a previous platform, which are given redirects or not-found responses.

Common mistakes

  • Concluding Google is wrong because the page loads. Your request and the crawler's are different requests at different times; read the logs for the crawler's.
  • Validating before checking live. A failed validation restarts the wait.
  • Allowing anything that claims to be a crawler. Use the protection provider's own setting for documented crawlers rather than a blanket exception for a name anyone can borrow.
  • Counting errors instead of sizing them. Errors on a revenue template matter more than many on junk addresses.
  • Turning errors into soft errors. An error page that answers with success hides the fault from the report without fixing it.

Frequently asked questions

Why does Search Console report a server error on a page that loads fine for me?

Because your visit and the crawler's visit are different requests. The crawler may arrive at a busy hour, during a deployment or an overnight job, in a burst that a protection rule treats as an attack, or at an address variant you never open. Run the live test in URL Inspection, then read the server logs for the crawler's requests to that URL; the logs show what the server actually answered and when.

Can server errors make pages drop out of Google?

The site's crawling guide notes that a server error or a timeout slows crawling and that a persistent one drops the page. Occasional errors mostly cost crawling; errors that keep returning on the same pages put them at risk. That is why errors on valuable templates are fixed first, whatever the count in the report.

Should I click Validate Fix as soon as I have changed something?

Only after the live test in URL Inspection and your own checks show the examples answering properly. The Search Console help centre documents validation as a recheck that takes time, and a validation that fails because the cause was not fully fixed starts the wait again. Check intermittent causes in the logs over several days before you start it.

Terms used on this page

Tools to try

Frameworks

Sources

  1. Page Indexing report — Google Search Console Help
  2. Large site owner's guide to managing your crawl budget — Google Search Central
  3. URL Inspection Tool — Google Search Console Help

Related