Why Google Search Console Reports 5XX Server Errors

A publisher asked Google’s John Mueller via Twitter about Google Search Console (GSC) reporting 5xx error on their site.  This article explores possible reasons why GSC will report having received 5xx errors and solutions.

The tweet said:

“I am getting server error (5xx). My new posts are not indexing. Any suggestion to FIX this?”

The publisher then posted a screenshot of the GSC message:

Screenshot of the Google Search Console message of a 5xx error.

GSC links to a help page that says this:

“This section of the report describes whether the page can be indexed by Google.

However, a positive result is no guarantee that it will appear in Search results.

Appearance in Search results additionally requires that the page and its structured data conform to quality and security guidelines.

The URL Inspection tool doesn’t take into account manual actions, content removals, or temporarily blocked URLs.”

The part of that help page that is relevant to the above user is “whether the page can be indexed by Google.”

Google’s John Mueller responded to that tweet by suggesting it’s not Google, it’s the site that has an issue.

That’s an error on your website which you need to fix.

— 🍌 John 🍌 (@JohnMu) July 27, 2020

This is What Causes 500 Errors

500 response messages are what the server sends when there is a critical error that stops the website or web page from rendering.

It sometimes indicates a coding error in the CMS that is causing the page or the entire site to not render. PHP code can be fussy and sometimes something as small as leaving out a comma is enough to break the site.

Which is why it’s always a good idea when editing code to leave a copy of the original file (renamed) on the server and to save a backup copy of the original un-edited file on your computer before editing a different version of that file.

That way if things go bad you can always upload the original and start again.

Other sources of 500 errors are:

• Misconfigured .htaccess files
• Core CMS config files have wrong information such as database information
• PHP version is incompatible with CMS
• Code in CMS template was altered in a way that added a mistake

The above are all errors I’ve made at one time or another that caused a 500 error.  It happens to everyone who edits website code. Nobody’s perfect.

What If Google Reports 500 Error But Site Works Fine for You?

There are situations where Google may report a 5XX error and when you visit your site it works fine.

In best case scenarios, this could be because your web host was updating.

But… the 5xx error that shows to Google and not to you can often be a symptom of an issue with your server environment.

This can be one of the downsides of a shared hosting environment momentarily going down at the moment when Google visits your site.

What sometimes causes that kind of error is a heavy load of bot visitors, either to your site or to the entire shared server environment.

This typically happens at night when both Google and hacker bots/scraper bots hit the site at the same time, which causes an under-powered server to temporarily start handing out 5xx error responses.

If a hacker bot or scraper bot are hitting your server hard the shared server can start serving 500 errors because it’s run out of allocated resources.

This can affect large websites as well as resource intensive sites like forums. So it could be a wake up call to upgrade your web hosting.

In cases of a Virtual Private Server (VPS) environment or even a dedicated server, the 500 error can be due to improper set up by the server admin.

An improper set up can involve issues such as inadequate server resources directed to websites hosted on the VPS. An example is inadequate memory allocation.

I asked Scott Polk, Senior Search Analyst at Bruce Clay, Inc., about this and he suggested that a possible issue is data lag.

Scott said,

“But that does seem more like an excuse.

I suspect they have crawling issues on top of data lag between the index and the tools.

For example, why are pages not showing as indexed in GSC, but showing in a site: search?”

500 Server Response Codes

If Google Search Console reports a 5xx server response code but the site works fine for you, then it could be crawling issues caused by an unresponsive server, particularly at night.

To test your server, you may want to give your site a crawl with a tool like Screaming Frog and keep raising the amount of user agents crawling the site simultaneously.

Alternatively you can also use a free tool like Xenu Link Sleuth to run a lot of crawlers on the site at the same time.

If the tools report 500 error responses, then that could be a sign that the server can’t handle high levels of traffic. There are many reasons for this ranging from the less than optimal settings in the CMS, improper resource allocations at the server level, an overloaded server or many other reasons, usually having to do with your site or the server.