Google’s John Mueller answered a question about at what point will 500 error response codes cause Googlebot to crawl content less often. Mueller explained scenarios where 500 response codes won’t affect crawling and when they may cause pages to drop from Google’s index.
Accidentally Blocked Googlebot and Caused 500 Errors
The person asking the question explained that their content delivery system was set up to block bad bots while allowing actual human users and legit bots to browse the site.
Everything was working fine until they updated their “server monitoring suite” and apparently the previous bot blocking filters and settings weren’t carried over to the new system.
Afterward they noticed that Google Search Console began listing 500 error response codes in response to Googlebot crawling.
The person finished explaining what happened then asked the question:
“The question came up from our technology team whether this represented real user impact and why we would look specifically at Googlebot and not real user metrics to prove that there’s an issue here.
So, given that context, I have a few questions.
The first is just to get the technology concern out …of the way.
From your perspective, how does Googlebot view 500 series errors and could you give any clarity on established …thresholds at which point Googlebot will …crawl source content …less, based on those errors?”
What is a 500 Error Code?
Servers communicate a variety of status response codes to browsers and bots. These response codes communicate, for example, whether the request for a page was successful, redirected to another page, blocked or unsuccessful.
- The 200 response code communicates that the request for a web page was successful.
- The 500 response code means that the request for a web page was unsuccessful.
The formal name of the 500 response code is:
"500 Internal Server Error"
There are other versions of the 500 response code that communicate a specific reason why the request for a web page was unsuccessful.
The 500 response code is somewhat generic in that it simply means that the page request was unsuccessful. But it often means that the server was down but not always.
Google Search Console and 500 Error Responses
It’s a good idea to monitor the Search Console for 500 Error Responses because they are a strong indicator that something is wrong.
Sometimes it could mean that the server is under heavy load, for example from too many bots and because of that it is unable to serve web pages.
In this case it appears that the problem was due to a misconfiguration in the Content Delivery Network (CDN) that inadvertently blocked Google from accessing web pages.
How Googlebot Indexing is Affected by 500 Errors
Mueller provided a helpful explanation of the steps Google takes in response to 500 error codes and how eventually continued errors could lead to those web pages being dropped from Google’s search index.
Google’s John Mueller responded:
“We don’t have any strong thresholds on that.
But essentially what happens with 500 errors is we’ll try to retry them.
And if we continue to see …the 500 errors then we will …slow down crawling.
And if we continue to see that there are 500 errors then we will drop those URLs from the index.
So that’s something if every now and then individual pages have a 500 error, it’s like no big deal.
We will retry them, they’ll remain indexed and the next time we retry them that’s fine.”
Next, Mueller explains that the reason Googlebot slows down the crawling is in case Googlebot itself is causing the server to overload.
“But if a large part of a site consistently has 500 errors and we might assume that maybe we’re causing the problem and we’ll slow down crawling of the whole site and at some point we’ll say well, it looks like these pages are really gone, we’re going to drop them.
So that’s essentially the effects that you would see there and if you’re talking about a large site and wondering like what percentage of 500 errors is okay… I don’t know.
My feeling is if you’re seeing something more than one percent then that sounds like something is kind of broken and probably would be something where we would start to slow down.
But I don’t think we have any hard thresholds where we’d say, this many requests and this many errors means this much slowing down.”
Monitor Search Console for 500 Errors
It’s up to every publisher to decide what their daily routine is. For some people checking the search console once a week is enough. Some may find it reassuring to check Search Console once a day in order to respond to unforeseen circumstances.
In any case, 500 error codes is something to monitor Search Console for. As John Mueller mentioned, large-scale 500 errors could be a flag that something is wrong and in need of immediate attention.
Citations
World Wide Web Consortium (W3C) List of Server Response Codes
How Does Google Respond to 500 Response Codes?
Watch the discussion at the 13:48 minute mark
