I was recently approached by someone outside of Google with information from Google about the April 2019 index outages. The Googler shared this information at the May 2019 Sydney Search Conference.
A second person, totally unknown and unrelated to the first person, related a similar account. The story they shared reveals what caused Google’s Search Index outages in April 2019.
The following is what the two witnesses remembered about the closed door session.
How Google Shared the Information
The first person related that a Googler shared the information recently at a Sydney Search Conference.
“We were in a “closed door” session with a bunch of other SEOs…”
He then related what he remembered the Googler said:
“…he… wrote down three things.
1. Developer mistakenly deletes the entire Google web index on all servers.
2. Google Rendering Team rolls out an update just before Google I/O to adjust font sizes (which many people saw) and accidentally introduces a bug that redirects millions of pages to Facebook.
3. Not able to fix second problem because of lockdown after the first. “
I want to stop right here and note that the first two points listed above are how the first person remembered it. What was actually said may have been different. A second person offered a clearer explanation of what was said.
The first person related more of what was shared:
“The Googler later added that a developer in Zurich deleted the Google index (not the live on, but one waiting in the wings) from all 18 distributed Google servers, so they locked everything down while they were trying to figure out what happened.
When the second issue happened, they were able to figure it out soon enough, but couldn’t fix it right away.”
I asked this person if he thought Google’s indexing issues in May 2019 were from a similar human error:
“So, it was really interesting to see Google have another very similar looking issue this week… no idea if it was for the same reasons.”
Account Confirmed by a Second Person
Hearing something like this from one person, no matter how credible they are, it’s still just one person. But I heard this from two unrelated people. This lends credibility to the story that a Googler shared what really happened.
This second person has no connection to the first person. Neither of these people know each other.
More Details About Human Error at Google
The second person disagreed that the Googler said the entire index was deleted.
Point 1 – The Index Was Not Deleted
The second person had a different recollection of what was said about the server being deleted.
“The entire index is an overstatement. It would be misleading to say that. If that had been the case the SERPs would have fallen apart in some way. And we all would have noticed.”
The details are all about the caching time on one shard the engineer wanted to update before pushing to live – 2 seconds if memory serves – that would have been fine for one shard, but it propagated to the entire set, and the system pushing fresh data into the index shards could not keep up (that alone is very interesting from mechanisms within google perspective 🙂
So to avoid that empty set of shards being pushed to the live index, they froze the existing live index in place while they investigated. That means the live index did not change for several days.”
Point Two – Google Did Not Redirect URLs to Facebook
The second person had a different recollection of what was said about Facebook the Facebook URL issue.
“They didn’t redirect people to Facebook. The URL showed Facebook but the link sent users to the correct page. It was 200 million green links that showed Facebook, but continued to have their Href tag correctly assigned.”
The differences between the two accounts vary, but the general outlines match.
Point Three – Google’s Index Was Locked Down
The second person agrees with what the first person related about point three, about not being able to fix a second outage because of the lockdown of the index.
“Third point = yes indeed. The index is frozen and they don’t want to mess with it.”
Are the May 2019 Index Outages Similar to the April 2019 Outages?
The second person offered this opinion:
“My feeling here is that what is important to get across is that April was a “special month” that will never be repeated and that May is probably suffering knock on effects.
An issue that size and that close to the bone, with a freeze in place is always going to create shockwaves / knock on effects.”
That’s his opinion about the May 2019 Google Index Outages, and it was shared as that, an opinion.
Whatever the case, this is the first information to come out of Google that identifies what is going on. It turns out that the reason Google’s been suffering outages to it’s index may in fact be human error.
Images by Shutterstock, Modified by Author
