There is growing evidence that Google is in the middle of undergoing an infrastructure update. Past history suggests that improvements to the algorithm follow once an infrastructure update has finished.
[UPDATE]
In early May 2019, it was revealed in a secret closed door session by a Googler that user errors were the cause of Google’s index outages.
Read: April 2019 Google Outages Due to Human Error
On May 28, 2019, Google announced a major update in the form of transitioning to 100% mobile indexing for new websites.
Read: Google Moves to Mobile-First Indexing by Default for New Websites
According to Google’s official announcement:
“Googlebot primarily crawls and indexes pages with the smartphone agent going forward.
Starting July 1, 2019, mobile-first indexing is enabled by default for all new websites (new to the web or previously unknown to Google Search). “
What is an Infrastructure Update?
Infrastructure updates can help speed up calculations or indexing. The Caffeine Update is an example of an infrastructure update.
According to Dave Davies’ overview of the Caffeine Update:
This new system allowed Google to crawl and store data far more efficiently.
In fact, by Google’s own account they were able to not only increase their index but provide far fresher results (50 percent fresher by their estimates).
Another example of an infrastructure update is the 2017 rollout of Espresso. According to Google:
“Espresso allows us to maintain performance and availability in a way that is not possible with existing router-centric Internet protocols. This translates to higher availability and better performance through Google Cloud than is available through the Internet at large.”
Read Google’s official Espresso announcement here
The Elephant is Officially in the Room
- Google “accidentally” removed thousands, maybe millions of URLs from it’s search index
- Google’s cache is one month out of date
- Google’s search indexer, GoogleBot, was upgraded from a 4 year old version of Chrome capabilities to the newest version.
The phrase Elephant in the Room is a metaphor for something that is too big to ignore. It’s pretty obvious that something is going on with Google’s search index.
We see it.
Google knows we see it.
Obviously, something is going on with indexing.
It looks like an elephant called Infrastructure Update.
This is an ecosystem where publishers support Google and Google supports publishers.
It would be reassuring if Google was more informative about what’s going on when the wheels start falling off their search engine in the manner as has been occurring since April 2019.
March 2019 Update Was Big
I received advance information that was leaked from Google that an update was coming. The information was that the March 2019 update would be one of the biggest in years. I reported that here.
Some were of the opinion that the impact of this algorithm did not seem to reach many websites. But that may have been misinterpreting what the Googler meant when they said it was going to be big.
A change to infrastructure is big. It might not immediately affect website rankings. But the improvements introduced by infrastructure updates allow the advancement of algorithmic processes.
One Month Later Wheels Fall Off Google’s Index
Google’s search index suffered an unprecedented meltdown one month after Google’s March 2019 update. I have been working on the web for 20 years and I have never experienced a breakdown in a search engine as happened one month after Google’s “big” update in March 2019.
“Google is working on fixing a widespread technical issue that has led to web pages getting removed from Google’s index.”
What happened is that many web pages went missing from Google’s index. The result is that many websites lost rankings because their web pages were no longer in Google’s index.
This is an important event because it means that something significant happened in Google’s infrastructure that caused it to lose web pages.
That is an infrastructure issue. It followed what was leaked to be a “big” update. It seems obvious to me that Google is right now undergoing a major infrastructure update. This update has affected web publishers yet Google has not announced that an update has been happening.
I believe there is a connection between Google’s “big” March 2019 update and the unprecedented indexing issue that followed a few weeks later in April.
Google’s Indexing Bot Has Updated
Another sign that Google is undergoing an infrastructure update is that GoogleBot, Google’s crawling software, has undergone a major update.
GoogleBot’s ability to download and render websites has lagged behind Chrome for the past four years. GoogleBot has been based on Chrome 41, a browser that was announced in March 2015.
Chrome is currently at version 74.
This means that until recently, GoogleBot was unable to see the web in the same way as people who were using modern browsers.
This changed in 2019, when Google announced that GoogleBot will henceforth be based on the most modern version of the Chrome browser.
This change to how Google indexes the Internet happened two months after Google’s March 2019 update, and one month after Google’s search index broke.
GoogleBot’s upgrade represents a major update to how Google crawls the Internet. It’s a very big deal. GoogleBot has jumped from the capabilities of Chrome 41 to Chrome 74.
It’s notable that this change to how Google crawls the Internet has come amidst the recent troubles at Google related to it’s cache and it’s index.
Infrastructure Update Preceded Penguin and Panda
Two of the most important updates to Google’s algorithm are called Penguin and Panda. They affect how links and content are ranked.
The Caffeine Update was rolled out in 2010. It was an infrastructure update. It helped Google handle massive amounts of data and calculations. This ability enabled Google to improve it’s algorithm by introducing the content focused Panda update in 2011 followed by the links focused Penguin update in 2012
I speculate that the Caffeine update helped Google make it’s algorithms function more efficiently, allowing it to eventually deploy Panda then Penguin.
Another infrastructure deployment, called Espesso, happened around 2015. This unannounced update was related to helping Google push content through it’s network more efficiently. Espresso uses Software-defined networking (SDN) in order to help provide content to users as fast as possible.
Read about Espresso here, Taking the Edge off with Espresso: Scale, Reliability and Programmability for Global Internet Peering
What Could a Google Infrastructure Update Be?
One can only speculate as to what is going on at Google. It’s obvious that Google’s infrastructure is changing and at times broken.
Google’s being quiet about what’s going on is awkward.
Google depends on websites. Businesses depend on Google. Web pages have gone missing, the cache is one month out of date, GoogleBot is new and improved… Does that sound like a major infrastructure update to you?
Images by Shutterstock, Modified by Author
