Googlers Gary Illyes and Cherry Prommawin explain how moving to a CDN can unexpectedly increase crawling and overwhelm the origin server, despite eventually improving website performance. They also share that attempts to control crawler traffic can sometimes accidentally block Googlebot and harm Search visibility.
Moving To A CDN Can Cause Googlebot To Increase Crawling
Illyes and Prommawin explained that moving to a CDN can cause Googlebot to increase crawling because faster server responses signal that the site’s infrastructure can handle more requests. Google’s automated crawling systems interpret the improved performance as an indication that more crawling is possible and responds by gradually increasing the crawl rate.
CDN caches are initially empty and the CDN only starts caching when a browser or bot requests a page at which point the request has to be forwarded to the origin server, where the actual website is at.
When Google increases the crawl rate, the initial crawl is handled by the origin server. But now Googlebot is hitting the server more aggressively and it’s this that can overwhelm the origin server, initially.
Gary Illyes explains:
“For one, suddenly the crawling metrics suggests the site can handle a larger amount of crawling. And on the flip side, because the CDN caches aren’t yet populated, the origin server still has to serve the requested pages.
Basically, CDNs take the valve off the hose, so now crawlers are drinking from a fire hose.”
Cherry Prommawin added more details:
“Under normal circumstances, our crawlers operate under an important rule. When a server responds faster, the crawlers assume the infrastructure is healthy and slowly ramp up crawl speed.
…So the crawlers see how fast the server responds, get really excited, and start to crawl the entire site at a higher, higher, and higher speed. But it’s a trap.
People assume a CDN can catch a crawl spike from day one, but …The cache has to be filled up with the assets off the site first.”
Gary Illyes explained that on a first request for a page the CDN has to fetch the page from the origin server first to populate the cache with the content. It’s not until the second request for the page that the cache is populated with the content and begins doing its job.
Illyes followed up:
“…which means your origin server may get slammed by the flood anyway. Now, to be fair, our crawlers take time to ramp up. It won’t happen in a millisecond.
An empty cache means your origin server is still doing all the heavy lifting. Until that cache is populated, your CDN isn’t acting as a shield just yet.”
Takeaway:
A CDN cannot serve an un-cached URL without retrieving it first from the origin server and that only happens after a request for that URL is first made. Crawlers visit large numbers of URLs virtually all the time and can generate a substantial amount of origin traffic before the CDN’s caches are fully populated.
CDN Bot Protection And Accidental Googlebot Blocking
This next issue involves accidentally blocking Googlebot and triggering a reduced search presence because Google can no longer crawl and index URLs from a website. Illyes and Prommawin explain it in the context of a website admin flipping a switch at Cloudflare and inadvertently blocking Googlebot.
This is a real problem, I recall an instance in which a popular website’s massive traffic dried up whenever Cloudflare was activated and would return once they turned it off. They had a devil of a time diagnosing why this was happening and the customer service people at the time were unable to help them. It may have been the case, as Illyes and Prommawin noted, that the admin had turned on a strict bot filter that inadvertently blocked Googlebot.
Illyes offered an example of an admin doing exactly that:
“They look at the spiking traffic, assume they are being aggressively scraped by botnets, and naturally, they panic.”
And what sometimes happens next is that an admin might turn on “blanket security rules” that block Googlebot.
Prommawin offered more details:
“In panic, one might log into their CDN dashboard and turn on the setting to block bots entirely.
…If the CDN detects automated requests, it intercepts them with a CAPTCHA challenge, asking them to click pictures of traffic lights to prove they’re human.
…It sees automated requests, flags it as a bot, and then intercepts with a CAPTCHA. And since our crawlers cannot identify a traffic light, they fail the test.”
What eventually happens with prolonged Googlebot blocking is that Google will begin dropping the pages from the index.
Prommawin suggested that site admins should configure their CDN to allow known and trusted crawlers. This is a method called whitelisting where only crawlers that are on the “allowed” list can get it and all other crawlers cannot.
They also recommend the use of CDN-level rate limiting of Googlebot if the requests on the origin server lead to downtime which can negatively impact legitimate human traffic to the website.
Pre-Warming CDN Cache Is Recommended
When a site initially starts out with a CDN, the CDN does not have a cache of a website’s content. The cache doesn’t get loaded until it’s requested by a browser or crawler visit.
Cache pre-warming is the process of requesting content from a site so that the content gets loaded onto the CDN. This ensures that the content is already loaded onto the CDN before users request it, which ensures that the content is delivered quickly without the CDN having to first retrieve it from the origin server (the server where the actual content files exist).
Illyes recommends “pre-warming” the cache by programmatically requesting URLs from it so that thE cache is populated in a controlled manner.
“And to survive augmenting your site with a CDN, see if you can pre-warm the caches by programmatically fetching your top URLs through the CDN before you open the floodgates to search engines.”
Takeaways
Google’s main point is that moving to a CDN can create an unexpected combination of faster crawling and increased origin-server demand. That in turn can lead to unintentionally setting aggressive security restrictions that prevent Googlebot from accessing a website.
Three especially important points for SEOs and site owners:
1. Using a CDN can trigger an increase in Googlebot crawling when it detects faster server responses, even though the CDN’s caches have not yet been populated, which in turn means an increase in requests for URLs from the origin server.
2. A CDN migration can inadvertently cause pages to disappear from Google Search due to aggressive security settings intended to stop malicious bots can also lock out Google’s crawlers.
3. Google recommends pre-warming caches, allowing trusted crawlers, and temporarily controlling crawling when server demand becomes excessive.
Read Google’s new CDN documentation here.
Watch the Google podcast here:
Featured Image/Screenshot