Google’s John Mueller answered a question about whether it’s okay to block Googlebot from indexing an age-verification popup that website visitors are required to see. His answer offers some clarity about whether it’s okay to purposely hide some kinds of content from Googlebot and explains how site owners can do it.
Question About Hiding Age Verification Popup
Google requires publishers to allow Googlebot to crawl everything that a user might see but that wasn’t always the case. There was a time in the early days of SEO where SEOs blocked Google from downloading JavaScript and CSS files in order to optimize the crawl budget and encourage better website indexing.
But there came a time when Google began rendering web pages and subsequently required site owners to allow Googlebot to download all resources that a user would see. Some sites that continued to block Google from downloading critical resources temporarily lost their rankings.
So the question about whether it’s okay to block Google from indexing a popup is reasonable.
The SEO asked on Reddit whether it’s okay to use robots.txt to prevent Googlebot from indexing an age-verification popup that a website is legally required to display to visitors.
The person described a hypothetical website that loads an age-verification popup from a separate domain through an API. The question was whether blocking Googlebot from crawling that external domain would prevent the popup from being rendered.
They asked:
“Say I have a website websiteA.com – for legal reasons it has an age verification pop up that sits on a separate domain and is inserted on the page via an API. Lets call it verificationpopup.com
Lets say I wanted to block Google from seeing this – if I added a disallow on verificationpopup.com/robots.txt would that prevent Google from rendering the pop up?
(this is purley a hypothetical and apologies for the stupid Q, I am not very tech minded)”
Mueller Says Blocking This Specific Popup Is Fine
Mueller confirmed that using robots.txt to prevent Googlebot from rendering the age-verification popup would generally work. He also suggested an alternative approach that involves blocking a JavaScript file responsible for loading the popup.
He responded:
“That would generally work. If you’re using a specific JS script file to do this, you could also disallow that specific javascript file via robots.txt. (If you only have one JS file that does a bunch of things, I wouldnt’ do that”
Mueller’s response indicates that Google’s algorithms may be looser than it used to be, at least toward something like this age verification popup. His answer confirmed that site owners can use robots.txt to prevent Googlebot from rendering an age-verification popup, even when the popup is displayed to website visitors.
He returned hours later to elaborate in more detail about two ways to accomplish this, addressing two ways the popup is loaded and which resources the site owner can control.
Two Ways To Prevent Googlebot From Rendering The Popup
Mueller explained that the first approach involves blocking Googlebot from crawling the external domain that provides the age-verification popup.
In the situation of an external domain that’s the source of the resource that needs blocking, the external domain’s robots.txt can disallow crawling and Googlebot will not fetch those resources and therefore cannot use them to render the popup. Mueller said that this is common with resources related to advertising where a JavaScript file responsible for showing ads is blocked by robots.txt.
Mueller explained:
“To elaborate the two options:
the embedded site’s content is blocked by robots.txt. If you embed content from verificationpopup-com on your site (say via iframe or by calling a script on verificationpopup-com, etc), and verificationpopup-com disallows all crawling, then crawlers that follow robots.txt (like Googlebot) would not fetch that content, and would not use it in rendering.
This is how many ads services work. Googlebot wouldn’t crawl these URLs like a separate page, but it would try to fetch them during the course of rendering, and if robots.txt blocks it, then it doesn’t fetch the content, and can’t use it. You need to make sure that your pages still load when those URLs are not fetched, but for something like this, that’s usually not a problem.
Your site uses a specific JS file to load this, and you block that yourself. For example, perhaps you use a “load-verification-popup.js” file to do this. If you block that page with robots.txt on your site, then the script won’t be loaded during rendering (blocked by robots.txt).
In my experience, this is less common, but it’s an option. If the robots.txt on verificationpopup-com is not something you can influence, you’d need to do this, if you wanted to prevent it from being visible/used during rendering.”
So to recap, the second approach involves blocking a JavaScript file that’s used to load the popup that’s on another domain where using a robots.txt is not an option.
Takeaway
The interesting part of Mueller’s answer is that he confirmed a way to deliberately prevent Googlebot from rendering an age-verification popup that human visitors are required to see.
He also explained that the website must continue to load properly when Googlebot cannot fetch the blocked resources. For an age-verification popup like the one described in the question, he indicated that this is usually not a problem.
The overall takeaway is that it’s okay to use robots.txt to prevent Googlebot from downloading and rendering an age-verification popup, even though human visitors are required to see it. Mueller’s answer shows that website owners exercise some control over which resources Googlebot uses during rendering, including resources that are a part of the experience presented to visitors.
Featured Image by Shutterstock/TierneyMJ
