In a Webmaster Hangout, Google’s John Mueller answered the question of how to stop Google from crawling and indexing a staging server.
What is a Staging Server?
A staging server, or staging site, is a section of a server or website that is used to test a new version of a website.
If a publisher is testing a new web design, they can create a subdomain and test the new website design there.
Traditionally, the most common way to block Google from indexing a staging site was to create a robots.txt file that keeps Google from crawling the staging site.
Google’s John Mueller offered the best solution.
How to Stop Google from Crawling a Staging Server
John Mueller answered that the best way to stop Google from crawling and indexing the staging site was to block the web pages with authentication.
This is how John Mueller answered the question:
“There are multiple ways… people do it in multiple ways. I think that the important part is that you don’t link to it. Because if we don’t find it then we can’t crawl it. But sometimes that still happens. “
Mueller’s comment is a reference to using Robots.txt to block crawling of the web page. If Google discovers a link to a blocked web page, Google will crawl the URL.
John then goes on to recommend a better way to block both crawling and indexing of a testing server.
“Ideally, what you would want to do is provide some kind of server side authentication on the server so that normal users when they go there they would get blocked from being able to see the content; that would include GoogleBot.
And you can do that on an IP address basis, you can do it with a cookie, you can do it with normal authentication on the server.
Anything where you have to prove that you’re the right person and you can actually look at that content.
I think that’s generally the best approach for staging servers…”
John Mueller then went on to state the reason why a staging site with authentication is the best approach.
“…it’s something that means you don’t have to change the normal settings on the site itself, in particular robots.txt but also noindex meta tags for example.”
What he means is that by using a password protected solution you will eliminate the possibility of making a mistake on your live because of a mistake with using the noindex meta tag or the robots.txt solution.
It’s more effective to use authentication to block Google and others from seeing the staging site.
How to Create a Password Protected Staging Site
There are multiple ways to create a password protected staging site. If you are on a dedicated server, you can use your control panel to create a password protected subdomain (read the online manual for your control panel or ask your web host for help).
If you’re on WordPress and on a shared host, you can use an external staging server like WP StageCoach or if you don’t want to pay you can use a plugin like WP Staging.
WP Stagecoach makes it super easy to stage a site in a password protected environment. When you’re ready to make the site live WP Stagecoach makes it as easy as clicking a button to move it to your live hosting environment.
I have used WP Stagecoach and can personally attest to the ease of using that platform.
I have also used the WP Staging plugin.
WP Staging offers a premium version that allows you to easily migrate the test version of the site to the live section.
I haven’t used the premium version but I did use the free version and found it to be a great solution.
If you are confident with WordPress then you can simply use WordPress’ native site export function to export the site theme and styles then import them into the live site. It’s a little trickier in that you have to read the manual to make sure you’re doing it right.
Of course, before migrating a site, be sure to back up your website. UpDraft Plus is my choice for a WordPress backup plugin. It’s the only one I’ve ever used because I’ve never had a need to test any other. I can’t say it’s the best but it sure works like the best for me in that it’s easy to use.
Returning back to John Mueller, I agree with his advice. It’s a good idea to stage the site behind a password protected directory or subdomain.
Watch John Mueller’s answer here:
