Google’s John Mueller and Martin Splitt discussed what happens when spammers abuse website search functions to create thousands of spammy search pages. They said that Google begins to treat those pages as hacked.
Website Search Spam And Quality Issues
Virtually every website has a search function, and something that’s not clearly understood is that using the search bar generates a search results page that contains the search query. What’s surprising is that it’s normal behavior for these web pages to generate a URL that can be used to access that search, creating a situation where a spammer can use the search bar to generate thousands of web pages containing mentions of a spammy brand, the URL, and spam-related keywords.
Google’s Mueller said that website search spam can become a quality issue if the spam pages can be indexed or are indexable. Mueller used the example of entering text into a search box to generate a web page that contains that spammy text, which is something that can happen.
Mueller says search spam can become a quality issue:
“There is one place where you could run into quality issues though with search results pages. Namely, if you let people search for things that are totally irrelevant to your website and your search results page includes those terms on those search results page and is indexable.
…And it’s not so much that someone has hacked your website to do this because your website is doing that freely and basically saying like, oh, you search for photos, here’s a photo. But because it’s accessible for any search term that comes up, it’s suddenly a liability. It’s more like a vector for other people to spam.”
That last bit about a search box becoming a vector for spamming is correct, if the web page allows those auto-generated pages to be indexed.
Google May Flag Internal Search Pages As Hacked
The other useful information that Mueller and Splitt shared is that they flag these kinds of spammed pages as if they are hacked, explaining that these kinds of pages can show up in Google’s search results.
Mueller explained:
“And we’ve seen that happen that people do that at scale. They will try to recognize common CMSs that don’t have their search results pages blocked and go off and link to thousands of sites with millions of pages, all with maybe some adult terms and a phone number or pharmaceuticals and a phone number or something else and a telegram address or some other kind of contact mechanism where the goal is not so much that people go to your site and kind of see your photos, but rather that in the search results they’ll see for these pharmaceuticals call this number.
And sometimes that does show up for these kind of queries. And when we see that happen, we might flag that as hacked. So in Search Console, you might see that as something that is flagged as hacked.”
Google Can Catch Spammed Search Result Pages Algorithmically
Mueller also explained that Google can algorithmically catch these kinds of pages and block them from showing up in search results. From the site owner’s side, this can feel like a win because Google handled it. Mueller cautions against assuming that the problem is taken care of.
How To Prevent Search Box Spam
Google’s Mueller and Splitt recommended two approaches for preventing search spam from becoming an issue:
- Prevent crawling
- Prevent indexing
Prevent Crawling
The first recommendation is to use a robots.txt to prevent Google and other search engines from crawling the auto-generated search box web pages.
Mueller and Splitt list four reasons to block indexing with robots.txt:
- Google doesn’t need to crawl search pages.
- It prevents infinite crawling.
- It reduces server load.
- It prevents crawl budget waste.
Regarding infinite crawling, Martin Splitt said that some implementations of search boxes can turn into an infinite “crawl space” where the search box keeps autogenerating web pages through a “did you mean” feature that’s triggered in response to Google’s crawling those search pages.
Splitt explains:
“So we learned a bunch of stuff about search results on websites. So they can be infinite crawl spaces because we can basically generate pages upon pages of these and maybe we even link to like, did you mean, and then we create even more that the crawler sinks into. And it sounds like they’re relatively easy to get rid of with robots.txt and or no index.”
The infinite crawl space issue is also linked to increased server load and putting stress on the website crawl budget.
John Mueller explained how infinite crawling, server load, and crawl budget waste are linked:
“And all of that basically means we find, or we could potentially find an infinite number of pages on your site. And I know some people are like, wow, if I had an infinite number of pages in Google, then I would be king.
But having an infinite number of pages known to Google is not a good thing because Google will try to crawl all of those pages. And you can imagine what happens when we see, I don’t know, 100 million pages that are new from Martin Split will go off and try to crawl those. So that’s something where then suddenly crawl budget becomes a question and your server load and your server’s like, oh my gosh.”
Those are all the reasons why Mueller and Splitt recommend using robots.txt to prevent crawling of search box pages. More precisely, they say to use as broad and general a robots.txt rule as possible that can catch all variations of auto-generated search box pages.
Noindex Directive
Mueller and Splitt mention using the robots noindex directive but say that robots.txt is both easier and the cleanest way to handle search box web pages.
Watch Episode 113 of Search Off The Record
Featured Image by Shutterstock/Tirachard Kumtanom
