Google’s Gary Illyes issued a reminder this week that sending automated searches to Google is against the Terms of Service. This is not a new rule, but it appears that the issue has been on the minds of Google’s Webmaster Trends Analysts as of late.
Friendly reminder that sending automated queries to search engines are generally against their Terms of Service.
— Gary Illyes ᕕ( ᐛ )ᕗ (@methode) December 14, 2016
Also this week, Google’s John Mueller had a watchful eye on a software company that was promoting its ability to conduct automated queries.
@greenlaneseo Is this a blackhat tool or does it abide by the webmaster guidelines & robots.txt? (just curious)
— John ☆.o(≧▽≦)o.☆ (@JohnMu) December 14, 2016
This goes to show that Google is always looking out for companies that may or may not be in violation of the company’s Webmaster Guidelines, and there’s no telling what the consequences might be.
We reached out to Bill Sebald, Managing Partner of Greenlane SEO, to get his take on the above interaction with John Mueller:
“[Regarding Barry’s article, and the tweets from Mueller and Illyes] Our worry was we would be seen as doing something blackhat. Since intent is nine-tenths of the law, we truly believed we created a unique function that helps SEOs and Google. Google wants to crawl the important pages, and often non-valuable pages slip into Google’s index. With our tool, it helps webmasters see what pages should be purged. All parties benefit.
We were very careful not to scrape or break any Terms of Service. We leveraged an unrestricted, existing Google API believing it was openly available for such use. We don’t want to do any damage to anyone, though I wish our industry could be made more aware of Google’s hard lines. The existing materials are often too fuzzy or too difficult to find.”
This is something that we’ve seen many SEOs struggle with. Sebald continues,
“What I’ve always struggled with is the dichotomy between where Google supports SEO, and where they dislike it. If they want white-hat SEOs to help them discover good content, then we need data and tools to understand how we’re doing. We need data to build marketing campaigns that fulfill users and Google itself. If Search Console was at all an attempt to placate SEOs, it doesn’t really provide enough, especially on the indexation and bot analysis front. I understand Google is not open-source, but why not let us make helpful tools?
Take cloaking for example. Google has always been relatively lenient on cloaking when done with the intent of overcoming platform inefficiencies. However, when black-hats use cloaking for spamming, that’s where the action was taken. I think Google might benefit from being more lenient on tool builders using their data, especially when the intent is good.”
In the past, Google has not hesitated to block a software’s ability to conduct searches and the company may be preparing to do so again.
I also reached out to Jon Henshaw, Co-founder of Raven Tools, to get his insight into what this might mean for third party tools. According to Henshaw, Google is notoriously inconsistent when it comes to enforcing its own Terms of Service:
“In regards to scraping search results, Google has been inconsistent with how they enforce their Terms of Service. In fact, there appears to be a great deal of disagreement inside of Google as to whether or not it should be enforced at all.”
Apparently, the AdWords team is more strict in their approach to enforcing the Terms of Service, which Henshaw explains his company has been a victim of in the past:
”For example, the AdWords team seems to care about it [the Terms of Service], and they actively try to enforce the policy by threatening to take away access to their API. That’s what happened to us a few years ago. They aggressively threatened to take away API access from our reporting software if we didn’t remove scraped rankings data from our platform.”
After being backed into a corner, Henshaw explains his company had no choice but to comply with Google’s demands:
”We chose to comply, because at the time we were concerned that they might leverage other APIs like Google Analytics in the future. What happened instead was a pause in their enforcement against companies we competed with. It was based on what I was told were disagreements between different departments inside of Google (AdWords, Analytics, Search, etc…).”
Going forward, Henshaw we’ll see no changes in Google’s inconsistent enforcement of its own policies. This puts companies like Henshaw’s at a competitive disadvantage since other companies continue to offer features that Raven Tools was forced to remove.
”To the best of my knowledge, Google continues to not fully enforce this policy, all while we continue to comply with their demands and operate with a significant competitive disadvantage.”
