Google is entering a new phase of SERP scraping due to AI agents. Some of it may be driven by networks of LLMs that are reverse engineering Google’s search results. The public face of Google’s war against scraping remains rank trackers but the real battle may be elsewhere.
Will the ability to track rankings become a thing of the past?
Massive AI Agent Traffic Expansion
What may be a reason behind Google’s actions to stop rank tracking is that the amount of AI Agent traffic has grown exponentially. According to Cloudflare, which is in a position to monitor web traffic, AI agent website traffic has increased 1,700% over the past year.
A recent article from Cloudflare explained:
“At the end of 2024, Cloudflare handled an average of 63 million HTTP requests a second. Today, it’s almost doubled to 115 million, with peaks above 150 million. Over the past year, daily requests from AI agents on our network grew by more than 1,700%. This year, for the first time, more than half of Internet traffic wasn’t human.”
The problem with AI agent web traffic is that some of it a hybrid human/agent visit, agents working on behalf of humans, which means that some of it still has value to websites, something Google may have already considered in formulating engineering responses to rank trackers.
Google’s AI Search Is More Expensive To Produce
The problem for Google is that a massive amount of traffic could cause latency issues in serving search results, including AI search results. This is a bigger problem for keyword that trigger AI-driven search results because those are computationally more expensive to generate.
Regular search ranks pre-indexed websites that are associated with various queries. AI-derived search results include LLM inference over multiple queries, which means that every time a user asks one question, the LLM generates multiple associated queries, plus various reasoning steps, before answering with a synthesized summary of all the sources.
AI Mode is inherently more computationally expensive than traditional Search, which is why Google is aggressively reducing its AI cost per query as usage scales through improvements in LLMs, hardware and engineering optimizations.
Google’s Sundar Pichai remarked multiple times on Google’s efforts to drive the search per query costs down in the Q2 Earnings call. The cost of AI queries was so important that it merited multiple mentions:
“Yesterday we announced new models — Gemini 3.6 Flash and Gemini 3.5 Flash-Lite — which are cost effective and highly efficient. We’re seeing tons of demand for our workhorse Gemini Flash series, because it hits the sweet spot of performance and cost.
As we serve more of these queries, we’ve continued to drive efficiencies. Thanks to our engineering and hardware optimizations, this quarter we reduced the cost of AI Mode responses to its lowest level since launch, even as we’ve brought more advanced AI capabilities.”
The high cost per query of AI-driven Search is a real problem. AI Search queries are computationally more expensive to serve and AI agent traffic can effect search results latency. Google may very well have taken a more adversarial approach to automated rank tracking in order to improve the cost per query calculation.
Networks Of AI-Generated Adversarial Content
Google has published multiple research papers focused on AI-generated content. The papers allude to attackers changing their content in response to Google’s reactions in order to unlock rankings. Core to unlocking rankings is tracking Google’s search results. Although the research papers don’t mention distillation, the process of reverse engineering an LLM through the outputs, the few details the research paper shares imply that is what Google is really fighting against.
Read about these research papers:
Google Has Deployed A New AI Spam Detector Called SAFE
Google’s Scalable Cluster Termination System (S-CTS)
SEOs Notice Effect On Rank Tracking
Ryan Jones (LinkedIn profile), developer of the SERPrecon service, extracts competitor keyword and historical search data that informs their keyword analysis for content optimization. As the principal behind this tool he is an authoritative source of information about rank tracking.
Jones recently tweeted:
“we’re getting to the point where nobody will have rank tracking anymore because of the crazy amount of AI scraping going on.”
Jones offered the following response to someone’s tongue-in-cheek suggestion to us AI to scrape the search results (SERPs):
“that’s why the rank trackers are failing. Google is blocking all the AI trackers AND AI itself… and the rank trackers are getting caught up”
Takeaways
Reverse Engineering Google’s Search Algorithm
Something that has not been mentioned by Google and is not frequently discussed is that a reason why Google is discouraging rank tracking is that the data can be used to understand Google’s ranking output to build an AI Model that can reproduce Google’s search results through a process called distillation.
It may very well be that Google’s adversarial anti-rank tracking actions may be primarily an attempt to stop companies from reverse engineering Google’s search results.
SERP Tracking Distorts Keyword Data
Google’s decisions are made based on data and one of the problems with rank trackers is the increased cost per query. Another issue is that rank trackers distort the actual keyword inventory data. The inflation of keyword inventory is a long-running problem that predates the AI agent era, going back to the earliest days of SEO. But what’s happening today is a massive increase that quite likely distorts Google’s click and traffic metrics.
Questions To Ask
AI-driven automated crawling on behalf of humans and AI search has surged across the web. Google has to respond to automated search results scraping. Ryan Jones says the two trends are connected.
- Will Google’s efforts to stop aggressive AI scraping causing rank tracking to reach a point where it may become unsustainable
- The other question I have is how much of Google’s adversarial actions are to prevent reverse engineering Google’s search results?
Featured Image by Shutterstock/Microba Grandioza
