Google quietly added a new bot to their crawler documentation that crawls on behalf of commercial clients of their Vertex AI product. The documentation says that the new crawler may crawl sites at the site owner’s request.
Vertex AI Agents
The new crawler, called Google-CloudVertexBot, crawls websites content for Vertex AI clients, which is different from the other bots listed in the Search Central documentation that are tied to Google Search or advertising.
The official Google Cloud documentation offers the following information:
“In Vertex AI Agent Builder, there are various kinds of data stores. A data store can contain only one type of data.”
It goes on to list six types of data, one of which is public website data. On crawling, the documentation says that there are two kinds of website crawling:
- Basic website indexing
- Advanced website indexing
Documentation
The documentation explains website data:
“A data store with website data uses data indexed from public websites. You can provide a set of domains and set up search or recommendations over data crawled from the domains. This data includes text and images tagged with metadata.”
The description of Basic website indexing doesn’t say anything about site owner verification but someone from Google contacted me and advised that the basic website indexing just uses of slice of what’s already crawled by Google.
Advanced website indexing, which uses the new Google-CloudVertexBot, requires domain verification and there are indexing quotas. It appears that the new crawler isn’t crawling public websites but rather it crawls on the “site owners’ request” so it may be that it won’t come crawling public sites.
The Changelog notation for this new crawler says this:.
Here’s what the changelog says:
“Introducing the Google-CloudVertexBot crawler
What: Added Google-CloudVertexBot to the list of Google crawlers, a new crawler that crawls sites on the site owners’ request when building Vertex AI Agents.
Why: The new crawler was introduced to help site owners identify the new crawler traffic.”
New Google Crawler
The new crawler is called Google-CloudVertexBot.
This is the new information on it:
“Google-CloudVertexBot crawls sites on the site owners’ request when building Vertex AI Agents.
User agent tokens
- Google-CloudVertexBot
- Googlebot”
User agent substring
Google-CloudVertexBot
Google-CloudVertexBot
The documentation indicates that the new crawler doesn’t index public sites and the changelog indicates that it was added so that site owners can identify traffic from the new crawler. Should you block the new crawler with a robots.txt just in case? It appears to not be necessary to add it to the robots.txt because it only crawls by site owner’s request.
Read Google’s new documentation:
Featured Image by Shutterstock/ShotPrime Studio
