Microsoft Bing announced in October the development of a new crawling protocol called IndexNow that promises to make crawling and indexing more efficient. Google however has remained silent about whether they will adopt it or not until now.
A Google spokesperson issued a statement affirming that Google will be testing the new IndexNow protocol.
IndexNow
Microsoft Bing and Yandex introduced a new open-source protocol called IndexNow that allows publishers to notify search engines when a web page is updated or a new page is added.
IndexNow alleviates the need for search engines to crawl websites to check if they’ve been updated, saving bandwidth and resources at the search engine side and on the publisher side.
Major content management systems, Content Delivery Networks and SEO companies have announced support for the new protocol.
The list of companies adopting IndexNow protocol includes:
- Wix
- Duda
- Cloudflare
- Akamai
- Botify
- oncrawl
- onely
- Yext
Google is Already Proactive on Crawl Efficiency
The widespread acceptance of IndexNow has somewhat hinged on what Google would do and until today, Google has not commented on the new protocol.
Sustainability and efficiency are top concerns at Google so it would seem that IndexNow would be a good fit.
According to a spokesperson from Google:
“Google’s crawling mechanism is highly efficient and it’s still being improved.
For example, last year we announced that Googlebot supports HTTP/2, the next generation of the fundamental data transfer protocol of the web.
HTTP/2 is significantly more efficient than its predecessors and it saves resources for both Google and web sites. We use HTTP/2 in over half of all crawls.”
Google Announces It Will Be Testing IndexNow
For now Google appears to be taking a measured approach to the new protocol, which makes sense. Crawling the web is the backbone of a search engine, so any changes in protocol should be accompanied by clear benefits.
According to Google:
“We take a holistic approach to sustainability at Google, including efforts to develop the most efficient and accurate index of the web.
We have been carbon neutral since 2007 and we are on pace to be carbon-free by 2030 across both our data centers and campuses.
We’re encouraged by work to make web crawling more efficient, and we will be testing the potential benefits of this protocol.”
Waiting for WordPress
The next most influential entity, behind Google, is WordPress. WordPress is still talking about it and they seem to be encouraging the development of a plugin instead of rushing to integrate it into the WordPress core itself. WordPress is also looking for wider industry buy-in, which of course means Google but also the other search engines.
A lead developer at WordPress commented:
- I’m still not 100% convinced that having WordPress ping each of the engines individually is ideal, however, it’s not the worst.
- I’m still not 100% convinced that having an API key / verification callback should be allowed.
- All supported providers would need to be defaulted in core, so as not to preference any given engine
I think this should be developed as a plugin first, and then proposed to WordPress core as a feature plugin, to allow development of it to occur separately and then a suggestion to add it to core once feature complete.
That would also allow site owners to opt-in to using this prior to WordPress fully implementing it (Which would be in WordPress 6.0 at the absolute earliest, Q2ish 2022 at a guess).
Google Will Be Testing IndexNow
Adoption of IndexNow by Google would be a big deal for the entire industry, perhaps enough to move WordPress to include it into their core and if not, into a plugin. It makes sense for Google to test it. There is no word as to how long this testing will last.
