Microsoft announced IndexNow, a major evolution in how all search engines can discover and index content. IndexNow is open source and open to all participating search engines. When content is published or updated on a website, IndexNow notifies all participating search engines, dramatically reducing the time to indexing.
The IndexNow sharing between search engines will begin on November 2021. There is a limit of 10,000 URLs per day.
The IndexNow protocol is open to all search engines with a noticeable presence in at least one market.
Evolution of Search Indexing from Pull to Push
There are two ways for search engines to obtain web page data: Pull and push.
Pull is when a search engine crawler visits a site to request web pages and “pulls” the data from the server. This is how search engines traditionally work.
What IndexNow does is changes content discovery to the push method where a CMS like WordPress tells the search engines when the content has been published or updated.
The benefit to publishers is that they no longer have to wait for search engines to crawl and discover the content.
Search engines know about it right away and visit the site to fetch the new or updated pages.
According to the Microsoft announcement:
“…historically one of the biggest pain points for website owners has been to have search engines quickly discover and consider their latest website changes.
It can take days or even weeks for new URLs to be discovered and indexed in search engines, resulting in loss of potential traffic, customers and even sales.
…once search engines are notified of updates they quickly crawl and reflect website changes in their index and search results.”
Who is Behind IndexNow
IndexNow was created by Microsoft and Yandex. The protocol is open source and open to all search engines to participate.
When one search engine is notified of an updated or changed web page all participating search engines will be notified.
“IndexNow is a new protocol created by Microsoft Bing and Yandex, allowing websites to easily notify search engines whenever their website content is created, updated, or deleted. Using an API”
How IndexNow Benefits Publishers
IndexNow benefits publishers because it will reduce crawling demands on the server, search engines won’t need to conduct exploratory crawls to check if web pages have been updated and it reduces the time for content to be discovered and indexed.
Reducing server load helps the server perform optimally without the added burden of redundant serving of web pages that search engines already have.
Ultimately it benefits the world by reducing the energy demands of crawling and indexing and helping to reduce global warming pressures.
Bing’s announcement notes:
“IndexNow is an initiative for a more efficient Internet… website owners provide a clear signal helping search engines to prioritize crawl for these URLs, thereby limiting the need for exploratory crawl to test if the content has changed…
In the future, search engines intend to limit crawling of websites adopting IndexNow.”
Large Companies are Adopting the New Protocol
Microsoft announced that large companies like “eBay, LinkedIn, MSN, GitHub, and Bizapedia” are planning to migrate to IndexNow from the Bing URL submission API.
It is presumed that other larger companies will follow suit because the benefits of the new system are clear.
Because of that smaller companies may wish to consider also adopting the new protocol in order to stay competitive and improve their indexing.
How Does the New Protocol Work?
There are three ways suggested for submitting URLs.
1. SEO Integration of IndexNow
The first way, OnCrawl and Botify have plans to integrate IndexNow.
2. Direct Submission through API
The second way is a direct submission through an API.
Developers can use an API to automatically submit URLs to IndexNow.
The announcement describes it like this:
“1. Generate a key supported by the protocol using our online key generation tool.
2. Host the key in text file named with the value of the key at the root of your web site.
3. Start submitting URLs when your URLs are added, updated, or deleted. You can submit one URL or a set of URLs per API call.Submit one URL is easy as sending a simple HTTP request containing the URL changed and your key.
https://www.bing.com/IndexNow?url=url-changed&key=your-key"
3. Integration with CMS and CDNs
But the third way may be how most sites will interface with IndexNow, accessing the new protocol through a function that is native to the core of whatever CMS a publisher uses.
Wix and Duda recognize the benefit to users and are already on-board with plans to provide the new feature to their users.
Cloudflare and Akamai are supporting the new IndexNow protocol as well.
Waiting for WordPress?
While competitors like Wix and Duda have recognized the importance of IndexNow, WordPress appears to be dragging its feet on the new protocol.
Maybe WordPress is going to adopt it, but for the past seven months the core developers have been discussing the proposal with no apparent resolution.
According to the public WordPress development ticket there appears to not be a decision on it yet.
Nevertheless, Microsoft has submitted the IndexNow open-source code to WordPress for inclusion into the core.
If adopted by WordPress this will make it automatic that a publisher can opt into submitting their URLs via IndexNow.
In response to a question by WordPress core developers, Microsoft disclosed that publishers will be able to choose which search engines to submit their changes and new URLs to.
That means they can block specific search engines from receiving the data about their pages.
According to the WordPress development ticket, the Microsoft representative said:
“WordPress and other CMS admins will be able to able to select which actors in the industry they want to notify, including everybody via centralized services if they want to notify all.”
IndexNow is the Evolution of Search Indexing
IndexNow is a big deal because it’s a major change in how search engines can discover updated and newly published documents.
It benefits publishers with fast indexing and less server load from bots constantly crawling their web pages.
Adoption by major search engines will be an evolution in the relationship between publishers and search engines that benefits both.
Citations
Read the Microsoft Announcement
Announcing IndexNow – Instantly Index your Web Content in Search Engines
Visit the IndexNow website
IndexNow WordPress Development Ticket and Pull Request
Instantly index WordPress web sites content in Search Engines
