In Google’s latest Q&A video, John Mueller answers a question from a site owner concerned about their site not being crawled often enough.
“How often does Google re-index a website? It seems like it’s much less often than it used to be. We add or remove pages from our site, and it’s weeks before those changes are reflected in Google Search.”
Mueller explains that it takes time to crawl entire websites. If Googlebot were to crawl a whole site in a short period of time it would cause unnecessary strain on the server. For that reason, Googlebot has a strict limit on the number of pages it can crawl every day (exact number was not mentioned).
Since Googlebot can only crawl a certain amount of pages, it gives priority to pages that should be crawled more frequently. The most “important” pages, according to Google, are the home page and high-level category pages. Those pages will be crawled most often, so changes will be reflected in Google search faster than, say, an old blog post.
Googlebot will re-crawl important pages at least every few days, and even more frequently in some cases. Mueller did not mention how often other types of pages are crawled. Given this information, it would be fair to say patience is key when waiting for changes on other pages to be reflected in Google Search.
John Mueller’s full response is quoted below.
“Looking at the whole website all at once, or even within a short period of time, can cause a significant load on a website. Googlebot tries to be polite and is limited to a certain number of pages every day. This number is automatically adjusted as we better recognize the limits of a website. Looking at portions of a website means that we have to prioritize how we crawl.
So how does this work? In general, Googlebot tries to crawl important pages more frequently to make sure that most critical pages are covered. Often this will be a websites home page or maybe higher-level category pages. New content is often mentioned and linked from there, so it’s a great place for us to start. We’ll re-crawl these pages frequently, maybe every few days. maybe even much more frequently depending on the website.”
