This is a sponsored post written by Botify. The opinions expressed in this article are the sponsor’s own.
Your biggest SEO win may be where you don’t expect it.
Contrary to common belief, search engines – including Google – do not have unlimited resources to explore all websites thoroughly. They have a “crawl budget” they have to spend carefully. The internet is virtually bottomless, and although search engines explore a staggering number of sites, they still have to be selective and leave some out. They spend the “crawl budget” they think your website deserves, no more.
As a result, it is extremely common to find e-commerce websites with less than half of their product catalog crawled by search engines, or publishing websites with only a portion of their articles crawled.
Products Not on Display Can’t Be Bought
If you owned a physical store, would you keep some of your products locked away in storage, where buyers can’t see it, let alone put it in their cart?
Of course you wouldn’t. But that very thing is happening on most websites. If Google or Bing doesn’t know a page exists, there is no way this page can rank in search results and generate traffic. It’s a shame, as these “invisible” pages may include quality content that searchers are looking for. All these pages need in order to start generating traffic is to be found and explored by search engines.
Is This Happening on Your Website?
Forced to be selective to remain within your website’s crawl budget, search engines ignore some pages knowingly, while other pages are missed by chance. If a page is linked only from pages the crawler did not explore, it is completely out of the field of vision, no matter how important the content.
This is an issue on most websites, and it takes huge proportions on large websites, where the part ignored by search engines may represent thousands or even millions of pages.
Crawl Ratio: The Foundation of High-Return SEO
Identifying priorities is often the most difficult step in SEO. You want to implement changes that will make a real difference and bring significant traffic from search engines, but how do you know what will accomplish your goals?
Maximizing crawl ratio – the proportion of pages Google explores on your site – and making sure all pages with traffic potential are explored is your first step towards achieving that organic traffic. What good will it do to tweak pages, verify that title tags and H1 tags are properly set, and make sure they include high-quality content, if they don’t even exist in search engines’ world view?
35% More Organic Traffic After Improving Crawl Ratio
Let’s look at the example of an e-commerce website, which significantly increased organic traffic by increasing crawl ratio – particularly on product pages.
Initially, product pages had a crawl ratio of 54% from Google. This means that almost half of the product catalog couldn’t appear in search results.
After taking action to improve Google’s crawl on product pages (by enhancing navigation to products, reducing page depth, increasing the number of links to each product, and by removing pages which were wasting crawl budget – in particular internal search results with near duplicates), 92% of product pages are now crawled.
And the results are spectacular, with a significant increase in weekly organic visits on product pages:
Why Haven’t You Heard About Crawl Ratio Before?
Simply because very few tools deliver this key technical indicator. To be able to provide this, the SEO tool must:
- Have visibility of the full inventory of pages on your website (provided by a website crawler)
- Have visibility of search engine crawl, with every single hit from their robots on your pages (provided by web server log files analysis)
- Combine the two types of information to show which pages are explored by search engines and which aren’t
Botify is the only SEO tool in the industry that combines all three features, along with segmented views of your website and tons of other key metrics, offering unparalleled insights into SEO priorities. With Botify you get everything you need to make the right decisions and get results.
Image Credits
Featured Image: Image by Botify. Used with permission.
In-post Photo #1: Image by Botify. Used with permission.
In-post Photo #2: Image by Botify. Used with permission.
In-post Photo #3: Image by Botify. Used with permission.
