Google’s John Mueller answered a Reddit question about indexing issues. Although the question was specific to an AngularJS issue, the solutions Google’s John Mueller offered are useful for all sites. His answer could be applied to almost anyone who asks, “Why isn’t Google indexing my content properly?”
Google’s Mueller offered five tips to diagnose canonicalization problem.
- Make Sure GoogleBot Can Index the Code
- Measure Content Processing Speed
- Minimize Resources Needed to Load Main Content
- Make it Easy to Reach Unique Content
- Canonicalization Issues May be a Symptom of Indexing Issues
1. Make Sure GoogleBot Can Index the Code
Today’s CMS templates are designed with users in mind. Consequently they use JavaScript and CSS to create good user experiences. However, the code can be too complex for GoogleBot to render.
GoogleBot renders code according to the web standards associated with Chrome 41. Chrome 41 was released in 2015. So if the JavaScript and CSS does not degrade gracefully in Chrome 41 then it’s possible that GoogleBot will not be able to index the website.
Here is a shortcut test John Mueller suggested to test if GoogleBot can index your site:
“If you’re sure that your code works for Googlebot (a quick way to test is the mobile-friendly test)…”
John Mueller suggested using Google’s Mobile Friendly Test as a way to test if GoogleBot can understand your code.
2. Content Processing Speed
Google’s John Mueller then encouraged web publishers to check if the amount of time necessary to process a page of content might be causing GoogleBot to improperly index content.
Here is what John Mueller said:
“…then it’s worth estimating if speed of processing might be an issue. The hard part here is that there’s no absolute guideline or hard cut-off point that you can focus on or test for, partially also because a page rarely loads exactly the same way across separate tests.
My way of eyeballing it is to see how long the mobile-friendly test roughly takes
and to check with webpagetest.org to see how long a page might take to load the critical / unique content, and how many resources are required to get there.”
3. Minimize Resources
Sites that load up fast convert more often, generate more sales and serve up more ad impressions. In order to speed up a site, review all the cookies and scripts needed to make that page load.
Sometimes there are JavaScripts loading that have nothing to do with that page, like a contact page script. Minimizing plugins, especially for things that are special effects. Site visitors are interested in content, not special effects.
According to Google’s John Mueller:
“The more resources required, the longer time until the critical content is visible, the more likely Google will have trouble indexing the critical content.”
4. Make it Easy to Reach Content
GoogleBot is rendering content as if it’s a mobile browser (Chrome 41). Another consideration is something like chained redirects (a URL that redirects to another URL that subsequently redirects to another URL).
Test your site to make sure it renders well in the most popular Apple and Android devices. Doing things like blocking JS and CSS could cause GoogleBot to not render your pages well.
John Mueller said that content that is difficult to reach will result in Google being unable to properly index the content:
“…it’s probably a sign that it’s too hard to get to your unique content — it takes too many requests to load, the responses (in sum across the required requests) take too long to get back, so the focus stays on the boilerplate HTML rather than the JS-loaded content. Reducing dependencies & latency can help there.”
5. Canonicalization Issues May Be a Crawling Issue
The question on Reddit that John Mueller was responding to was about canonicalization issues. The person was asking what could he do to make Google canonicalize the right content.
“I’m having issues with canonicals on my site and I believe it’s due to the angular JS used in the HTML. For thousands of pages on the site Google is ignoring the canonical link and selecting their own…”
If you find that Google is canonicalizing and ranking the wrong page, it could be a symptom of an indexing problem. It might be that Google is having a difficult time rendering the web page. It could be too many resources are needed to load the page. It could also be that some resources are being blocked from seeing the unique content.
More Takeaways
- Make it easy to render the site
- Make sure the site renders on Chrome 41
- Use Google’s Mobile Friendly test to gauge how long it takes to load
- Minimize resources need to load a web page
- Make it easy to get to the content
Read the Reddit discussion here: Canonicals and angular JS
More Resources
- How Search Engines Crawl & Index: Everything You Need to Know
- How Often Google Crawls and Indexes
- Google: How to Get New Content Indexed Fast
- Google’s John Mueller Explains Why Google Crawls Non-Existent Pages
Images by Shutterstock, Modified by Author
