In a recent Javascript SEO Office Hours Hangout, Google’s Martin Splitt answered a question about infinite scrolling and preventing indexing of scrolled content as if it were part of the main content.
Infinite Scrolling
Infinite scrolling is a way to keep feeding similar content to users. A JavaScript will sense when a user nears the end of the web page and load additional content for the site visitor to read.
The problem though is when Google sees the additional content and mistakes it as being a part of the main content. That would be a catastrophic outcome because it would likely lead to lower rankings.
Here is what Vahan (Search Engine Journals lead developer) asked:
“…We have implemented infinite scroll on mobile. In the past we had it on the desktop. My concern is would Google index the infinite scroll articles as part of the main article which is first?
The Ajax URL for each of the page queries has a no-index applied. Is there any guarantee that the appended content will not be indexed as a part of the main web page?”
Martin Splitt answered:
“The answer is it depends on how it is implemented and how we see it in the rendered HTML.
I would highly recommend checking out the testing tools to see the rendered HTML because it depends a lot on how you build your infinite scroll and how we can discover additional content.
But if it’s like for instance using some sort of link that tells us to go to another URL and then that URL is no-indexed then we would not see that content.”
Martin apparently declined to say definitively whether Google is seeing the additional content or not. His suggestion that this is something that should be checked using Google’s tools is the best way to confirm how Google is rendering a web page.
Martin Splitt didn’t mention it, but Google’s Mobile Friendly Test will display the HTML of the rendered web page. So if Google is indexing the additional content then it may be shown in in the HTML section of the rendered web page results.
Martin didn’t mention it but I’m pretty sure the tool is an accurate representation of the rendered page.
Vahan then added:
“The feature is implemented in the following way, like when you scroll down it loads the article through AJAX at some point like when you are about to end reading the article.
But the AJAX URL which sends the content of the next article has a noindex header tag ( x-robots-tag: noindex ) applied. So that makes me somehow confident that appended content will not be indexed.
But I would like to like how I can make sure that if next scroll the articles will not be indexed as a main part of the article.”
Martin again indicated that he can’t say for certain and that this is something that should be tested in Google’s tools.
Martin’s response:
“I don’t know. I’m not fully sure how we see the rendered HTML.
Use the testing tools, specifically the URL inspection tool can help you figure out how the rendered HTML looks like if rendered HTML somehow still contains the additional content because the viewport has changed or something like that then we would index it as part of the main page as in like the page that you’ve seen.
And then no-indexing that doesn’t really help that much. “
Martin then lists things that can go wrong with infinite scrolling:
“It can also be that you accidentally no-index the content that was previously on the page so that you might end up no-indexing too much.
I would always test these things and look at the rendered HTML. The rendered HTML tells you what we are seeing. You can use the URL inspection tool to see what we have crawled, so you see it in the crawled rendered HTML.
But you can also use the live test to see what we see if we would do it again.
So it depends really is the answer in that case.”
Watch the Google Office Hours Hangout:
