To be successful in SEO, we need to have an understanding of how users engage and interact with the websites that we manage.
One way we can do this is by running tests that show us what users want from our sites and how they react to the content on our pages.
Website testing allows us to experiment with ways to increase conversion rates and UX by comparing a control page against a test group to analyze performance impact based on changes made.
This is something that search engines are happy with because they want to direct visitors from their SERPs to sites that will provide them with a positive, seamless experience.
It’s a win-win situation.
Before we get into the details of the dos and don’ts of website testing, let’s take a closer look at A/B testing and multivariate testing and the key differences between them.
What Is A/B Testing?
A/B testing (which is also referred to as split testing) involves changing one or more elements of a page, and comparing how the original page and test page variant perform when shown to different segments of an audience.
Your original page will be variant A, and your test page version will be variant B.
This is usually a straightforward test that compares the performance of both pages (or all variant pages in a set if you have more than two) and tells you which one converted at a higher rate.
These are some of the most common use cases for A/B testing:
- Testing CTA messaging.
- Testing color changes.
A/B testing usually involves serving different versions of a page’s content on different URLs, and the user will be redirected to a particular version depending on your test thresholds.
What Is Multivariate Testing?
Multivariate testing is a bit more complex than A/B testing, as it involves simultaneously testing multiple sections on a page to see which combinations of changes on a page are most effective for users, rather than just telling you whether or not one page was more successful than another.
Multivariate test example from Google OptimizeThis can offer more granular insights into page optimization opportunities.
You can understand how the different elements of a page can work best together and how they impact one another.
These are some of the most common use cases for multivariate testing:
- Testing combinations of headlines and images.
- Testing combinations of body text and buttons.
Multivariate testing dynamically changes elements of a page on-the-fly, and all users are directed to one page in which the variations are inserted, rather than being sent to different URL variations.
How A/B Testing & Multivariate Testing Impact SEO
Although it’s beneficial to run tests to better understand how users interact with our sites, it’s important to be aware of the negative impacts that tests can have on SEO when they are not properly managed with best practice in mind.
I was inspired to write this post while doing some recent analysis on a client’s website that was struggling with ranking fluctuations and keyword cannibalization.
The culprit?
A/B testing gone rogue.
The reason why tests can have this negative impact is that they involve either changing the content on a page, or serving duplicate pages with slight differences.
Here are some of the main issues you can run into with website testing:
- Cloaking: When test versions vary drastically and search engines are seeing a significantly different version of a page to what users are seeing, this is seen as cloaking. Cloaking goes against Google’s Webmaster Guidelines, and can cause your site’s rankings to be demoted, or be removed from organic search entirely.
- Page duplication: Having multiple, nearly identical sets of pages available to search engines without correct management can create duplication issues.
- Inconsistent redirects: With A/B testing which serves different URLs depending on the user and session, search engines can see the main URL sometimes, but then be redirected to another URL other times. This can cause confusion around which page is the primary one which should be indexed.
To gain a deeper insight into the SEO impact of website testing, let’s take a look at what search engines are advising us.
Overall, Google is looking for stability and consistency from sites that are running tests. It doesn’t want to be redirected to a brand-new page every time it crawls your site.
It wants to be able to index a page that is a representative of what a user would usually see, and wants to be confident in matching the experiences of the page that is being promised in the SERPs, and the one that the user will be directed to on your site.
“Essentially what we want is that the long-term stable situation on your site is such that we trust that the content you’re providing is actually the content that you’re showing to users as well. […]
If you’re doing changes where one version has a lot of textual content and that’s the version you show Google, and users when they go there they essentially just see 2 or 3 words of content and maybe a big image then that would be a bit awkward for us.
For the long run it should really be that the content is equivalent.” – John Mueller, Google Webmaster Hangout
“If you’re doing A/B testing, we recommend showing Googlebot the version that the majority of your users are seeing. […]
What I wouldn’t recommend doing is randomly switching that version because that makes it a lot harder for us to actually index your pages properly.
Every time we look at the page we see a different version of the content and that really makes it hard for us to understand: are things significantly changing here on this page all the time, or is this basically just two versions that are swapping back and forth?
So I’d try to stick with one version if that’s at all possible.” – John Mueller, Google Webmaster Hangout
For further information, take a look at Google’s best practices for website testing.
Bing
To learn more about whether other search engines handle website testing differently, I got in touch with the Bing Webmaster team to get their stance.
Here’s what Fabrice Canel, Principal Program Manager at Bing, had to say on the matter:
“When running human-targeted experiments, I highly recommend always outputting the control (version A) for Bingbot, and other search engines.
Search engines prefer stability versus instability. It’s preferable to output a well-established, same version of the site.
Like for JavaScript, if you basically output minor content difference between version A and B, we will not consider that as cloaking. However, we will review if we see major differences.
In the case of SEO, I recommend running experiments over the course of weeks, not hours, to truly understand the impact and limit the scope to versions A and B… not A, B, C, D, X, Y and Z.”
8 Tips for Improving Your Tests & Processes
To keep search engines happy while we’re running these tests for our users, I’ve put together some key best practice tips to make sure that you can continue testing to maximize the performance of your site, without damaging it in the process.
1. Ensure Google Is Served the Primary Content of a Page That You Want to Rank With
If you have a variant page where a section of key body copy has been removed, when the search engine accesses the page, nothing is left with which to assess the overall topic of a page and what it should rank for.
This also applies to instances where you’re swapping out sections of content.
I have seen tests where blocks of copy from subcategory pages were being inserted into the homepage, which caused cannibalization issues as the homepage started ranking for key terms associated with those lower-level pages.
Carefully assess key elements like title tags, body content, internal links, and images, and make sure search engines can access a version of the page where these remain intact.
The most important question to ask yourself about your website tests: will this affect how Google can crawl, understand, and index the content on the page?
Changing the size of the checkout button won’t affect this, but swapping out an H1 might, for example.
2. Don’t Create Pages That Are Too Distinctly Different From One Another
Search engines are able to detect when there are minor changes between pages for testing purposes, and generally have no problem with this.
However, if the page variations differ drastically, this could be flagged as cloaking and earn you a manual action.
The page that search engines end up accessing should match the primary topic of the original page.
If your original page was targeting restaurant keywords, but the variant page is about life insurance, then this would be a big red flag for search engines.
3. Use the rel=”canonical” Tag for Pages That Have Different Test URL Variations
If you’re running tests that create multiple page variants on separate URLs, Google recommends adding a canonical tag to specify the original page that should be indexed.
This helps to avoid search engines choosing another duplicate test page to index instead of the original page.
The canonical tag is only a signal and not a directive, so make sure other elements like your internal linking and sitemap URLs consistently point to the original page as well to give search engines a clear picture of which page should be indexed as the primary version.
4. Be Careful When Using Noindexing or Blocking Test URLs
Adding a noindex tag to a page in a set of duplicate pages is risky.
This is because Google will pick its own canonical tag based on the other signals from your website, and it might pick a variant URL rather than the original URL as the one to index and show in search.
If you’ve noindexed this variant URL, however, Google will drop that page from the index and no version will be shown in search as the rest of those pages will have been grouped together as duplicates.
This has the same effect as noindexing your original canonical page.
Also, avoid blocking test pages via robots.txt as search engines will need to see the differences between page variations to be able to respect canonical tags.
Blocking test pages in a robots.txt file could constitute as cloaking if you’re stopping search engines from seeing a test page version for users that differs greatly from the version they can access.
As long as you follow the rest of the tips in this article, there shouldn’t be a problem if search engine bots end up crawling your test page variants.
5. Use 302 Redirects When Running Split Tests That Involve Redirection
Using a 302 redirect between your test URLs for A/B testing tells search engines that the redirect is temporary and the original control page should be kept in the index, rather than indexing the test page that it is redirecting to.
Using 301 redirects would indicate to search engines that they should pass link equity and page signals to the variant page, and replace the original page in the index.
6. Retain the Original Page and Make Content Updates There Where Possible
It’s important to preserve all link equity that has been built for a page by maintaining the existing URL and making any required content changes off the back of your tests to that page.
If you run tests that involve separate URLs, don’t just switch off the old page and leave the new one live instead if the variant performed better.
Update the existing page with the changes that appeared on the new page version.
If there isn’t an option to update the existing page, then make sure you add a 301 redirect to the winning variant page.
This will ensure that the majority of ranking signals are passed across, and will prevent page duplication by leaving them both live.
7. Only Run Tests for as Long as Necessary
It’s important to get the balance right between running a test long enough to be able to gather meaningful data and insights, and ensuring that these tests don’t impact organic search performance.
Search engines have an expectation for how long website tests usually run for and can accommodate for this.
However, running tests that are changing your content beyond this expected time period can be seen as an attempt to deceive them.
Google recommends that you keep an experiment running until it meets one of these two conditions:
- Two weeks have passed, to account for cyclical variations in web traffic during the week.
- At least one variant has a 95% probability to beat baseline.
8. Avoid Running Tests During Significant Site Changes
Running an A/B test during a site restructuring or migration can impact search engines’ abilities to pick up the correct signals from your site and how it has been changed.
This will impact what can be indexed, as well as how quickly and easily your site’s signals can be moved across to the new version.
“If during a site migration you’re doing some fancy A/B testing and sometimes we see redirects to one direction and sometimes we see the site migration redirects, then that can certainly throw us off in the bigger picture and can delay the overall move of one version to another.” – John Mueller, Google Webmaster Hangout
In Conclusion
When left unchecked, A/B and multivariate tests can negatively impact how search engine crawlers interact with your site, and ultimately, your rankings in organic search.
However, these tests are still crucial for discovering opportunities for increasing conversions and website performance.
Search engines even actively encourage us to run them.
By following the tips outlined in this article and running your tests with SEO best practice in mind, you can make the most of the benefits that website testing has to offer – without hurting your rankings.
More Resources:
- A 10-Step Process for Understanding Site Performance & Automating Speed Testing
- The Scientific Method for SEO: 6 Steps to Find Your True Ranking Factors
- Do Ugly Sites Convert Better?
Image Credits
Featured Image: Unsplash
Screenshot taken by author, May 2020
