Multivariate Testing: What it is and When to Use It

At Yoast, we’ve written quite a few posts on Conversion Rate Optimization and A/B testing in the last few years. However, we’ve never really touched the subject of multivariate testing. In this post, I’ll explain what multivariate testing is, its pros and cons, and when you should (and should not) be using it.

What is Multivariate Testing?

Explaining what multivariate testing is, seems quite easy. In fact, when you look up multivariate testing on Wikipedia, it says the following:

However, I find this to be oversimplifying things a bit. With A/B tests, you’re testing different versions of a complete page. This is not what you do with a multivariate test (MVT). With an MVT you’re testing elements within a page.

So let’s say you have two versions of a headline, two versions of an image, and two versions of a block of text. With a multivariate test you will test every possible combination against the others to figure out which combination yields the best conversion rate. And this is where things get tricky because in the setup I mentioned above, there are already 8 combinations (2 x 2 x 2 = 8):

1. Headline A + Image A + Textblock A
2. Headline A + Image A + Textblock B
3. Headline A + Image B + Textblock A
4. Headline A + Image B + Textblock B
5. Headline B + Image A + Textblock A
6. Headline B + Image A + Textblock B
7. Headline B + Image B + Textblock A
8. Headline B + Image B + Textblock B

Advantages

The first advantage of multivariate testing is that you can see the effect small changes on your site have. Of course, small changes can also be tested with an A/B test, but that’s suboptimal as you can only test one small change at a time. Generally, A/B tests are used for the big changes and multivariate testing is used for optimizing smaller elements.

Also, by using a multivariate setup, you’re actually able to not only test the effect of changing one element, but you’re also able to test the combined effect (interaction effect) of several elements. Will changing an element still have the same effect if you change another element on that same page? These are questions you wouldn’t be able to answer with an A/B test, but you can answer them with multivariate tests. With multivariate tests you can pretty accurately see what the effect of each element is in which setup or situation.

Limitations

The biggest drawback of multivariate testing is that you need an even bigger amount of traffic and especially conversions than you do for an A/B test. I always say you should have at least 100 conversions on each variation. So if your multivariate test has 3 different versions of 3 different elements, you’ll need at least 2700 conversions (3 x 3 x 3 = 27 combinations with each at least 100 conversions).

And that’s a lot for most websites, especially if we’re talking sales on a specific product. For most websites, this also means the page needs to have quite some traffic on it, since most conversion rates aren’t that high.

Lastly, the multivariate test setup looks at a lot more variables than an A/B test and also looks at how these variables interact. This means there’s a lot bigger chance that mistakes or errors can occur in the reporting. So you should check your multivariate test results even more than your A/B test results.

When to Use Multivariate Testing

As I’ve already mentioned above, multivariate tests are used to test smaller elements on a page. You’re testing small variations of the same element, instead of overhauling the complete page as you would in an A/B test. So first and foremost, multivariate testing isn’t supposed to be used as a starting point.

Obviously, before starting to test at all, there are a few things you have to do. First of all, you should check whether there is even enough traffic and conversions on the page you want to run a multivariate test on. Since an MVT setup increases your amount of variations really quickly, it’s important that you have an abundance of traffic and conversions. You usually already know whether your traffic and conversion rate are up to par because you’ve already A/B tested the page in question.

What’s Next?

The next step is to find out what kind of changes you think are needed for that page. To do this, you first need to be clear on what the goal of the page is. What do you want people to do on your page? Write down what you think could be the cause of visitors not completing this goal. You can find this out by being critical yourself, doing user testing (or even just a survey) and looking at your analytics.

Now you know what your page’s goal is and what could be causing a lower conversion rate on this goal.

The next step is to find out what you could possibly change to make more people fulfill that goal. You have to think of different versions of the factors that could be hurting your conversion rate.

When your traffic and amount of conversions are acceptable, you’ve found your page’s goal and causes for it being less than optimally completed, you should formulate hypotheses. Write down what you’ll be changing, what effect you expect this to have and maybe even why. For instance:

• If we shorten the checkout form, more people will complete their purchase.

To my mind, hypotheses are even more important for multivariate testing, because it is so easy to just add a few more variations. Formulating hypotheses will prevent you from randomly adding new variations. Make sure you have a hypothesis for every single variation you’re creating.

Multivariate or A/B?

If you find your hypotheses are about changes that drastically change the page’s layout or look, then you’re better off choosing for an A/B test. As said, A/B tests give you the possibility to find out whether one version of a page performs better than the other.

If, however, you find your hypotheses are about small changes (for example, text on your call-to-actions) then multivariate testing could be a good choice. But you have to be sure your page meets the criteria for multivariate testing that I mentioned above.

What Do You Think?

Multivariate testing isn’t as easy as it might seem. It’s something you should only get into if you know what you’re doing. So I’m wondering: have you ever tried multivariate testing? And did you follow my steps? Or do you have other steps you think should be followed? Let me know in the comments!