Multivariate vs. A/B Testing
When’s the best time to use A/B testing? How about multivariate testing? What do those even mean? I hope the following article outlines the two types of CRO tests and which is best in certain situations.
A/B Testing
Remember in science class when you would try different experiments with a number of inanimate objects? Perhaps dropping a rock in non-salt water vs. salt water, and recording the drop time? Regardless, A/B testing is essentially an experiment. You pin version A of a web page against version B and measure/track the goals. 100% of traffic is included in the test, with most likely a 50/50 split (you can always customize traffic allocation if need be). A/B testing uses a simple amount of variables, that way there is a subtle change to users.
A-B comparison. Split testing.
As web designers, A/B testing can be very beneficial. It is normally used for drastic changes or hypotheses of a design strategy. It allows the experimenter to employ two different strategies at once with common goals. The outcome then helps decide which strategy is most effective for the given website user base.
These tests are often the quickest to deploy and can deliver results in a short period of time. It helps anyone in the practice support their ideals with data, in which we use A/B everyday for that very purpose.
Multivariate Testing
Multivariate testing is very similar to A/B testing in its prposes and deployment, but uses a much higher number of variables then just A and B, there’s C, D, E, A1, B27, tons of options included in normal multivariate tests.
“As in an A/B test, traffic to a page is split between different versions of the design. The purpose of a multivariate test, then, is to measure the effectiveness each design combination has on the ultimate goal.”
– Optimizely
It’s much easier to identify specific elements’ effects on a page with multivariate testing. For example, let’s look at a webpage like this:
If we were going to deploy an A/B test on this page, we might try an entirely different layout to test against this, version A. However, for multivariate, we can test multiple things per element. These include (for the page above):
- 5 different colors for the “Add to Cart” button
- The order of the images, each one can be tested in first position
- Video play button icons can be added to test against the control
- The top header (in yellow) can employ 5 different CTA messages
To visualize multivariate check out this diagram:
*Graphic by Optimizely*
Wrapping Up
Don’t think of these two types of CRO tests as opposites or two different methods to do the same thing. They both have great benefits to them that should be employed for any website/company currently exercising CRO.
