On August 24th, 2021, the Google Analytics Twitter channel had a tweet about multivariate testing.
The tweek linked to a page on the Optimize Resource Hub that talks about experiments. (By the way, in case you’re not sure what it is, Google Optimize is basically a platform you can use to run experiments on your website.)
The tweeted page describes 3 types of experiments: A/B, multivariate, and redirect tests.
A/B Tests (Also Called Split Tests)
I actually wrote about split tests here, which, in my opinion, serves as a good description of what they are.
That said, I can summarize what an A/B test is.
Let’s use a hypothetical example.
Let’s say you run a non-dairy ice cream shop. You have 2 options of sprinkles available, but you want to know which one most of your customers prefer, or if there’s one clear difference.
So, you make those 2 options available: sprinkle option A (say, chocolate chips), and sprinkle option B (cookie dough).
For every customer that wants sprinkles, they have a choice of option A or option B.
You keep count of each customer and, if they wanted sprinkles, the option they made.
After about 500 customers (which you figure is a statistically large enough number) who preferred sprinkles, you count: 75% of those who wanted sprinkles chose chocolate chips.
That’s an example of an A/B test. You test 2 variations of a variable (in this case, the variable is ice cream sprinkles, and the variations are chocolate chips and cookie dough).
Of course, that was an offline example, but the exact process applies to the online world.
Multivariate tests (MVT) are where, instead of testing 2 variations of a single variable, you may test more than 3 variations, or even multiple variables in certain combinations. (Google has a good definition of multivariate tests here.)
A redirect test can be part of the set up of an A/B or split-test. I defined a split test as testing 2 variations of a variable.
Well, let’s say you’re working on a web page, and you’re 99% done, but you’re uncertain which headline to use. You have 2 options of headlines you can use.
So, as part of your A/B testing, you set up Page A (with headline A), and Page B (with headline B).
You then send equal numbers of your traffic to each of these variations. Then, once you’ve sent a statistically significant amount of traffic to these variations, you count your sales from each headline, and the headline that brought in the most sales (as long as it’s by a statistically significant enough margin) wins.
(That said, I think you can use redirect tests in multivariate testing as well.)
In short, that’s what A/B, multivariate, and redirect tests are, and Google Optimize is a great platform to use to start learning the mechanics behind these.
Source: Google Analytics Twitter channel