- How is multivariate testing different from A/B Testing?
- Why should I use it in my email marketing?
- How do I do multivariate testing?
Imagine Katie owns Kangaroo Kickboxing, a business that offers martial arts and self-defence classes.
Thanks to her great workouts, she’s built up a large email subscriber list that’s made up of current customers and other people who want to learn more about Kangaroo Kickboxing.
Katie decides to open second kangaroo kickboxing on the other side of the town so she can increase her customer base. She knows email marketing will be key to getting the word out and filling her new studio with people.
After brainstorming what the email content should be, she narrows it down to a couple different images and calls to action (CTAs).
She’s been playing with different combinations but isn’t sure which will be the most effective.
You just participated in a simple form of multivariate testing. You helped Katie efficiently figure out which email elements (AKA variables) resonate with her customers the most and work the hardest.
Multivariate testing is a lot like A/B testing after it’s gone through Kangaroo Kickboxing strength training.
While A/B testing can only work with one element at a time, multivariate testing can evaluate different, more complex combinations of elements at the same time. For example, let’s say Katie wants to test 2 CTAs with 4 different images.
With A/B testing, she’d first test which CTA worked best. Then, to find out which image worked best, she’d have to test each of them individually with her winning CTA.
Unfortunately, Katie might not get the most reliable results. For example, what if the combination of the losing CTA with the winning image would actually perform the best?
With multivariate testing, she could simultaneously test 8 different versions of her email (2 CTAs x 4 images) to see which combo worked best.
This testing method is fast, effective, and reliable, and can be incredibly valuable to those with large email lists. It even helps you with more than just the current email being tested. It improves all your future emails, too.
That’s because the more of this testing you do, the more you’ll know what works best for successfully marketing your email subscriber list.
Multivariate testing can help you:
- Find Trends: for example, your audience might like drawings over photos
- Optimize Content
- Stayed Focused
If A/B testing puts variables into a bracket tournament, then multivariate testing has variables going full fight club on each other. And the first rule of email Fight Club is: choose your variables.
Services like MailChimp help you do multivariate testing, and most let you test up to 8 versions of your emails. That’s a lot of variables you can throw into the ring.
Your variable might be what time you send your email, content, subject line, sender’s name, background colour, whether or not you include a photo of your new product… basically, anything goes. Don’t be afraid to get creative with your testing.
That being said, you should conduct focused intentional tests based on hypotheses. For example: “I think customers will respond better to direct commands vs. gentle wooing. I’ll test a pushy subject line against a subtler, more enticing one.
Even if multivariate testing proves your hypothesis wrong, it’s still a win. You’ve just learned something new and valuable about your audience’s preferences.
Okay, you’ve chosen your variable and now you’re ready to test. But wait… did you give your email versions enough room and time to properly compete?
There’s no definitive answer on how many subscribers you need or how long should run your test. It depends on what you’re testing (opens, clicks, or purchases) and how big of a change you’re looking to find.
In general, you’ll need a large enough testing pool to make sure your test results and the decisions you make from them represent a good amount of your audience.
And, you need to run your tests long enough for the right customer engagement data (click on the links, sales on your website, etc.) to come in.
To help you figure out your testing pool size, you can use “sample size” calculators on sites like Optimizely and Qubit. As for test length, one rule of thumb is to conduct a test for at least 4 hours.
Once you’ve started running your tests, look for meaning in this multivariate world. That is, know what to look for in your results and how to interpret those insights.
On your testing service’s report page, check out your combination results to see which mix of variables is most effective. This can help you discover insights like: “People hate this CTA and this image on their own, but love them together.
Next, look at results per variable. This tells you how much each individual variable affected your email’s effectiveness across all combinations. That way you’ll know if, for example, your CTA matters more than the image you chose.
Also, know what success looks like. Industry standards say a 20-30% open rate is generally the best indication of a strong email. However, if you’re mainly testing your email’s content, click rate is a good statistic to consider.
Hold off on multivariate testing for now. Depending on where you are, your next steps might be to focus on your email marketing strategy or to continue building your mailing list.
References: Google Webmasters, Think With Google, Google Primer
