An A/B test is easy to understand: you change one element (a headline, a button colour, a photo), split traffic in half, and see which one converts better. A multivariate test (MVT) looks, at first glance, like a more ambitious version of the same thing: instead of changing one element, you change several at once (headline, image, button, offer text) and test every possible combination between them. The question almost nobody asks before jumping into an MVT is whether it actually makes sense for the traffic volume their business handles, because for most SMEs, it does not.
How a multivariate test actually works
If you test two versions of a headline, two versions of an image and two versions of a button at once, you are not comparing two pages, you are comparing eight different combinations (2 x 2 x 2), each receiving only a fraction of the total traffic. With three elements and three variants each, that is already 27 combinations. The number of combinations grows multiplicatively, not additively, and each combination needs enough traffic of its own for the result to be statistically reliable.
The real problem: almost no small business has enough traffic
Here is the main reason MVT rarely makes sense for an SME: if a simple A/B test already needs several thousand visits per variant to reach statistical significance within a reasonable timeframe, a multivariate test with eight combinations generally needs several times that traffic volume to reach equally reliable conclusions about each combination. A business with a few thousand monthly visits to the page in question can take months (or never get there) to reach reliable MVT results, while the same test split into several sequential A/B tests would have given useful answers much sooner.
When a multivariate test does make sense
MVT is mostly useful when the goal is not just knowing which variant wins, but understanding how several elements interact with each other (for instance, whether a specific headline performs better paired with a specific image, and worse with other combinations), something a series of independent A/B tests cannot fully capture. This only adds real value when traffic volume is high (sites with hundreds of thousands of monthly visits to the page in question) and when there is a genuinely reasoned suspicion of interaction between elements, not just generic curiosity about testing several things at once.
The usual alternative: sequential A/B tests
For the vast majority of businesses, the more practical alternative is running several A/B tests one after another, testing first the element suspected to have the most impact (usually the headline or the main value proposition), applying the winner, and then testing the next element on top of that already-improved base. It is slower in total number of tests, but each individual test reaches a reliable conclusion within a reasonable time, and the learning accumulates more clearly and traceably.
How to decide which element to test first in an A/B sequence
Not every element on a page has the same impact potential. As a general rule, it is worth prioritising elements higher up the page (the first thing seen) and those most directly related to the conversion decision (the main headline, the value proposition, the price or the offer), ahead of minor details like an icon's exact colour, whose impact tends to be much more limited.
Common mistakes when setting up either type of test
The most common mistake, in both A/B and MVT, is calling the test finished before reaching statistical significance, simply because one variant "is ahead" after a few days. Natural daily traffic variation can make a variant look like a winner at a given moment and lose that lead a few days later, so it is worth setting the required sample size in advance and waiting to reach it before drawing final conclusions.
The cost of making the test more complex than necessary
Beyond the traffic needed, an MVT also means more setup complexity, more risk of technical error (a specific combination displaying badly on some device, for example), and results that are harder to explain within the team. For most business decisions, a series of clear conclusions from simple A/B tests delivers more practical value than a statistically more sophisticated multivariate result that is much harder to translate into concrete actions.
An example of a poorly framed decision
An online business school, with moderate traffic to its enrolment page, wanted to test three different headlines, two different header images and two different button colours all at once, twelve combinations in total. After six weeks of testing, no individual combination had reached the visit volume needed to declare a winner with reasonable statistical confidence, and the team faced the hard choice of either continuing to wait with no useful conclusion, or cancelling the test having learned nothing solid. On reframing the approach as a sequence of three A/B tests (first the headline, then the image on top of the winning headline, and finally the button colour on top of both already-made decisions), the first test gave a clear result within just ten days, and overall, the full sequence of three tests wrapped up in less time than the failed multivariate attempt had already taken without getting anywhere.
How to communicate the results of a test sequence to the rest of the team
An additional advantage of sequential A/B tests, beyond speed, is that each individual result is much easier to explain to someone with no statistical background: "we changed the headline and enrolments went up 8%" is a sentence anyone on the team understands and can use to justify future decisions, while a multivariate result with interactions between variables usually requires a much longer, more technical explanation that rarely lands the same way across the rest of the organisation.
Calculating sample size before starting, not after
A practice that separates people who run A/B tests with rigour from those who run them by gut feeling is calculating, before launching the test, how many visits per variant are needed to reliably detect the minimum difference considered relevant to the business. Free sample size calculators exist that only ask for the current conversion rate and the minimum improvement you want to be able to detect, and the result avoids both ending tests too early (with unreliable conclusions) and running them longer than necessary, wasting traffic that could be validating the next hypothesis.
Step by step: designing and launching the first A/B test sequence
For a team that has never run an A/B test, it is worth starting with a guided process. First, choose the page with the most traffic and the greatest potential business impact (usually the homepage, a campaign landing page, or the first step of a purchase process), not a low-traffic secondary page. Second, identify the element most likely to have an impact: on most pages, that element is the headline or main value proposition, not a minor visual detail. Third, calculate the required sample size with a free calculator, entering that page's current conversion rate and the minimum improvement worth detecting (usually between a 10% and 20% relative improvement). Fourth, create the alternative variant, changing only that element and keeping everything else exactly the same. Fifth, launch the test and resist the temptation to check results daily; only review once the calculated sample size has actually been reached. Sixth, apply the winner, document the result with the concrete improvement figure, and repeat the process with the next priority element. Following this sequence with discipline, instead of launching standalone tests with no prioritisation criteria, is what turns experimentation into a steady source of cumulative improvements.
Frequently asked questions
What traffic volume counts as enough to seriously consider a multivariate test?
There is no universal number because it depends on the base conversion rate and the number of combinations to test, but as a rough guide, most tools and consultants only recommend it starting from tens of thousands of monthly visits to the page in question.
Can I combine A/B and MVT tests on the same website?
Yes, they are not mutually exclusive: it is common to use A/B for most decisions and reserve MVT for very high-traffic pages where understanding complex interactions between elements genuinely matters, like a high-volume homepage or a heavily visited checkout process.
Do I need specialised software to run a multivariate test?
Almost always yes, because manually managing traffic allocation across multiple combinations and the subsequent statistical analysis is much more complex than a simple A/B test, which can sometimes be approximated with more basic tools.
Does a slower, sequential A/B approach give worse results than a well-run MVT?
Not necessarily worse, just different: MVT captures interactions between elements that the A/B sequence does not see, but for most small businesses that extra information does not make up for the additional time and traffic it requires, so the final practical result tends to be similar.
How long should I let an A/B test run before looking at results?
At minimum, long enough to cover a full weekly cycle (so as not to confuse behaviour variation between weekdays and weekends with the test's real effect), and always until reaching the sample size calculated in advance, not an arbitrary fixed number of days.
Is it worth running A/B tests if my site gets very little traffic?
With very low traffic, tests take a long time to reach reliable statistical significance. In those cases it usually pays off more to prioritise changes based on already widely validated best practices (heatmaps, session recordings, usability tests with a few users) rather than relying on formal A/B tests.