Definition
A multivariate test, often shortened to MVT, is a controlled experiment with three ingredients.
- Elements are the parts of the page being changed, such as the headline, the hero image or the call to action.
- Versions are the alternatives written for each element, including the current one.
- Combinations are every possible way of assembling one version of each element. The number of combinations is the product of the number of versions per element.
Visitors are randomly assigned to combinations, and the analysis estimates two kinds of effect. A main effect is the average difference one element's versions make, across all the others. An interaction effect is a difference that appears only in certain pairings: a headline that works with one image and fails with another. Interactions are the reason to run the design at all; if the elements are independent, a series of simpler tests would find the same answers with less traffic.
Full factorial and reduced designs
A design that tests every combination is called full factorial. It is the cleanest form, because each element's effect can be estimated against every version of every other element, and it is the form most people mean by a multivariate test. Some testing tools also offer reduced designs that run only a chosen subset of combinations and estimate the rest. They save traffic by assuming that some interactions are too small to matter, which is exactly the thing a multivariate test is usually run to check, so their results deserve more caution.
In practice: traffic requirements and when it fits
Traffic decides almost every case. Each combination is in effect its own arm of the experiment and needs a Sample SizeThe number of users or sessions an experiment needs to detect a given effect reliably. large enough to detect the difference that matters, so the visitors required grow with the number of combinations. Many simultaneous comparisons also raise the chance that one of them looks like a winner by luck, which makes reaching Statistical SignificanceA test result unlikely enough under 'no real difference' to pass a threshold chosen in advance. harder to interpret. The sample size and significance guide explains how a sample is planned; for a multivariate test that plan is made per combination, not per page. The conversion rate calculator is a quick way to see the baseline rate each combination would start from.
The method fits a high-traffic page, such as a main Landing PageThe page a visitor arrives on from a campaign, built to turn that visit into one action., a product template or a checkout step, where several independent elements are worth tuning together and the team suspects they interact. It also suits refining a page whose layout has already been settled by bolder tests. It fits poorly on pages with modest traffic, for redesigns, and for any question that is really about one change. The test prioritisation guide describes the choice between running overlapping tests on one page, running them in sequence, and folding them into a single multivariate design.
Multivariate versus A/B and A/B/n tests
An A/B TestControlled comparison of two variants. compares complete versions: a control and one variant, or several variants in an A/B/n test. It answers whether a version wins, not which of its changes caused the win. A multivariate test answers the second question and the interaction question, at a much higher traffic cost. The A/B testing versus multivariate testing comparison sets the two side by side on traffic, duration, analysis and risk of false positives. For the mechanics shared by both, from randomisation to the primary metric, the A/B testing guide is the reference, and the experimentation hub places both among the other test designs.
A print shop sizes its test before running it
Wickerquill Print Shop wants to test three headlines, two hero images and two button labels on its business-card page, which gets about 9,000 visitors a month and converts at 3%. The design has 3 × 2 × 2 = 12 combinations, so each would receive about 750 visitors a month and record around 22 orders. The team's sample size plan needs roughly 4,000 visitors per arm to detect the lift it cares about, which would mean more than five months. It runs the headlines as an A/B/n test on the business-card page first, then the image, and keeps a multivariate design for its home page, which receives about 60,000 visitors a month.
Related terms
An A/B TestControlled comparison of two variants. is the simpler design that most tests should use. Sample SizeThe number of users or sessions an experiment needs to detect a given effect reliably. planning shows, before launch, whether the combinations can be filled. Statistical SignificanceA test result unlikely enough under 'no real difference' to pass a threshold chosen in advance. is harder to read when many combinations are compared at once. A high-traffic Landing PageThe page a visitor arrives on from a campaign, built to turn that visit into one action. is the typical place where a multivariate design is worth its cost. The wider discipline is set out in the conversion rate optimisation guide.
Common confusions
- An A/B test with several changes is not a multivariate test. It tests one bundle against another and cannot separate the changes.
- More elements do not mean more learning. Without the traffic, they mean an inconclusive result.
- A winning combination is not proof of an interaction. Main effects can explain a win on their own.
- Reduced designs are not free. They save traffic by assuming away some interactions.
- Multivariate is not the advanced version of A/B testing. It answers a different question and is the wrong tool for most of them.
