Multivariate Test Design, Or the Case Against One

Read this before commissioning one: Most requests for a multivariate test should be an A/B test. Four factors at two levels is sixteen cells. To detect a 10 percent relative lift on a 3 percent baseline conversion rate at 80 percent power you need roughly 52,000 sessions per cell, which is about 830,000 sessions for the full factorial. If that is more than a month of traffic on the page in question, the design is wrong and no tool fixes it. About half the multivariate briefs that reach me end here, inside a forty-minute call, and that call is the entire engagement. When it is the right instrument: When the question is genuinely about interaction. Whether a price anchor behaves differently with a 14-day trial than a 30-day one. Whether the hero image that wins alone still wins under the longer headline. An A/B test answers neither: it averages the interaction away and you ship a combination nobody measured. If you can state the interaction you suspect in one sentence before the test, this is worth designing. Fractional designs: The full factorial is rarely the efficient answer. A Resolution IV fractional factorial handles four to seven factors in eight runs with main effects clear of two-factor interactions, which is a fraction of the traffic for the question most teams actually have. Where the brief is screening rather than estimation, a Plackett-Burman design in 12 runs ranks a dozen factors by main effect and tells you which three deserve a real test. The aliasing structure is written out explicitly in the design document, because a fractional design whose confounding pattern nobody understood is worse than no test at all: it produces a number, and the number is the sum of two effects. On Taguchi arrays, which come up in almost every brief: they were built for manufacturing, where a run can be replicated and interactions are treated as noise. Web traffic does not replicate and the interactions are usually the point. I do not use them here. What the design document contains: The factor list with level definitions written tightly enough that two engineers cannot implement them differently. The design matrix and its aliasing structure. A power table across plausible baselines, so you can watch the design fail on paper before anyone builds it. The pre-registered analysis plan, including how main effects and interactions are tested and how the family is corrected. The stopping rule and the calendar horizon. An implementation spec for whichever tool you run: cell definitions, assignment keys, exposure events. And the abort conditions, meaning what has to be true two weeks in for the test to be stopped rather than nursed to the finish. Not included: I do not build the variants and I do not run the test day to day. No copywriting, no design production, no implementation in your codebase under this engagement. If you want design and build from one person that is a different service, priced differently, and I will say which one you actually need. Who this is not for: Teams below the traffic threshold, which is most teams who ask. Teams who want a multivariate test because a sequence of A/B tests felt slow: it is not faster, it is a different question with a larger appetite. And teams who intend to run the design as personalisation, multiplying cells by segments, which turns sixteen cells into a hundred and guarantees the answer never arrives. What I do instead, often: Convert the brief into a two or three test sequence with a stated order and one explicit interaction check at the end. It looks less impressive on a roadmap and it resolves inside a quarter, which is the trade I recommend taking.