Multivariate test design, powered for the interaction

A two-way interaction the same size as a main effect takes roughly four times the sample to detect. That one fact settles most multivariate briefs before design starts: at 4,000 eligible sessions a week against a 3.2 per cent baseline, a 2x2x2 factorial is a five-month commitment, and the useful answer is usually a planned sequence of two-arm tests instead. We say that in week one, and we have said it to close to half the multivariate briefs that reached us. When a factorial earns the traffic: Two conditions. You expect the factors to interact and you can name the interaction you expect, and the levels are genuinely different products of thought rather than three shades of the same button. A pricing page where plan framing and the annual-discount presentation plausibly pull against each other is worth a factorial. A page where someone wants to try four headlines is a sequence, and we will design that sequence instead for the same fee. What we design: Full factorial where the cell count allows it. Where it does not, a resolution V fraction so two-factor interactions stay unaliased with main effects. We do not use Taguchi orthogonal arrays: they alias interactions into main effects, and interactions are the entire reason a factorial is on the table. Randomisation is at user level via a salted hash of the user identifier, salted per experiment so assignment does not correlate across concurrent tests. Blocking by device where mobile and desktop behave differently enough to inflate variance. Sample size comes from simulation on your own historical data rather than from a calculator, because your outcome is a ratio with a heavy tail and the calculator assumes it is not. The design document: Factor list with levels and the argument for each level. One hypothesis per factor, written so that it can turn out false. Primary metric, two guardrails, and the smallest effect each can detect at the agreed horizon. Power curves at 70, 80 and 90 per cent so the sponsor chooses the runtime with the cost visible. Fixed horizon in both days and sessions. The analysis model written and agreed before launch, interaction terms included, so nobody chooses the model after seeing the cells. A QA script for the implementation, and the pre-agreed response if the split check fires mid-run. What we will not do: We will not hand over a design where the winner is picked by the highest observed cell mean; cell means from an eight-cell test are noisy and the model exists for that reason. We will not run bandit allocation on a factorial. Bandits optimise allocation and destroy the clean contrasts you commissioned the design to get; if the goal is learning which factor matters, the traffic has to be spread on purpose. We will not design around factors whose levels three separate teams have not yet agreed, and we will not build the variants ourselves. This is not for you if: You need the winning combination inside three weeks. Your primary metric produces fewer than about 500 conversions a month. Your real question is which of five headlines performs best, in which case you are buying the wrong instrument and a two-arm sequence costs you less. Or the design is wanted mainly so the quarterly report can say a multivariate test was run. How it runs: Two to three weeks from kickoff to signed design, one workshop with the people who own the factors, one working session with whoever implements. We stay available through the run at no charge for questions about the split check and the stopping rule, because a design nobody can execute is not a deliverable.