Variant landing sets with a test design that can resolve

We do not take a test that cannot collect roughly 400 conversions per arm inside four weeks. Below that, the smallest effect a two-arm test detects at 80 percent power is about a 20 percent relative lift, and honest landing page changes are usually worth 5 to 12. Under the threshold we build one page, ship it, and tell you to spend the test budget on a question the data can answer. Before we agree to a test: We ask for four weeks of conversion counts on the destination, the split you intend to run, and the metric you will act on. Then we do the arithmetic in front of you. Roughly half the tests brought to us do not survive that conversation, which is the most useful hour we spend with a new client and the cheapest. What a variant is here: Each arm carries one falsifiable claim about why the buyer hesitates: the price is not credible without a comparison, the lead time is the real objection, the compliance question outranks the offer. Each gets the layout its argument needs. Button colours, headline synonyms and hero image swaps are not arms, they are noise with a project code. How variants are served: Assignment happens at the edge, in a Worker or in middleware, with the arm held in a first-party cookie and stamped onto the conversion record. Client-side testing tools hide the body behind an anti-flicker snippet whose timeout is measured in seconds; on a paid landing page that snippet becomes the LCP element, and the test ends up measuring its own instrumentation. We do not use them for this work. How a result gets called: Sample ratio mismatch is checked before anything else, and a split failing at p below 0.001 stops the test and starts an instrumentation fix instead of a readout. Most disputed winners we have been asked to review were mismatches. The primary metric, the arm count and the stop date are written down before traffic starts. Where a client needs to look early we use a sequential method that stays valid under repeated inspection, rather than pretending nobody peeks. Not included: Ad creative and media buying. Personalisation engines and audience segment builds. Retrospective analysis of a test that already ran without a pre-registered plan; we will read it and give you an opinion, but we will not put a confidence figure on it. Tool licences, which you hold directly. Who this is not for: Anyone who needs a winner by a board date. The test finishes when it finishes, and a result called early to fit a calendar is a coin toss with a slide deck behind it. Also not for products taking a handful of conversions a day, where the answer arrives a quarter after the decision it was meant to inform. What we refuse: We refuse full factorial designs on this kind of traffic. The interaction cells stay empty and the report is decorative. We refuse to declare a winner on a micro-conversion when the money metric moved the other way, and we will show you both series side by side. And we refuse to keep a losing arm alive because someone senior wrote the headline; the pre-registered stop date exists to make that conversation short.