A/B Test Program Setup and Decision Governance

When I re-analyse an existing experimentation programme, most of what was called a win does not survive. Across the last four audits, the share of shipped winners that failed re-analysis ran between 55 and 80 percent, and the causes repeat: no primary metric declared before launch, results read while the test was still running, sample ratio mismatch nobody checked, and one person designing the test, analysing it, and being measured on the number it moved. The setup: Platform choice follows where the decision has to happen. If your changes touch money, sessions or account state, testing runs server-side behind feature flags with assignment logged from the server — GrowthBook self-hosted on your own infrastructure when the data has to stay there, Statsig or Eppo when you want the analysis layer managed. Client-side tools such as AB Tasty or Kameleoon keep their place on marketing surfaces, where flicker is tolerable and nothing financial is at stake. I do not put payment or account logic behind injected client-side scripts and I will not sign off a design that does. The statistics, in the amount you actually need: One primary metric per test, written down before traffic starts. A minimum detectable effect derived from your traffic rather than from what somebody hopes to find: if the honest number is 9 percent, the test hoping to prove 2 percent should not run. Sequential analysis where the team genuinely cannot wait for a fixed horizon, so that looking early becomes legitimate instead of cheating. Variance reduction from pre-period behaviour, which cuts the required sample materially on returning-user metrics. An automatic sample ratio mismatch check that aborts the test rather than filing a warning nobody opens. Secondary metrics reported as exploratory, every time, without exception. The governance, which is the part that fails: A registry every test enters before launch, holding the hypothesis, the primary metric, the detectable effect, the guardrails and the stop rule. A standing decision meeting where results are read aloud by someone who did not build the variant. A rule for rolling losers back out that has actually been used, because a programme which never removes anything is not testing, it is shipping. And a small permanent holdout, usually 5 percent, so that once a year you can answer the only question your finance director asks: what the whole programme was worth against doing none of it. What I refuse: Velocity targets. A team told to run twelve tests a quarter runs twelve underpowered tests and reports false winners; I have seen a programme's claimed cumulative uplift exceed the company's actual revenue growth several times over. I also refuse to both design and adjudicate. My work ends at the decision rule, and being asked to own the KPI as well is a different and worse arrangement. Not included: writing the variants, building the pages, data engineering beyond the event contract for assignment and exposure, and BI dashboard work. I train your analysts. I do not become them. Not for you if: your site cannot resolve a 10 percent effect on a primary metric within a month. There is a real answer for that situation — qualitative research, sequenced shipping with pre/post reads, painted-door tests — and it is not a testing platform with a monthly licence. What you get: the platform running, the event contract documented, the registry and its templates, four tests taken end to end with your team watching, and a written decision protocol short enough that people follow it after I have gone.