Independent statistical readout of concluded experiments

Of the 62 concluded experiments clients asked us to re-read in the last two years, 17 lost significance once we corrected for how many times the test had been checked while running, and 4 changed direction once the analysis unit matched the randomisation unit. That is the work on offer here: we read experiments that have already run and tell you what they actually support. What arrives: A readout is a decision document, not a dashboard export. For every test: the point estimate and interval on the primary metric, the same on two guardrails agreed before we look at the data, a sample ratio mismatch check against the intended split flagged below p = 0.001, a triggered-population analysis wherever the change was only visible to part of the traffic, and one recommendation with a reason attached. Ship, revert, or run again at a stated sample size and horizon. How we read a test: Fixed-horizon frequentist by default. Sequential only where the team genuinely needs the option to stop early, and then with always-valid intervals rather than a t-test repeated every morning. Ratio metrics such as revenue per session or listings viewed per visit get delta-method standard errors, because per-session variance is wrong the moment sessions cluster inside users. Revenue is trimmed at the 99th percentile and we report the trimmed and untrimmed number side by side so nobody has to take the trim on trust. Where the warehouse holds pre-period behaviour we apply CUPED; on B2B SaaS trial-start metrics that has generally taken a quarter to a third off the runtime. Past four variants or metrics, Benjamini-Hochberg across the family, stated in the document. What we work from: Event-level data, not a vendor summary screen: GA4's BigQuery export, Snowplow, Amplitude, PostHog, Statsig, Eppo, GrowthBook. Assignment logs matter more than conversion logs. If we cannot see which unit was assigned to which variant and at what moment, we cannot check the split, and a readout with no split check is an opinion in a document with charts on it. What we refuse: We do not write a test up as a win because it crossed 95 per cent at some point during the run. We do not report relative uplift without its interval; plus eighteen per cent with bounds from minus three to plus forty-one is not a result, and the document your CEO reads will say so in the first line. We do not re-cut a flat test by device, browser and country until something emerges, and a segment found after the fact is labelled as a hypothesis for the next test, never as a finding. We do not sign a readout on an experiment whose randomisation we were not permitted to inspect. This is not for you if: The test has already been announced internally as a win and what you want is a second opinion that agrees. Your primary metric is a click on the element the test changed. You need the answer within 48 hours of the test stopping; the checks take days and speed is not what this engagement sells. How it runs: One named analyst, and a second who reviews any readout that reverses a decision the company has already made. Every readout ships as a notebook, R or Python, pointed at your warehouse, so your own analyst can change an assumption and watch the answer move. English or Dutch. We keep no rights over your data and we never publish your numbers.