CTV Incrementality Testing With Powered Geo Holdouts

Why most CTV tests come back inconclusive: A geo holdout has to be powered before it is bought. At a 3 percent baseline conversion rate, $200,000 of media and an eight week window across the 210 US DMAs, the smallest lift separable from noise sits somewhere around 8 to 12 percent. If the number your finance team wants validated is a 4 percent lift, that test cannot answer the question, and an inconclusive result will be read around the table as proof that CTV does not work. So the power analysis comes first, before any media is committed, and occasionally it ends the engagement. That is a cheaper ending than the alternative. How the test is built: Two years of weekly DMA-level outcome data where it exists, one year as the floor. Markets are paired by synthetic control against your own history rather than by population size, using GeoLift or CausalImpact, and the holdout takes 20 to 30 percent of markets weighted by baseline volume rather than by count. Holding out thirty small DMAs protects nothing and costs you the read. Then a two week burn-in, an eight to ten week measurement window, and a decision rule written down and agreed before the first impression serves: this lift number leads to that budget action, signed by whoever owns the budget. What comes back: A point estimate with its confidence interval, incremental cost per acquisition, and a plain statement of what the test could not see. If the interval crosses zero, that is the finding, and it is delivered as the finding rather than dressed up with a supporting metric that happens to look better. What I refuse: I do not run a test the power analysis says cannot resolve. I do not move the holdout mid-flight, however loudly a regional sales lead objects to being dark. I do not report a view-through or last-touch figure alongside the causal one, because when two numbers sit in the same room the larger one wins the argument regardless of which is true. And where I have also bought the media, I do not deliver the verdict on it: the design is agreed with your analytics lead up front and the raw geo series go to them or to your marketing mix vendor to read. Not included: Marketing mix model construction. Attribution instrumentation and tagging, which is separate work. User-level matched testing and clean room panel studies. Media buying, planning or negotiation inside this engagement. Creative testing, which needs a different design entirely and usually a different budget. Who this is not for: Businesses running under roughly 400 conversions a week nationally, where the noise floor swallows any lift worth finding. Considered purchases with a sales cycle longer than the window, where eight weeks reads the wrong period. Anyone who needs an answer inside three weeks. And teams where a negative result would be politically unusable. If the honest answer cannot be acted on inside your organisation, do not commission the test; that is a defensible position and it saves everyone the money and the argument. What arrives: A power analysis document with the minimum detectable effect at your actual baseline and spend, which is useful even if you never run the test. The market assignment with its matching diagnostics, so your analyst can audit the pairing rather than trust it. A pre-registered analysis plan. The final read with confidence intervals and code, in R or Python, that reproduces every number on your own data.