Geo Lift and Incrementality Testing, Designed for Power

Most incrementality tests fail at design, on power, and nobody finds out until the readout is inconclusive. To read a ten percent effect you generally want thirty to forty comparable geographic units and about five weeks of treatment. South Africa has nine provinces, so a province-level split can only detect something enormous; we run at metro and municipality level, or we tell you the test is not worth its holdout and design something else. The design step: We start with twelve to twenty-four months of weekly outcomes by geography, and the outcome is your outcome, not a platform conversion: policies issued, verified deposits, booked jobs. Treatment and control assignment is solved rather than chosen, matching on pre-period level, trend and volatility. Then a power curve, which tells you the smallest effect this test could detect and how long it takes. If the smallest detectable effect is larger than the effect anybody expects, we say so and stop. That happens on roughly a third of the briefs we scope, and it costs a fraction of what running the test anyway costs. Before launch we run placebo tests on the pre-period: apply the same analysis to windows where nothing happened and confirm the design reports nothing. A method that finds lift in a period with no treatment will find lift in your test too. The commitments we ask for: The metric, the window and the analysis are written down and agreed before the test starts, and the analysis is not re-cut afterwards to find a window that reads well. Everything else in the plan holds still for the duration: no creative refresh, no bid strategy change, no landing page test, no promotional pricing in treatment markets. Finance signs off on the holdout before launch rather than three weeks in, when the pressure to end it early peaks. The readout: An interval, not a point estimate, with the counterfactual plotted against the observed series, and the sentence you can take to a board. Where the interval crosses zero we write that plainly, including what the test does rule out, which is often the useful part. We convert the estimate into an incremental cost per outcome so it can sit next to the numbers your team already uses. On platform lift studies: Meta's conversion lift and ghost-ad based studies answer a narrower question than the one usually asked of them, whether that platform's ads moved that platform's measured conversions. We use them as a cross-check and we have seen them disagree with a geo test by a wide margin. They do not settle a budget allocation across channels and we will not present them as if they do. When geo does not work: Some businesses cannot be split geographically: a national brand campaign, a market with too few units, a category with legal restrictions on regional targeting. For multi-location local-services work we use switchback designs on call and booking volume instead. Where only two markets exist we use a paired difference-in-differences and report it with the weaker claim it deserves rather than dressing it up. Who this is not for: Teams who cannot hold the media plan still for a month. Anyone needing an answer inside three weeks, since the pre-period and treatment window are not negotiable. Advertisers hoping the test will confirm a decision already made; roughly one test in three tells you the channel is doing less than the platform claimed, and that result is the reason to run it.