A Creative Testing Framework for Sparse Conversion Data

Detecting a twenty percent difference between two creatives at eighty percent power takes several hundred conversions per arm. An account producing forty qualified leads a month reaches that figure in roughly a year, by which time the offer, the price and the competitive set have all moved. So the first thing we do is stop you testing headlines and start you testing concepts. The hierarchy we test on: Three levels, and only one of them gets a real experiment. Message territory is the claim the ad makes about why you and not the incumbent supplier. Format is the way that claim is carried. Execution is color, cut, the first three words. Territories are tested against each other with real budget and a decision rule written before the test runs. Formats are rotated on a calendar. Executions are refreshed on a schedule and never declared a winner, because at your volume the difference between two executions is noise wearing a hat. The leading indicator: Reading a territory test on qualified leads takes a quarter you do not have, so we validate a faster signal first. We take twelve to eighteen months of your own data and test which early metric actually tracks qualified lead rate in your account. For most industrial accounts it turns out to be landing page sessions over thirty seconds; for brokerages it is usually WhatsApp conversations opened; for a few accounts nothing correlates at all and we say so rather than inventing a proxy. Click-through rate has never once survived this test in an account we have run, and we will not call a creative on it. Holdouts: Every quarter one territory runs against a geographic holdout, because incrementality is the only reading that survives an attribution window change or a platform update. In Brazil this is unusually easy to do well. States differ enough to matter and cost little enough to hold out. Language: Creative translated from English into Portuguese tests badly, and it is not a language problem, it is a proof problem. What persuades a Brazilian industrial buyer is lead time, homologation, and which company in his own state already buys from you. So the pt-BR work is built as its own territory rather than as a localized version of an English one. The same applies in reverse when a Brazilian brand tests into the United States. What we hand over: A naming taxonomy that survives Advantage+ consolidation, the test calendar, the decision rule agreed before spend, a results log that records the losers and the reasoning, and a briefing document your production team or ours can execute against. Not included: We do not shoot video or produce animation in-house. We write briefs, direct and review; you use your producers, or we bring ours at their own rates with no margin added. We do not manage the media buy under this engagement. We do not run tests inside dynamic creative optimization, because it will not tell you which element won and you will end up guessing with more data. Who this is not for: Accounts spending under roughly fifteen thousand dollars a month on a single platform. There is not enough budget to split and the honest advice is to pick one strong concept and run it until it fatigues. Also not for teams who want a fixed number of new creatives delivered every month. That is a production contract, it is a legitimate thing to buy, and we are not the cheapest place to buy one.