Push Creative Testing With Pre-Registered Thresholds

Detecting a 15 percent relative change on a 1.2 percent click rate takes roughly 58,000 sends an arm. Most push testing we are asked to inherit has been calling winners on 4,000. That is the whole problem: the tests are real, the arithmetic is not, and the base gets spent proving nothing. We pre-register the effect size, the sample and the stopping rule before the first send, and we test one class of variable at a time. What actually moves a push: The title. On Android Chrome the notification collapses to roughly 40 to 50 characters before the body hides behind an expand control, so the title carries the offer and the body is a second chance almost nobody takes. We test title classes, not word swaps: named price, named deadline, named object, question. Icon and image are tested once a quarter as a pair, because a 192-pixel icon and a 2:1 hero image are a single visual unit on the lock screen and testing them apart produces winners that do not replicate. Action buttons are tested last and usually removed. Japanese creative is not translated creative: A Japanese title truncates around 20 to 24 characters. An English title that fits in 45 has no Japanese equivalent that fits in 22, so a translated matrix arrives pre-broken and every arm loses to control for reasons that have nothing to do with the offer. We build the Japanese matrix from the offer, in Japanese, and we treat kanji-dense against kana-open phrasing as its own variable class. It moves click rate more than any headline formula we have tried, and it moves it in opposite directions on iOS and Android. What a winner has to survive: Every test that wins on click gets a 28-day follow. We compare the winning arm's cohort against control on opt-out per send and on click rate at day 28. Creative that beats control by 20 percent on click and costs 40 percent of that cohort's fourth-week click rate is a loss, and we report it as one. This is the reading testing programmes skip, and it is why bases that test constantly still decay. What we will not write: No fake system chrome. No lock icons, no fake reply threads, no notification dressed as a delivery status, a bank alert or a message from a person. No countdown that resets. No pharmaceutical or medical device claims in a Japanese push title, because the qualifying language required under the Act on Pharmaceuticals and Medical Devices does not fit in 22 characters and a compliant push in that category is usually a push that should not be sent. We will not run a test whose result cannot change a decision you have already scheduled. Who this is not for: Bases that cannot supply 58,000 sends an arm inside a fortnight; the test will move slower than the market it is testing. Teams who want a creative volume commitment, because a programme producing forty variants a month is producing them for a report rather than a decision. Advertisers whose approval chain adds a week between draft and send, which makes the stopping rule unenforceable. What arrives: The test register: each test with its pre-registered effect size, sample, stopping rule and decision date, filled in before the send and closed after the 28-day follow. The creative set in English and Japanese, with truncation checked in a real notification on a real handset rather than in a design file. A written reading of what won, what failed to replicate, and what it cost the base.