Icon and Screenshot Testing With Pre-Registered Samples

Whether you can test at all is the first question: To read a five percent relative lift on a thirty percent baseline conversion rate at an honest confidence level you need on the order of thirty thousand product page views per arm. Most medtech and utility apps I see do not reach that in a quarter. So the opening deliverable is arithmetic rather than design: your current page view volume, the smallest effect worth acting on, and how many weeks the test would need. When that comes back as eleven months, the correct answer is to stop testing and make a decision. I would rather sell you the decision than a test that will never resolve, and in this category that conversation happens more often than not. How the resolvable ones get run: Apple's Product Page Optimization allows up to three treatments against the baseline, runs up to ninety days, and reports at ninety percent confidence with an improvement range rather than a point estimate. That range is the number that matters and the one teams quote least, because a range spanning minus two to plus nine percent is not a win. Google Play's store listing experiments work differently and report differently. Results do not transfer between the stores, and a screenshot order that wins on Play regularly loses on iOS because the search results surface presents a different number of assets. One variable, pre-registered: Before anything goes live I write down the hypothesis, the metric, the minimum detectable effect and the stop date, and that document does not change afterwards. Tests are not stopped early because a treatment is ahead on day four. The icon and the first screenshot are not moved in the same test. This is the least popular part of the engagement and the reason the results still stand when somebody senior asks whether they are real. What actually moves in these categories: The first two or three screenshots are visible in search results without a tap, so they are doing acquisition work rather than explanation work. In regulated apps the thing that moves conversion is rarely a brighter caption. It is the visible institutional signal: the device class, the health system or utility the app connects to, where the data is held. I have watched that beat a full visual redesign, and it costs less to produce. The claims constraint, again: Caption text inside a screenshot is a claim. A caption promising an outcome will fail review, or worse, pass review and leave an exposure sitting in your store listing. Screenshot copy routes through the same approval path as the listing text, which is why test cycles here carry a fortnight of lead time rather than two days. Plan the calendar accordingly. Not included: Screenshot and icon design production. I direct and specify, and work with your designer or one of mine at your cost. Also excluded: preview video production, metadata text, and paid creative testing, which is a separate discipline with different statistics. Who should not buy this: Teams who want a test to settle an internal disagreement about taste. The traffic does not care about the argument and usually cannot resolve it anyway. Teams who want to move five things at once, which produces five stories and no answers. And anyone who wants the word significant attached to a result the sample cannot carry, because I will not write it.