Icon and Screenshot Testing on Play Experiments and Apple PPO

The power calculation comes before the quote. On a 25 percent product page conversion rate, detecting a five percent relative change at 95 percent confidence needs roughly 20,000 product page views per arm, so a control plus three treatments is about 80,000 views, and a listing producing 900 views a day is looking at three months for one answer. We run that arithmetic on your actual App Store Connect and Play Console numbers before you commission anything. If the maths says the test cannot conclude, we say so on the first call rather than sell it and report a coin flip in week six. What we test, in what order: The icon first, because it is the only element that moves the impression to page view step, and because it is the cheapest thing to be wrong about. Then the first two screenshots, which on an iPhone search result are most of what anybody actually sees. Then the rest of the gallery, if there is traffic left to spend. We do not test caption wordings against each other. A four-word change on screenshot four cannot plausibly move conversion by five percent, so testing it burns the traffic a real question needed. How the tests are run: On Play, Store Listing Experiments at 95 percent confidence rather than the 90 percent default, and never stopped early because a variant is ahead. Play's live reporting makes peeking easy, and peeking is how shops produce winners that never replicate. On Apple, Product Page Optimization: up to three treatments against baseline, a 90-day ceiling, results read against the treatment split and not against last month. Custom product pages are a different instrument and we use them for campaign landing, not for testing. When your traffic is not enough: We say so, then offer panel testing instead: a recruited audience, a simulated store result, click and dwell measured. It is directional and it is labelled directional every time it appears in a document. It tells you which of two icons a stranger prefers. It does not tell you what your search traffic will do, and treating a panel result as a store result is the most reliable way to produce confident nonsense. Category specifics: Beauty apps live and die on the before-and-after frame, and Apple's handling of cosmetic claims means the version that can ship is often not the version that tests best. We design the treatments knowing that, rather than discovering it at review. Banking screenshots have to show sample data with a visible disclaimer, and that disclaimer occupies vertical space in exactly the area where the value proposition should sit. The trade-off is itself worth testing and almost nobody tests it. Not included: Brand identity work. If your icon has to change because the company rebranded, that is a design engagement and we are not it. We test candidates you already have and we produce alternates within an existing identity. Also excluded: metadata rewriting, video production, paid media management. Not for you if: Your listing produces under roughly 500 product page views a day and you want statistically valid answers, because the numbers do not exist and no methodology conjures them. Not for you if a test has to conclude before a board meeting, because we will not shorten one to fit a calendar. And not for you if you want a dozen variants at once. Running four arms on traffic that supports two is how a year of testing ends with nothing anybody can act on.