Hypothesis
If the first screenshots show the app's core outcome instead of its interface, more store visitors from search will install, because the listing answers what the app does before they scroll. The decision behind it: replace the store listing's screenshots or icon with a new set, or keep the current listing.
The hypothesis names a source, store search, on purpose. People who search for a category or problem arrive with a need and judge the listing on whether it meets it, which is exactly what outcome-led screenshots address. People browsing charts or categories, and people arriving from a campaign, are in different states, and the same screenshots may help one group and do nothing for another. The app store optimisation guide covers what screenshots, icons and previews should do; this test takes one proposed change and decides whether it earns its place. Write the hypothesis, the priority source and the guardrail limits into the experiment hypothesis template before launch.
Metrics and guardrails
The primary metric is install conversion rate, first-time installs per product page view, for the priority traffic source. Report the other sources too, each on its own, but decide on the one named in advance. The conversion rate calculator page for app store listings explains which console figures make up the CVRShare of visits or clicks that convert. and why page views must come from one source at a time; use its definitions, unchanged, for both arms.
The guardrails catch wins that cost something later. Early retention of installers, read by install CohortA group of customers who share a starting event and period, tracked together over time. in the app's analytics or the mobile measurement partnerAn independent service that attributes app installs and in-app events across ad networks., shows whether the new screenshots attract people who stay; a listing can win installs by promising more than the first session delivers. The ratings and reviews trend during and after the test shows whether new installers are disappointed. CPIAverage ad cost of each app install. for campaigns landing on the listing shows whether the change helps or hurts paid acquisition, since every paid install passes through the page.
Design
Use the store's native listing experiment. Apple's product page optimisation runs treatments of the icon, screenshots and previews against the original and reports conversion for each; Google Play's store listing experiments do the same for the icon, graphics and descriptions, with options for localised tests. The stores assign visitors between the current listing and the variant, so the unit of randomisation is the store visitor, assigned by the experiment. Change one asset set only, and keep the title, description, ratings display and everything else the same.
| Design element | Choice | Reason |
|---|---|---|
| Unit of randomisation | Store visitor, assigned by the store's listing experiment | The store controls the split on its own page |
| Arms | Current asset set versus one new asset set | One hypothesis per test |
| Changed asset | Screenshots or icon, not both unless the set is the decision | Keeps the result interpretable |
| Traffic sources | Search, browse and paid read separately | Sources convert differently; blended rates move with the mix |
| Priority source | Named before launch | Prevents choosing the winning source afterwards |
| Retention check | Installs from the test joined to cohort reports | Catches listings that over-promise |
Read search, browse and paid traffic separately wherever the store's reporting allows, using the console's source breakdown for the test period alongside the experiment's own result. Campaigns that land on a custom product page or a custom store listingA variant of an app's store page, with its own address and screenshots, built for a specific campaign, audience or feature. send their traffic to a different page from the default listing under test; the app install campaigns guide explains how those pages are matched to ads, and the guardrail should be read only for campaigns that land on the default listing.
Sample size reasoning
Store traffic sources convert very differently, so size the test for the source that matters most rather than for blended views. A source with high conversion and plenty of page views can detect a modest change quickly; a thin source may never reach a reliable read, which is a reason to name a different priority source, not to blend them. The conversion rate calculator page for app store listings shows how the uplift input sizes a listing experiment, and the sample size and significance guide explains how the baseline rate, the minimum effect and the confidence level combine into the page views each arm needs. Decide the smallest lift that would justify producing and maintaining new assets before launch.
Duration
Run for at least two full weekly cycles in the store experiment, and longer for low-traffic listings, so that weekday and weekend audiences are both represented. Avoid launching during a store feature, a launch spike or a large campaign burst, any of which would change the mix of visitors for part of the test. Then wait for the early-retention window to close for installs from the last days of the test before making the final decision.
Reading the results and the decision rule
Read the store's result for the priority source at the planned end, with its confidence range, then the other sources, then the guardrails. The rule, written before launch:
| Result | Decision |
|---|---|
| Install conversion rises for the priority source and early retention holds | Apply the new assets |
| Conversion rises only for sources whose installs retain poorly | Keep the current set |
| Conversion rises but early retention or ratings fall | Keep the current set and revise the screenshots to match the first session |
| No reliable difference for the priority source | Keep the current set, or test a bolder change |
The app installs through app stores page explains why listing work comes before store ads in an installs plan, which is where a winning listing pays off across every channel.
A running app tests outcome-led screenshots
Pacecraft, an invented running-plan app, replaces its first three screenshots, which showed menus and settings, with images of a finished plan and a race-day result. The team names store search as the priority source and runs the store experiment for three weeks. In store search, each arm receives about 25,000 product page views; install conversion is 30.0% for the current set and 32.4% for the new one, with a 95% interval for the difference of roughly 1.6 to 3.2 points. Browse traffic shows no reliable difference. Seven-day retention of installers from the test period is level, and ratings do not move. Pacecraft applies the new screenshots.
Pitfalls
Blended traffic sources are the first pitfall. A week with more browse traffic lowers the overall rate without any change in the listing, and the reverse flatters it; reading one source at a time removes that noise. Featuring spikes in one period are the second. If the app is featured partway through the test, the visitor mix changes for both arms; the split protects the comparison, but the result then describes a different audience, so extend the test or rerun it. Installs that do not retain are the third: the store reports the install, not what follows, so the retention join is not optional.
Watch for seasonal relevance as well. Screenshots tied to a season or an event can win during it and lose after; run the test in an ordinary period unless the change is meant to be seasonal.
When a winning set is applied, run the app store listing checklist on the updated listing before it is submitted, so the new screenshots go out with accurate text and privacy details.
Common mistakes
Listing tests are easy to run and easy to misread. The push permission prompt test is the matching design for the first minutes after install, and the experiment library lists every design.
- Reading blended page views. Decide on the priority source named before launch.
- Changing several assets at once. Then nothing says which change mattered.
- Ignoring retention. A listing can win installs by over-promising.
- Stopping after a few good days. Run full weekly cycles and avoid featuring spikes.
- Judging paid traffic on custom pages. Those visitors never saw the listing under test.
- Testing on taste instead of a hypothesis. State what the new assets should change and for whom.
