Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Experiment Library

App Store Screenshot Test

An experiment design for app store screenshots or icons using the stores' listing experiments, read on install conversion per traffic source.
Experimentintermediate6 min readUpdated

Hypothesis

If the first screenshots show the app's core outcome instead of its interface, more store visitors from search will install, because the listing answers what the app does before they scroll. The decision behind it: replace the store listing's screenshots or icon with a new set, or keep the current listing.

The hypothesis names a source, store search, on purpose. People who search for a category or problem arrive with a need and judge the listing on whether it meets it, which is exactly what outcome-led screenshots address. People browsing charts or categories, and people arriving from a campaign, are in different states, and the same screenshots may help one group and do nothing for another. The app store optimisation guide covers what screenshots, icons and previews should do; this test takes one proposed change and decides whether it earns its place. Write the hypothesis, the priority source and the guardrail limits into the experiment hypothesis template before launch.

Metrics and guardrails

The primary metric is install conversion rate, first-time installs per product page view, for the priority traffic source. Report the other sources too, each on its own, but decide on the one named in advance. The conversion rate calculator page for app store listings explains which console figures make up the CVR and why page views must come from one source at a time; use its definitions, unchanged, for both arms.

The guardrails catch wins that cost something later. Early retention of installers, read by install Cohort in the app's analytics or the mobile measurement partner, shows whether the new screenshots attract people who stay; a listing can win installs by promising more than the first session delivers. The ratings and reviews trend during and after the test shows whether new installers are disappointed. CPI for campaigns landing on the listing shows whether the change helps or hurts paid acquisition, since every paid install passes through the page.

Design

Use the store's native listing experiment. Apple's product page optimisation runs treatments of the icon, screenshots and previews against the original and reports conversion for each; Google Play's store listing experiments do the same for the icon, graphics and descriptions, with options for localised tests. The stores assign visitors between the current listing and the variant, so the unit of randomisation is the store visitor, assigned by the experiment. Change one asset set only, and keep the title, description, ratings display and everything else the same.

Design elementChoiceReason
Unit of randomisationStore visitor, assigned by the store's listing experimentThe store controls the split on its own page
ArmsCurrent asset set versus one new asset setOne hypothesis per test
Changed assetScreenshots or icon, not both unless the set is the decisionKeeps the result interpretable
Traffic sourcesSearch, browse and paid read separatelySources convert differently; blended rates move with the mix
Priority sourceNamed before launchPrevents choosing the winning source afterwards
Retention checkInstalls from the test joined to cohort reportsCatches listings that over-promise

Read search, browse and paid traffic separately wherever the store's reporting allows, using the console's source breakdown for the test period alongside the experiment's own result. Campaigns that land on a custom product page or a custom store listing send their traffic to a different page from the default listing under test; the app install campaigns guide explains how those pages are matched to ads, and the guardrail should be read only for campaigns that land on the default listing.

Sample size reasoning

Store traffic sources convert very differently, so size the test for the source that matters most rather than for blended views. A source with high conversion and plenty of page views can detect a modest change quickly; a thin source may never reach a reliable read, which is a reason to name a different priority source, not to blend them. The conversion rate calculator page for app store listings shows how the uplift input sizes a listing experiment, and the sample size and significance guide explains how the baseline rate, the minimum effect and the confidence level combine into the page views each arm needs. Decide the smallest lift that would justify producing and maintaining new assets before launch.

Duration

Run for at least two full weekly cycles in the store experiment, and longer for low-traffic listings, so that weekday and weekend audiences are both represented. Avoid launching during a store feature, a launch spike or a large campaign burst, any of which would change the mix of visitors for part of the test. Then wait for the early-retention window to close for installs from the last days of the test before making the final decision.

Reading the results and the decision rule

Read the store's result for the priority source at the planned end, with its confidence range, then the other sources, then the guardrails. The rule, written before launch:

ResultDecision
Install conversion rises for the priority source and early retention holdsApply the new assets
Conversion rises only for sources whose installs retain poorlyKeep the current set
Conversion rises but early retention or ratings fallKeep the current set and revise the screenshots to match the first session
No reliable difference for the priority sourceKeep the current set, or test a bolder change

The app installs through app stores page explains why listing work comes before store ads in an installs plan, which is where a winning listing pays off across every channel.

Illustrative example — not a real result

A running app tests outcome-led screenshots

Pacecraft, an invented running-plan app, replaces its first three screenshots, which showed menus and settings, with images of a finished plan and a race-day result. The team names store search as the priority source and runs the store experiment for three weeks. In store search, each arm receives about 25,000 product page views; install conversion is 30.0% for the current set and 32.4% for the new one, with a 95% interval for the difference of roughly 1.6 to 3.2 points. Browse traffic shows no reliable difference. Seven-day retention of installers from the test period is level, and ratings do not move. Pacecraft applies the new screenshots.

Pitfalls

Blended traffic sources are the first pitfall. A week with more browse traffic lowers the overall rate without any change in the listing, and the reverse flatters it; reading one source at a time removes that noise. Featuring spikes in one period are the second. If the app is featured partway through the test, the visitor mix changes for both arms; the split protects the comparison, but the result then describes a different audience, so extend the test or rerun it. Installs that do not retain are the third: the store reports the install, not what follows, so the retention join is not optional.

Watch for seasonal relevance as well. Screenshots tied to a season or an event can win during it and lose after; run the test in an ordinary period unless the change is meant to be seasonal.

When a winning set is applied, run the app store listing checklist on the updated listing before it is submitted, so the new screenshots go out with accurate text and privacy details.

Common mistakes

Listing tests are easy to run and easy to misread. The push permission prompt test is the matching design for the first minutes after install, and the experiment library lists every design.

  • Reading blended page views. Decide on the priority source named before launch.
  • Changing several assets at once. Then nothing says which change mattered.
  • Ignoring retention. A listing can win installs by over-promising.
  • Stopping after a few good days. Run full weekly cycles and avoid featuring spikes.
  • Judging paid traffic on custom pages. Those visitors never saw the listing under test.
  • Testing on taste instead of a hypothesis. State what the new assets should change and for whom.

Frequently asked questions

Why read the result by traffic source?

Because people who searched for the app or a close keyword convert very differently from people browsing charts or arriving from a campaign. A blended rate moves with the traffic mix as much as with the listing. Choosing the priority source before launch, and reading the test on it, keeps the decision tied to the audience the new screenshots were made for.

Can we test the screenshots and the icon together?

You can, but the result will not say which change did the work. If the decision is to replace the whole visual set, test it as one variant. If you want to know whether the icon or the screenshots matter, test them in separate experiments, one after the other.

Why check retention if the store experiment only reports installs?

Because a listing can win installs by over-promising. Screenshots that show an outcome the app does not deliver quickly bring in people who leave in the first days. Join installs from the test period to the app's own cohort reporting and check that early retention held before rolling out.

Terms used on this page

Tools to try

Frameworks

Where this page is used

Also related

Sources

  1. Play Console Help — Google
  2. App Store — Apple Developer

Related