Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Experiment Library

Push Permission Prompt Test

A test plan for when and how an app asks for push permission, read on the share of new users who opt in, with early retention and uninstall guardrails.
Experimentintermediate7 min readUpdated

Hypothesis

If the app explains the value of notifications on its own screen and asks for permission after the user's first meaningful action instead of at first launch, more new users will grant permission, because the request arrives with a reason at a moment of engagement. The decision behind it: ask for push permission at first launch, or after an explanation and the user's first meaningful action.

The hypothesis follows the practice described in the push notifications and in-app messaging guide: the system prompt is a one-time asset, so it should be spent when its value is obvious. The test exists because practice is not proof for a particular app. A delayed prompt has a cost the first-launch prompt does not: some users leave before they ever reach the meaningful action, and they are never asked at all. Whether the higher acceptance among those who are asked outweighs the people never asked is an empirical question. Name the meaningful action, such as a first saved item, a first order or a first booking, and the wording of the explanation screen in the experiment hypothesis template before launch.

Metrics and guardrails

The primary metric is the push opt-in rate among new installs: the share of all installs assigned to each arm that have granted notification permission by the end of the early window. The denominator is every assigned install, whether or not the prompt was shown. On platforms where notification permission is granted by default for some users, read opt-in as the share with notifications enabled, and record that difference by platform.

Three guardrails keep the opt-in figure honest. Early retention, read by install Cohort over an agreed early window, is the guardrail that matters most: a prompt that wins permissions but irritates new users could raise churn and lose more than it gains. Onboarding completion rate shows whether an extra explanation screen interrupts the flow. Uninstall rate, where the analytics stack or the mobile measurement partner reports it, catches people who leave outright. The deep links and app analytics guide covers the event taxonomy and cohort reporting these metrics depend on; make sure the permission result, the meaningful action and the arm are all logged as events before launch.

Design

Randomise at the level of the new install, at first launch, using the app's experiment tooling, such as the A/B testing that Firebase and most engagement platforms support. Assign before any onboarding screen so both arms start identically. The control shows the system prompt at first launch, as today. The challenger shows nothing at launch; after the user completes the meaningful action, it shows the app's explanation screen, and only if the user agrees there does it trigger the system prompt.

Design elementChoiceReason
Unit of randomisationNew install, assigned at first launchPermission is asked once, at the start of a user's life
ArmsFirst-launch system prompt versus explanation then prompt after the meaningful actionTests timing and framing together, as one decision
System promptUnchanged in both armsIt cannot be redesigned; only its trigger moves
DenominatorAll installs assigned to the armCounts users who never reach the delayed prompt
Platform splitRead iOS and Android separately, then pooledPermission models differ between operating systems
Messages sentSame push programme for anyone who opts inRetention differences then come from the prompt, not the content

The explanation screen has to be honest about what will be sent. Apple's App Store Review Guidelines say push notifications must not be required for an app to function and must not be used for marketing unless the person has opted in to receive it, so if the app intends to send promotions, the screen should say so. A user who declines the explanation screen has not used up the system prompt and can be asked again later; record those later asks, but read the test on the early window only. Where the meaningful action sits in the first session, and which other permissions compete with this one, is set out in the guide to app onboarding and permission requests.

Sample size reasoning

Opt-in is a frequent event, so the sample needed to detect a meaningful difference in the primary metric is modest. Early retention is the guardrail that matters, and it needs more users to read, because the differences that would matter there are smaller relative to their noise. Size the test for the harder of the two: decide the smallest fall in early retention the team would refuse to accept, and make sure the test can detect it. The sample size and significance guide shows how a baseline rate, a minimum effect and a confidence level set the users each arm needs.

Duration

Enrol until the planned sample is reached, across full weekly cycles, since installs from weekend campaigns and weekday searches can behave differently. Then follow the last cohort through the early-retention window before reading. Avoid launching alongside a major release, a store feature or a large paid campaign that would change the mix of installs during enrolment.

Reading the results and the decision rule

Read once, after the last cohort's early window closes. Compare opt-in among all installs, then early retention, onboarding completion and uninstalls, first pooled and then by platform. The rule, written before launch:

ResultDecision
Delayed, explained prompt raises opt-in and early retention does not fallAdopt the delayed, explained prompt
Opt-in rises but early retention or onboarding completion fallsKeep the first-launch prompt, or move the ask to an earlier action and retest
Opt-in falls because many users never reach the delayed promptKeep the first-launch prompt, or choose an earlier meaningful action
No reliable differenceKeep the explained prompt only if it is simpler to maintain; otherwise keep the current flow

Once the test has finished, a significance reader for two conversion rates confirms the primary read. Enter the installs in each arm as visitors and the opted-in users as conversions. It reads a completed test; it does not size one.

Push opt-in significance checker

Illustrative example — not a real result

A grocery list app delays its permission ask

Sproutlist, an invented grocery list app, assigns 40,000 new installs evenly between its first-launch prompt and a flow that waits until the user saves a first shopping list, explains that notifications will remind them when they are near a shop on their list, then asks. Among all installs, opt-in is 38.0% in the control arm and 44.5% in the challenger, with a 95% interval for the difference of roughly 5.5 to 7.5 points. Seventy-two per cent of challenger installs reach a first saved list. Seven-day retention is 31.0% against 31.2%, with an interval that includes zero, and uninstalls are level. Sproutlist adopts the delayed, explained prompt.

Pitfalls

Users who never reach the later prompt are the main pitfall. If the meaningful action is placed too deep in the product, the challenger asks fewer people and can lose even when it persuades better. Counting all installs in the denominator makes the cost visible; choosing an early, common action reduces it. Users who arrive via a Deep Link may skip the onboarding route to the meaningful action entirely, so check that the challenger's trigger fires on every path into the app.

Operating system differences are the second pitfall. iOS lets an app show the system prompt only once, and recent Android versions use an explicit runtime permission for notifications, while older Android versions do not ask at all. Pooling those groups without reading them separately can hide an effect on one platform behind a default on the other. Reading opt-in without retention is the third: a higher permission rate is worth nothing if the users who granted it leave sooner. The SMS vs push notifications guide explains why the permission determines how large the push audience can become, which is why the decision is worth testing carefully.

Common mistakes

Permission tests usually go wrong in the denominator or the platform split. The onboarding email cadence test is the matching design for the email side of a new user's first days, and the experiment library lists every design.

  • Dividing by users who saw the prompt. Divide by all installs assigned to the arm.
  • Changing the push programme between arms. Opted-in users must receive the same messages.
  • Pooling iOS and Android without a split. Permission works differently on each.
  • Choosing a meaningful action few users reach. The delayed prompt then asks too few people.
  • Promising one thing and sending another. The explanation screen must describe what will be sent.
  • Skipping the retention read. Opt-in alone cannot decide the test.

Frequently asked questions

Why measure opt-in among all new installs rather than among people who saw the prompt?

Because the delayed prompt is only shown to people who reach the meaningful action. Measured among those who saw it, it would look excellent while ignoring everyone who left before getting there. Dividing by all new installs in each arm counts those people as not opted in, which is what they are.

Does the test change the operating system's permission dialog?

No. The system dialog cannot be redesigned and, on iOS, can be shown only once. The test changes when the app triggers it and whether an explanation screen comes first. That explanation screen, often called a soft prompt, is the app's own and can be designed and tested freely.

What should the explanation screen promise?

Only what the app will actually send. Apple's App Store Review Guidelines say notifications must not be required for an app to work and must not be used for marketing unless the person has opted in to that, so if promotional messages are planned, the explanation should say so honestly.

Terms used on this page

Tools to try

Frameworks

Where this page is used

Also related

Sources

  1. Firebase documentation — Google
  2. App Store Review Guidelines — Apple Developer

Related