Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Experiment Library

Deferred Account Creation Test

An experiment design for moving account creation from the first screen to the moment a new user commits, read on activation per install cohort.
Experimentintermediate9 min readUpdated

Hypothesis

If new installs can browse and choose before creating an account, and are asked for one only when they save, book or buy, more of each install cohort will reach the activation moment within the agreed window, because the account form stops standing between people and the value they came for, unless fewer accounts are created and early retention falls. The decision behind it: ask new app users for an account at first open, or only when they have something to save, book or buy.

The guide to app onboarding and permission requests lists account creation among the steps usually deferred, because browsing and choosing rarely need an account while committing does. It also warns that a change that delays the account may raise activation and lower sign-ups that matter later, and asks for changes to be run as randomised tests rather than before-and-after comparisons. Its own illustrative rebuild moves several screens at once, which is the right call for a redesign and the wrong one for learning what each screen costs. This design moves the account alone. The sign-up flows guide makes the same argument for web sign-up; this is its app counterpart, where the account form arrives before anything useful has happened.

Metrics and guardrails

The primary metric is the share of each install cohort that reaches the activation moment within the agreed window. Define the moment before launch from evidence, not convenience: the action after which new users are markedly more likely to stay. The tutorial on finding the app activation event does that analysis from the app's own data, and the AARRR framework warns that an activation event chosen because it is easy to count, or because it flatters the funnel, defeats the purpose. The resulting Activation Rate is a conversion rate (CVR) through the first session, counted against every install in the arm's cohort, not only those who saw a particular screen.

Three guardrails keep the result honest:

  • Share of the cohort with an account by the end of the window. Accounts carry the email address, the saved state and the ability to reach people later. Some fall is expected when the form moves; the limit on how much is agreed in advance, from what the business does with accounts.
  • Early retention at the return points agreed before launch. A deferred account that raises activation but loses people who would have returned through a reminder has traded the second session for the first.
  • Permission opt-in counted across the whole cohort. The permission screen does not move, but the people who reach it may differ between arms. Count the Push Opt-In Rate against every install, so an arm that sends fewer people to the prompt cannot look better by asking fewer.

Design

This is an A/B test on new installs. Assign each install at random at first open, store the arm with the install's identifier, and keep it there for the rest of the test, including after the person creates an account. Read results by weekly install Cohort, with iOS and Android reported separately.

Design elementChoiceReason
Unit of randomisationNew install, assigned at first open and kept in its armThe account screen acts on the first session; reinstalls and existing users are excluded
ArmsAccount at the first screen; account at the first save, booking or purchaseThe only decision is when the form appears
Held constantPermission screens, tour, content, offers, onboarding messages and the account form itselfOnly the account step moves, so the result can be attributed to it
PlatformsiOS and Android read separately where each has enough installsTheir permission models and first-session behaviour differ
Primary metricShare of each install cohort reaching activation within the agreed windowMeasures what the first session is for, not the form's completion
ReadoutWeekly cohorts, each followed to the end of the window and the retention pointsCohorts read before they close understate both arms unevenly

Two conditions apply in both arms. The account form is the same form, asking for the same details, so the test measures its timing rather than its length. And the permission screen stays where it is. The onboarding guide records, from Apple's App Store Review Guidelines, that push notifications must not be used for marketing unless the person has opted in. Separately, on iOS the system prompt is a single chance per install, which is platform behaviour rather than a guideline. Between them that is reason enough not to move it at the same time. The push permission prompt test is the design for its timing.

The account step has a rule of its own. Apple's App Store Review Guidelines on account sign-in (5.1.1(v)) say that an app without significant account-based features should let people use it without a login. For some apps, then, the deferred arm is the expected design as well as a candidate for testing, so check the guideline's current wording against what the app actually does before the test is planned.

Events matter as much as the arms. Log each screen view, the account prompt, its completion and the activation event, following the event taxonomy in the deep links and app analytics guide, so the drop between any two steps can be seen. People who arrive through a deferred deep link should land on the item they tapped in both arms; only the moment of the account prompt differs.

Sample size reasoning

Activation is a rate per install, so the inputs are the current activation rate, weekly installs on each platform and the smallest change in activation worth rebuilding the first session for. Read the platforms separately only if each has enough installs on its own; otherwise read them together and treat the split as a secondary view. The sample size and significance guide explains how the inputs combine, and the conversion rate calculator turns each cohort's counts into the rates the guide works with, so the team can see how many weekly cohorts the test needs before it starts. Paid install pushes inflate weekly installs but may change who installs; count the cohorts the app usually sees, not a launch week.

Illustrative example — not a real result

How effect size sets the number of weekly cohorts

Take a 31% activation rate and the usual 5% significance level with 80% power. Detecting a rise to 38% needs roughly 720 installs per arm; detecting a rise to 34% needs roughly 3,800. At 1,500 installs per arm each week, the smaller effect takes three weekly cohorts, and reading iOS and Android separately needs that count on each platform. The smaller the change worth rebuilding for, the longer the test runs.

Duration

Run enough weekly install cohorts to reach the sample, and follow each to the end of the activation window and the agreed early-retention points before reading it. The last cohort's window decides when the reading can happen, not the day enrolment stops. Freeze the conditions for the whole period: no app release, store listing change or paid install push that reaches one arm only, because any of them changes who installs or what they see. If a release cannot wait, ship it to both arms on the same day and note the date in the plan.

Reading the results and the decision rule

Read each cohort once it has closed, and decide once all the planned cohorts have closed. Compare activation between arms on each platform with the interval around the difference, then read the guardrails: the share with an account, early retention and opt-in across the cohort. Write the rule before launch:

ResultDecision
Activation rises reliably and the account share and early retention hold within their limitsShip the deferred account
Activation does not move reliablyKeep the account first
Activation rises but the account share or retention falls past its limitDefer only the steps that do not feed retention, such as profile details, and keep the account where reminders depend on it
One platform improves and the other does notShip on the improving platform and investigate the other before retesting

Remember what the test cannot say. It measures when the account is asked for, not whether the form is the right length or whether the app's value is clear. Once accounts arrive later, the onboarding email cadence test is the design for what the app sends to the new account afterwards.

Pitfalls

A cohort read before its window closes is the first trap. The deferred arm creates accounts later, so a cohort read early shows it short of accounts and, sometimes, short of activation too. Hold every reading to the planned window.

An activation event chosen because it flatters is the second. If the event is something the deferred arm makes trivially easy, such as viewing an item, the test is won before it starts. Fix the event from the retention evidence before anyone sees results.

Accounts lost that later re-engagement needs are the third. An install without an account cannot receive the email that brings it back, and some apps rely on that heavily. If the business needs accounts for re-engagement, set the account-share limit with that team rather than alone.

Releases landing mid-test are the fourth. A release that changes the first session reaches both arms but may interact with one; a store listing change alters who installs. Log every change and its date, and pause enrolment if one lands by surprise.

Example: an invented plant-care app tests asking for an account later (illustrative)

Illustrative example — not a real result

Ombraquet Plants tests asking for an account after the first watering plan

Ombraquet Plants, an invented plant-care app, has always opened on an account form. Its activation moment, chosen from its own retention data, is setting a watering plan and confirming the first watering within seven days. In the deferred arm, a new user can identify a plant and set a watering plan first, and is asked for an account only when they save their plant collection. The notification screen stays where it was. Over six weekly cohorts, 18,000 new installs are split evenly. Before launch the team agrees that the account share may fall by no more than ten points and that day-30 retention may not fall at all.

ArmInstallsActivated in 7 daysAccount by day 7Day-30 retentionNotification opt-in
Account first9,00031%58%18%34%
Deferred account9,00038%49%19%35%

Activation rises by seven points, with an interval of roughly 5.6 to 8.4 points, and the gain appears on both iOS and Android when read separately. The account share falls by nine points, inside the limit, and retention holds. Ombraquet ships the deferred account and keeps watching the account share, because its reminder emails depend on it.

Common mistakes

Account timing tests usually fail on their scope or their window. The experiment library lists the other app and onboarding designs that sit beside this one.

  • Moving several screens at once. A redesign can be right, but it cannot tell you what the account step cost.
  • Judging on accounts created. The first session exists to deliver value; accounts are a guardrail.
  • Counting opt-in only among people asked. Count against every install in the cohort.
  • Reading cohorts before they close. The deferred arm creates accounts later by design.
  • Pooling iOS and Android by default. Read them separately where each can stand on its own.
  • Shortening the form in the deferred arm. Change the timing or the form, not both in one test.

Frequently asked questions

Why is activation the primary metric rather than accounts created?

Because the account is a means, not the goal. An account form at first open can produce plenty of accounts belonging to people who never reach the value they came for, while a deferred account produces fewer, later accounts from people who already have something worth saving. Activation, defined beforehand as the moment after which people are markedly more likely to stay, measures what the first session is for. Accounts remain a guardrail, because the business needs them for messages and re-engagement later.

Should the notification permission move along with the account?

Not in this test. The permission screen stays exactly where it is in both arms, so the only thing that differs is when the account is asked for. Moving both at once would leave the team unable to tell which change produced the result. Permission timing has its own design in the library, and it can run once the account question is settled.

Can we read iOS and Android together?

Read them separately where each platform has enough installs, because their permission models and first-open behaviour differ, and a change that helps on one can be neutral on the other. If one platform is too small to read alone, report it as a secondary view and decide on the larger platform, noting that the smaller one was not tested to the same standard.

Terms used on this page

Tools to try

Frameworks

Sources

  1. App Store Review Guidelines — Apple Developer

Related