Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Experiment Library

Web-to-App Switch Test

An experiment design for app prompts on mobile web pages: visitors split between a prompt and none, read on orders across web and app together.
Experimentadvanced10 min readUpdated

Hypothesis

If mobile web visitors on pages with an app equivalent see a small, dismissible prompt to continue in the app, orders per visitor across web and app together will rise, because people who switch use the service more, unless the prompt only moves orders into the app or interrupts web orders that are then lost. The decision behind it: show a prompt to continue in the app on mobile web pages that have an app equivalent, or keep visitors on the web.

The guide to web-to-app journeys plans the journey: who is worth moving, where prompts help and where they harm the visit, links to the matching screen, sign-in that survives the switch, and three measures of a route. Its measures are the switch rate, the interrupted visit and behaviour after the switch, and it suggests running the prompt on part of the traffic to read the second of them. This design turns that sentence into a plan, and adds the step the three measures leave apart: one outcome, counted across both surfaces, for everyone who could have seen the prompt. If the business has not yet decided whether it needs an app at all, the guide to choosing between an app and the mobile website comes first.

The growth loops framework applied to app installs explains why the link between web and app is the joint that most often breaks. A prompt that moves people but loses them on the way, or moves people who were already going to order, looks like a working loop in an install report. Write the definition of the eligible pages, the prompt's wording and placement, and the limit on each guardrail into the plan before launch.

Metrics and guardrails

The primary metric is orders per mobile web visitor across web and app combined. Where the business's main goal is not an order, use the main goal action instead: a booking, a subscription renewal, a saved list used. Count every order placed by visitors in an arm during the reading window, whether it was placed on the website or in the app, and divide by the visitors assigned to that arm. The join between the two surfaces is the shared user identifier set at sign-in on both, which the deep links and app analytics guide and the web-to-app guide describe. Without it, an order that moves from the web into the app looks like a lost web order in one report and a new app order in another, and the test cannot tell moving from adding.

Read the switch rate and the app opens by arm as diagnostics. They explain a result; they do not decide it. Nor does the CPI of paid campaigns, which this test does not touch: an install from the website costs no media, but it can cost the web order it interrupted.

Three guardrails protect the visit the prompt sits on:

  • Web conversion rate on the pages that carry the prompt. The web-to-app guide calls this the interrupted visit, the measure teams skip. Read the web CVR on prompted pages in the treatment arm against the same pages in the control arm; the conversion rate calculator gives the rate for each arm from its visits and conversions.
  • Repeat showings to visitors who dismissed the prompt. The guide's rule is that a dismissal is respected. Count the visitors who saw the prompt again after closing it; the target is none, and any count above that is a defect in the build.
  • App uninstalls among visitors who switched. A switch that ends in an uninstall soon after has moved nobody. Read uninstalls among the treatment arm's switchers wherever the app's own analytics can observe them, as a sign that the first app session did not deliver what the prompt promised.

Design

Randomise at visitor level, and hold each visitor in one arm across sessions. Signed-in visitors are preferred, because their arm can be stored on the account and their activity joined across web and app. Visitors who are not signed in can be assigned on a first-party identifier, but their app activity usually cannot be joined, so analyse them separately or leave them out of the primary reading. Either way, decide before launch.

Design elementChoiceReason
Unit of randomisationMobile web visitor, held in one arm across sessions; signed-in visitors preferredLets web and app activity be joined to one person and one arm
Eligible pagesOnly pages with an app equivalentThe web-to-app guide's rule: the switch continues what the person was doing
Treatment armA small, dismissible prompt that opens the matching app screen; a dismissal is rememberedThe placement the guide recommends, built to its rules
Control armThe same pages with no promptMeasures what the prompt adds over doing nothing
Held constantPages, offers, prices, the app itself, emails and every other route into the appOnly the prompt differs
Primary metricOrders per visitor across web and app combinedSeparates adding orders from moving them

Build the prompt as the web-to-app guide sets out. It links each page to the matching app screen, and for visitors without the app a deferred deep link carries the destination through the store, which the deep links guide notes depends on a link service or measurement partner. Test the routes in the system browser and in social apps' in-app browsers before launch, so the treatment arm is not quietly a broken link.

The placement is fixed to one that leaves the content readable. Google's page experience guidance asks whether pages avoid intrusive interstitials; as the page experience guide and the web-to-app guide explain it, an overlay that covers the main content on arrival works against that, while a small banner is a different matter. This design claims nothing more specific about how Google treats any particular prompt. It keeps the placement small because the decision rule would not keep anything else.

The exit intent popup test is the nearest design on the web side, and it weighs an overlay on web revenue alone; here the revenue is split across two surfaces, which is why the join matters. Inside the app, the push permission prompt test covers the next prompt a switched visitor is likely to meet, and its timing should not change during this run.

Sample size reasoning

The metric counts orders on both surfaces, so only visitors whose web and app activity can be joined give a clean reading. That usually means signed-in visitors, and a sample much smaller than the page traffic suggests. The inputs are the joined visitors' current orders per visitor, how widely orders vary between visitors, the number of eligible joined visitors per week and the smallest change worth showing the prompt for. The sample size and significance guide shows how they combine, and why an orders-per-visitor metric, with many visitors ordering nothing and a few ordering often, needs more visitors than a simple rate.

If the joined sample cannot see a change worth acting on, narrow or reframe the test before launch: restrict it to the pages where switching matters most, such as order and account pages, or read the web conversion rate on prompted pages as the primary metric, which is the interrupted-visit question on its own. Label that second option as an answer to a smaller question.

Duration

Run whole weekly cycles, long enough for switched visitors to place the repeat orders the app is meant to win. The first order after a switch says little; the case for the app rests on later ones, so the reading window follows each visitor for the same period from assignment, read as a Cohort by week of entry. Keep app releases, store listing changes and app install campaigns out of the run, or, if one cannot be avoided, make sure it reaches both arms equally and note it.

Reading the results and the decision rule

Read once, at the end of the reading window. Compare orders per visitor across both surfaces between arms, with the interval around the difference. Then check the guardrails: web conversion on prompted pages, repeat showings after dismissal and uninstalls among switchers. Break the result down by page type only if that split was named before launch. Write the rule into the plan:

ResultDecision
Orders per visitor across both surfaces rise reliably and web conversion on prompted pages holds within its limitKeep the prompt on the eligible pages
Orders per visitor do not change, but app orders rise and web orders fall by a similar amountRemove the prompt: it moves orders between surfaces without adding any
Web conversion on prompted pages breaks its limitRemove the prompt, whatever the combined result, and revisit placement
Any placement the page experience guidance treats as intrusiveNever keep it, whatever it wins

Pitfalls

Orders counted on the web only make every prompt look harmful, because any order that moves into the app vanishes from the web report. Orders counted in the app only make every prompt look helpful. The join is the design; without it, the test answers neither question.

Loyal customers switching anyway is the second trap. Comparing switchers with non-switchers inside the treatment arm credits the app with the habits of people who already ordered often. Compare arms, not people within an arm.

Prompts repeated after dismissal turn a small banner into a nag and break the guide's rule. Store the dismissal on the account for signed-in visitors as well as in the browser, and audit the guardrail during the run.

Links that drop people on the app's home screen throw away the reason to switch and make the prompt look weak. Map every eligible page type to its screen before launch and test each route, including in in-app browsers.

Example: an invented grocery delivery service tests its app banner (illustrative)

Illustrative example — not a real result

Lirkadew Groceries tests an open-in-app banner

Lirkadew Groceries, an invented grocery delivery service, has an app and a mobile website, and most repeat orders already come through the app. It shows a small open-in-app banner on product and order pages to half of its signed-in mobile web visitors, remembers every dismissal, and links each page to the matching app screen. About 24,000 signed-in visitors enter over four weeks, and each is followed for six weeks from assignment.

ArmVisitorsWeb ordersApp ordersOrders per visitor, both surfacesWeb conversion on prompted pages
No banner12,0405,1304,2100.786.1%
Banner11,9604,5205,0000.805.8%

Orders per visitor across both surfaces are 0.02 higher in the banner arm, with an interval of roughly minus 0.01 to plus 0.05, so most of the banner's effect is orders moving from web to app. Web conversion on prompted pages fell by 0.3 points, inside the limit of half a point the team set before launch. No dismissed visitor saw the banner again. Lirkadew keeps the banner on order pages, where switchers placed most of their extra repeat orders, and removes it from product pages, as its plan allowed for a page type named before launch.

Common mistakes

Web-to-app tests fail on what they count more often than on how they split. The experiment library holds the other prompt and overlay designs.

  • Reading installs as the result. Installs say a prompt was tapped, not that the business gained an order.
  • Counting one surface. Orders must be joined across web and app, or the test measures where orders happen, not how many there are.
  • Including visitors who cannot be joined. Unjoined visitors dilute the primary metric; analyse them apart or leave them out.
  • Changing the app during the run. A release or store change lands on every switcher and confuses the reading.
  • Testing an overlay that cannot be kept. Spend the traffic on placements the decision rule allows.
  • Ignoring dismissals. A prompt that returns after being closed is a defect, not a variant.

Frequently asked questions

Why not judge the prompt on installs or app opens?

Because a switch is only worth something if the person does more with the business afterwards. A prompt can produce many installs while moving orders from the web into the app, or while interrupting web orders that never happen. Counting orders per visitor across both surfaces shows whether the prompt added anything, and the switch rate remains useful as a diagnostic beside it.

Why compare at random instead of comparing people who switched with people who stayed?

The people most likely to switch are the most loyal customers, who would have ordered more anyway. A comparison of switchers with non-switchers credits the app with their habits. Assigning visitors at random, and comparing everyone who could see the prompt with everyone who could not, removes that selection, because both arms contain the same mix of loyal and occasional customers.

Can the test try a full-screen overlay as one of the arms?

This design does not include one. Google's page experience guidance asks whether pages avoid intrusive interstitials, and the site's page experience guide explains that an overlay covering the content on arrival works against that. A placement the guidance treats as intrusive is not kept whatever it wins, so testing it spends traffic on an option the decision rule has already excluded.

Terms used on this page

Tools to try

Frameworks

Also related

Sources

  1. Understanding page experience in Google Search results — Google Search Central

Related