Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Experiment Library

Exit-Intent Popup Test

A design for testing an exit-intent popup on revenue per visitor, so captured emails are weighed against interrupted purchases and annoyance.
Experimentintermediate7 min readUpdated

Hypothesis

If an email capture popup appears when visitors show intent to leave, more first-time visitors will join the list without reducing purchases in the session, so revenue per visitor including later email revenue will rise. The decision is whether to run an email capture popup on the site, and if so with which trigger, or to rely on in-page sign-up.

The hypothesis carries two claims, and the test has to check both. The first is that the popup grows the list, which it almost certainly will. The second is that it does so without costing purchases, which is far less certain: an overlay that appears at the wrong moment interrupts someone who was about to buy, and usability research such as the articles published by Nielsen Norman Group has long described how intrusive overlays frustrate people trying to complete a task. The list building and consent guide sets out where sign-up placements work and what to watch for with overlays; this test measures the trade on your site. Record the trigger, the offer, the pages in scope and the follow-up window in the experiment hypothesis template.

Metrics and guardrails

The primary metric is Revenue per Visitor: revenue from each arm's assigned visitors during the collection period, plus revenue from the subscribers each arm produced, popup and in-page alike, over a fixed follow-up window, divided by the number of assigned visitors. Including later email revenue is what lets the popup's benefit appear at all, since most sign-ups do not buy on the day; fixing the window is what stops the popup arm accumulating credit indefinitely.

Three guardrails catch the costs. Session conversion rate, the CVR of visits to purchases in the visit itself, shows whether the popup interrupts buyers; a fall here is the cost the hypothesis says will not happen. Return-visit rate shows whether visitors who saw the popup come back less, which is the slower form of annoyance. The unsubscribe rate of popup sign-ups, read over the follow-up window, shows whether the people the popup captured wanted the emails; a placement that fills the list with people who leave in the first fortnight adds to List Growth Rate on paper and to complaints in practice.

Design

Randomise at the level of the visitor, assign on the first visit and keep the visitor in the same arm on return. The control shows the site as it is, with the in-page sign-up form. The variant shows the same site, same in-page form, plus a popup triggered by exit intent, carrying the same offer, the same form fields and the same consent wording as the in-page form. Each visitor sees the popup at most once in a set period.

Design elementChoiceReason
Unit of randomisationVisitor, assigned on the first visit and kept in one armReturn visits and later purchases belong to the same arm
ArmsIn-page sign-up only, against in-page sign-up plus exit-intent popupIsolates the popup
Offer, form and consent wordingIdentical in popup and in-page formTests the format, not the offer
Frequency capPopup shown at most once per visitor in a set periodLimits annoyance and keeps the treatment stable
MobileA separate trigger and display rule written before launchExit intent and overlays behave differently on phones
Primary metricRevenue per visitor including email revenue over a fixed windowWeighs captured emails against lost sales

Where the popup sits relative to the page's own call to action matters. The anatomy of a landing page guide explains how each page should have one job; a popup that fires on a checkout or lead form page competes with that job directly, so exclude those pages from the trigger. Whether the popup uses single or Double Opt-In should match the in-page form, so that the two arms produce comparable subscribers.

Sample size reasoning

Revenue per visitor is skewed and the email revenue arrives later, so the test needs a larger sample and a fixed follow-up window. Most visitors spend nothing, a few spend a lot, and the email revenue from sign-ups adds a second, delayed source of variation. The sample size and significance guide covers skewed metrics and why they need more visitors than a conversion rate to detect the same relative change. The conversion rate calculator gives the baseline session conversion and sign-up rates the plan starts from. If the site's traffic cannot carry a revenue test, read session conversion as the primary metric with a strict non-inferiority margin, and say plainly in the plan that the email value is estimated rather than measured.

Duration

Run at least two full weekly cycles to collect visitors, then a fixed window for sign-ups to receive and act on emails. The collection period ends on a planned date regardless of how the numbers look; the follow-up window then runs for every subscriber, including those who joined on the last day. Keep the welcome series identical for sign-ups from both arms, as the welcome series tutorial describes, so that email revenue differences come from who signed up, not what they received.

Reading the results and the decision rule

Read once the follow-up window has closed. Compare revenue per visitor between arms with its interval, then read session conversion, return visits and the unsubscribe rate of popup sign-ups. Read the device split named before launch last. The A/B testing guide covers the reading discipline in general. The rule, written before launch:

ResultDecision
Revenue per visitor rises and session conversion holdsKeep the popup with this trigger
Revenue per visitor rises but session conversion falls on some pagesKeep the popup, excluded from those pages, and re-test
Sign-ups rise but popup sign-ups unsubscribe quicklyDrop the popup or change the trigger; the list is growing with the wrong people
Revenue per visitor falls or session conversion falls overallDrop the popup and rely on in-page sign-up

Pitfalls

Reading sign-ups alone is the defining pitfall. A popup produces more sign-ups almost by definition, and a test that stops there will always recommend it, whatever it costs in interrupted purchases and lost goodwill. Revenue per visitor over a fixed window is the only reading that weighs both sides.

Mobile interstitials harming page experience are the second. Overlays that cover the content on a phone are far more disruptive than a box on a desktop screen, and Google Search Central's page experience documentation describes intrusive interstitials as part of what makes a page harder to use. Write the mobile rule into the plan, prefer a smaller format on phones, and read the device split you named in advance. The third is consent wording differing between arms. If the popup's wording is vaguer or its checkbox behaves differently from the in-page form, the arms produce subscribers with different consent, and the test compares lists that are not equally usable. The ICO's direct marketing guidance explains the UK's rules on consent for marketing email; keep one approved wording in both places.

Illustrative example — not a real result

A coffee roaster tests an exit-intent popup

Quarrybrook Coffee Roasters, an invented online coffee roaster, has an in-page sign-up in its footer. It tests an exit-intent popup with the same offer, a free brewing guide, over three weeks with 40,000 visitors per arm, then follows every sign-up for 30 days. The popup arm signs up 1,320 visitors against 520 in control. Session conversion is 2.30% in control and 2.24% in the popup arm, inside the noise. Revenue per visitor including the 30-day email revenue is £1.31 in the popup arm and £1.24 in control, with an interval of roughly £0.01 to £0.13. Popup sign-ups unsubscribe at a higher rate in the first fortnight, but within the team's limit. On phones, session conversion falls in the popup arm, so Quarrybrook keeps the popup on desktop only and re-tests a smaller mobile format.

Interstitials are one of several page experience questions Google lists, and the guide to page experience beyond Core Web Vitals shows how to review a template for overlays, ads and mobile display together.

Common mistakes

Popup tests fail on what they count and when they stop counting. The social proof placement test is a quieter way to change a page's conversion, and the experiment library lists every design.

  • Counting sign-ups as the result. The popup wins on sign-ups by construction.
  • Using a better offer in the popup. The test then measures the offer, not the format.
  • No follow-up window. Email revenue never arrives in the reading, or never stops arriving.
  • Showing the popup on checkout and form pages. It competes with the page's one job.
  • Ignoring phones. Overlays on small screens interrupt far more.
  • Different consent wording in each arm. The subscribers are not equally usable.

Frequently asked questions

Why not judge the popup on sign-ups?

A popup will almost always produce more sign-ups than an in-page form, because it puts the form in front of everyone who triggers it. That is not the question. The question is whether the extra subscribers are worth more than the purchases and goodwill the interruption costs, which only revenue per visitor over a follow-up window can show.

Should the popup offer a discount?

Use the same offer as the in-page sign-up so the test measures the popup, not a better offer. If the in-page form offers nothing and the popup offers a discount, a win cannot be credited to the format. Test the offer separately once the format question is settled.

How should the test handle mobile visitors?

Write a separate trigger and display rule for phones into the plan before launch, and read the device split you named in advance. Exit intent is usually detected from pointer movement, which phones lack, and a full-screen overlay on a small screen interrupts far more than a desktop box does.

Terms used on this page

Tools to try

Frameworks

Where this page is used

Also related

Sources

  1. Nielsen Norman Group articles — Nielsen Norman Group
  2. Understanding page experience in Google Search results — Google Search Central
  3. Direct marketing guidance — Information Commissioner's Office (UK)

Related