Hypothesis
If an email capture popup appears when visitors show intent to leave, more first-time visitors will join the list without reducing purchases in the session, so revenue per visitor including later email revenue will rise. The decision is whether to run an email capture popup on the site, and if so with which trigger, or to rely on in-page sign-up.
The hypothesis carries two claims, and the test has to check both. The first is that the popup grows the list, which it almost certainly will. The second is that it does so without costing purchases, which is far less certain: an overlay that appears at the wrong moment interrupts someone who was about to buy, and usability research such as the articles published by Nielsen Norman Group has long described how intrusive overlays frustrate people trying to complete a task. The list building and consent guide sets out where sign-up placements work and what to watch for with overlays; this test measures the trade on your site. Record the trigger, the offer, the pages in scope and the follow-up window in the experiment hypothesis template.
Metrics and guardrails
The primary metric is Revenue per VisitorRevenue divided by the number of visitors in the same period.: revenue from each arm's assigned visitors during the collection period, plus revenue from the subscribers each arm produced, popup and in-page alike, over a fixed follow-up window, divided by the number of assigned visitors. Including later email revenue is what lets the popup's benefit appear at all, since most sign-ups do not buy on the day; fixing the window is what stops the popup arm accumulating credit indefinitely.
Three guardrails catch the costs. Session conversion rate, the CVRShare of visits or clicks that convert. of visits to purchases in the visit itself, shows whether the popup interrupts buyers; a fall here is the cost the hypothesis says will not happen. Return-visit rate shows whether visitors who saw the popup come back less, which is the slower form of annoyance. The unsubscribe rate of popup sign-ups, read over the follow-up window, shows whether the people the popup captured wanted the emails; a placement that fills the list with people who leave in the first fortnight adds to List Growth RateNet change in an email list over a period as a share of its size at the start. on paper and to complaints in practice.
Design
Randomise at the level of the visitor, assign on the first visit and keep the visitor in the same arm on return. The control shows the site as it is, with the in-page sign-up form. The variant shows the same site, same in-page form, plus a popup triggered by exit intent, carrying the same offer, the same form fields and the same consent wording as the in-page form. Each visitor sees the popup at most once in a set period.
| Design element | Choice | Reason |
|---|---|---|
| Unit of randomisation | Visitor, assigned on the first visit and kept in one arm | Return visits and later purchases belong to the same arm |
| Arms | In-page sign-up only, against in-page sign-up plus exit-intent popup | Isolates the popup |
| Offer, form and consent wording | Identical in popup and in-page form | Tests the format, not the offer |
| Frequency cap | Popup shown at most once per visitor in a set period | Limits annoyance and keeps the treatment stable |
| Mobile | A separate trigger and display rule written before launch | Exit intent and overlays behave differently on phones |
| Primary metric | Revenue per visitor including email revenue over a fixed window | Weighs captured emails against lost sales |
Where the popup sits relative to the page's own call to action matters. The anatomy of a landing page guide explains how each page should have one job; a popup that fires on a checkout or lead form page competes with that job directly, so exclude those pages from the trigger. Whether the popup uses single or Double Opt-InA sign-up process in which the subscriber confirms through a link sent to the address given. should match the in-page form, so that the two arms produce comparable subscribers.
Sample size reasoning
Revenue per visitor is skewed and the email revenue arrives later, so the test needs a larger sample and a fixed follow-up window. Most visitors spend nothing, a few spend a lot, and the email revenue from sign-ups adds a second, delayed source of variation. The sample size and significance guide covers skewed metrics and why they need more visitors than a conversion rate to detect the same relative change. The conversion rate calculator gives the baseline session conversion and sign-up rates the plan starts from. If the site's traffic cannot carry a revenue test, read session conversion as the primary metric with a strict non-inferiority margin, and say plainly in the plan that the email value is estimated rather than measured.
Duration
Run at least two full weekly cycles to collect visitors, then a fixed window for sign-ups to receive and act on emails. The collection period ends on a planned date regardless of how the numbers look; the follow-up window then runs for every subscriber, including those who joined on the last day. Keep the welcome series identical for sign-ups from both arms, as the welcome series tutorial describes, so that email revenue differences come from who signed up, not what they received.
Reading the results and the decision rule
Read once the follow-up window has closed. Compare revenue per visitor between arms with its interval, then read session conversion, return visits and the unsubscribe rate of popup sign-ups. Read the device split named before launch last. The A/B testing guide covers the reading discipline in general. The rule, written before launch:
| Result | Decision |
|---|---|
| Revenue per visitor rises and session conversion holds | Keep the popup with this trigger |
| Revenue per visitor rises but session conversion falls on some pages | Keep the popup, excluded from those pages, and re-test |
| Sign-ups rise but popup sign-ups unsubscribe quickly | Drop the popup or change the trigger; the list is growing with the wrong people |
| Revenue per visitor falls or session conversion falls overall | Drop the popup and rely on in-page sign-up |
Pitfalls
Reading sign-ups alone is the defining pitfall. A popup produces more sign-ups almost by definition, and a test that stops there will always recommend it, whatever it costs in interrupted purchases and lost goodwill. Revenue per visitor over a fixed window is the only reading that weighs both sides.
Mobile interstitials harming page experience are the second. Overlays that cover the content on a phone are far more disruptive than a box on a desktop screen, and Google Search Central's page experience documentation describes intrusive interstitials as part of what makes a page harder to use. Write the mobile rule into the plan, prefer a smaller format on phones, and read the device split you named in advance. The third is consent wording differing between arms. If the popup's wording is vaguer or its checkbox behaves differently from the in-page form, the arms produce subscribers with different consent, and the test compares lists that are not equally usable. The ICO's direct marketing guidance explains the UK's rules on consent for marketing email; keep one approved wording in both places.
A coffee roaster tests an exit-intent popup
Quarrybrook Coffee Roasters, an invented online coffee roaster, has an in-page sign-up in its footer. It tests an exit-intent popup with the same offer, a free brewing guide, over three weeks with 40,000 visitors per arm, then follows every sign-up for 30 days. The popup arm signs up 1,320 visitors against 520 in control. Session conversion is 2.30% in control and 2.24% in the popup arm, inside the noise. Revenue per visitor including the 30-day email revenue is £1.31 in the popup arm and £1.24 in control, with an interval of roughly £0.01 to £0.13. Popup sign-ups unsubscribe at a higher rate in the first fortnight, but within the team's limit. On phones, session conversion falls in the popup arm, so Quarrybrook keeps the popup on desktop only and re-tests a smaller mobile format.
Interstitials are one of several page experience questions Google lists, and the guide to page experience beyond Core Web Vitals shows how to review a template for overlays, ads and mobile display together.
Common mistakes
Popup tests fail on what they count and when they stop counting. The social proof placement test is a quieter way to change a page's conversion, and the experiment library lists every design.
- Counting sign-ups as the result. The popup wins on sign-ups by construction.
- Using a better offer in the popup. The test then measures the offer, not the format.
- No follow-up window. Email revenue never arrives in the reading, or never stops arriving.
- Showing the popup on checkout and form pages. It competes with the page's one job.
- Ignoring phones. Overlays on small screens interrupt far more.
- Different consent wording in each arm. The subscribers are not equally usable.
