Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Experiment Library

Double Opt-In Test

An experiment design for one sign-up form: submissions split between single opt-in with validation and double opt-in, read on email revenue per sign-up.
Experimentintermediate9 min readUpdated

Hypothesis

If sign-ups from one web form go through double opt-in instead of single opt-in with validation, the list will gain fewer addresses but revenue from email per form submission over the following weeks will hold, because unconfirmed addresses rarely buy, while bounces and complaints from the form's sign-ups fall. The decision behind it is concrete: confirm sign-ups from a form with double opt-in, or add them on single opt-in with address validation.

The comparison of single and double opt-in sets out the trade-off and chooses per source from the risk each carries: single opt-in where the address has just been proven, as at checkout, and double opt-in on forms exposed to typos, invented addresses and bots. Many forms sit between those cases. A footer form on a site with modest bot traffic, or a sign-up attached to a small incentive, could go either way, and the comparison's reasoning cannot say which without the business's own data. That is the source this design is for.

Write the form, the window, the margin within which revenue counts as holding and the limit on each guardrail into the experiment hypothesis template before launch. The experimentation loop puts this step first because a hypothesis that predicts revenue will hold needs its margin fixed in advance, or any small drop can be argued away afterwards.

Metrics and guardrails

The primary metric is revenue from email per form submission over a fixed window after sign-up. For each arm, take every submission, follow the address for the same number of weeks from the moment it was submitted, add the revenue attributed to email from that address, and divide by the number of submissions in the arm. Where revenue is too sparse, use conversions from email per submission on the same basis. The denominator is fixed at submission, so the people double opt-in loses, the ones who never confirm, count against it with zero revenue. The window starts at submission in both arms, so the delay before a confirmed subscriber's welcome arrives is part of what is measured.

Three guardrails cover what double opt-in is supposed to protect and what it costs:

  • Hard bounce rate on the first sends. The high bounce rate guide traces many hard bounces back to the form; read them by arm on the welcome message and the first campaigns.
  • Spam complaint rate on the first sends. The Spam Complaint Rate on the same sends shows whether the arm is mailing people who did not mean to join.
  • Share of submissions that become mailable subscribers. In the single opt-in arm, submissions that pass validation; in the double opt-in arm, submissions that confirm. This is the cost side of the trade, and it drives the form's contribution to List Growth Rate.

The guide to valuing an email subscriber explains why attributed revenue is not the same as value, and why a source whose subscribers often complain can carry a negative value. This test compares two arms on the same attribution, so the bias is shared, but the guardrails matter because complaints cost something revenue does not show.

Design

Randomise at the level of the form submission, assigned at the moment it is submitted. The form itself is identical in both arms, with the same fields, the same consent wording and the same record kept. After submission, the single opt-in arm validates the address and adds it to the list at once, and the double opt-in arm sends a confirmation message and adds the address only when its owner clicks. If the same address submits twice, keep its first assignment.

Design elementChoiceReason
Unit of randomisationForm submission, assigned at submitThe process acts on each submission; the form stays identical
ArmsSingle opt-in with address validation; double opt-inCompares double opt-in with the strongest single opt-in, not a careless one
ConsentSame wording and the same record in both arms, plus the confirmation time in the double opt-in armBoth arms must meet the market's rules; only the confirmation step differs
Held constantThe form, its placement, the welcome series and the campaign calendarIsolates the confirmation step
Follow-upEach sign-up followed to the same fixed age from submissionRemoves differences in how long each address has been mailed
Primary metricRevenue from email per submissionCounts the addresses double opt-in loses

Consent comes first. The list building and consent guide explains, from the GDPR and the UK Information Commissioner's Office's direct marketing guidance, that consent must be a clear indication of the person's wishes and that the sender must be able to demonstrate it was given. Double opt-in is one strong way of evidencing it, not the only one, which is why the comparison page treats it as a choice rather than a requirement. Run this test only where both processes are acceptable for the form and the market. This is general direction, not legal advice.

Both arms feed the same welcome series. The welcome series guide covers its content; in the double opt-in arm it starts at the confirmation click, and in the single opt-in arm at once. Design the confirmation message as the comparison page and the list building guide describe: sent immediately, plain about what it is for, one action, with unconfirmed addresses expiring after a stated period.

Sample size reasoning

Revenue per submission is mostly zeros with a few buyers, so the test needs many submissions or a long window. The inputs are the form's submissions per week, the revenue per subscriber from that source and the smallest difference worth changing the process for. Because the hypothesis is that revenue holds, the question is whether the test can show that the difference sits inside the margin set before launch, which usually needs more submissions than showing a difference exists. The sample size and significance guide explains how those inputs combine and why skewed metrics need large samples.

If the form is too small, compare the arms on conversions from email per submission instead, which is less skewed. The conversion rate calculator turns each arm's conversions and submissions into rates to compare.

Duration

Take entries for whole weekly cycles, so that weekday and weekend sign-ups fall equally in both arms. Then follow each sign-up to the same fixed age, long enough to cover the welcome series and several regular campaigns, so that both the early difference in welcome timing and the later difference in buying are counted. The last entries set the reading date: it falls when they reach that age.

Reading the results and the decision rule

Read the test once, on the reading date. Compare revenue from email per submission between the arms, with the interval around the difference, against the margin set before launch. Then check the guardrails. Write the decision rule into the plan in advance:

ResultDecision
Revenue per submission holds within the margin and bounces or complaints fallUse double opt-in for this form
Revenue per submission falls reliably and the guardrails show no problem in the single opt-in armKeep single opt-in with validation
Revenue holds but the guardrails show no differenceKeep the simpler process to run, which is usually single opt-in with validation
Any resultApply it to this source only

Pitfalls

Shared sender reputation. Both arms send from the same domain. The deliverability guide explains that mailbox providers judge a sender largely by how recipients react, and it summarises Google's email sender guidelines, which ask senders to keep reported spam low, authenticate their mail and make unsubscribing easy. Complaints from one arm can therefore lower Deliverability for both, so watch complaints for the whole list, not only by arm, and stop the test if the list-wide figure moves towards its limit.

A confirmation email that silently fails. If the confirmation lands in spam or stops sending, double opt-in loses genuine subscribers and the test measures a broken message. Send to test addresses every day of the entry period and check the confirmation arrives in the inbox.

Results applied to other sources. The comparison page's point stands: the answer depends on the source. Run another test, or apply the comparison's reasoning, for each other form.

Revenue read per subscriber instead of per submission. Double opt-in subscribers are a filtered group, so they earn more each almost by definition. Read per submission.

Two neighbouring designs answer different questions about the same list: the exit-intent pop-up test tests where and when a sign-up form appears, and the inactive subscriber sunset test decides what to do with subscribers who stop engaging after they have joined.

Illustrative example — not a real result

Fennolux Prints tests double opt-in on its footer sign-up form

Fennolux Prints, an invented online art print shop, runs single opt-in with validation on its footer form and is unsure whether the form attracts typos and bots. Over eight weeks, 4,800 submissions are split at random, 2,400 to each arm, and each address is followed for ninety days from submission. The plan sets a margin of £0.30: double opt-in counts as holding if its revenue per submission is no more than that below single opt-in.

ArmMailable subscribersEmail revenueRevenue per submissionHard bounces, first sendsComplaints, first sends
Single opt-in with validation2,330 (97%)£5,544£2.313.8%0.21%
Double opt-in1,560 (65%)£5,376£2.240.3%0.05%

Revenue per submission is £0.07 lower with double opt-in, with an interval of roughly −£0.22 to +£0.08, inside the margin, while bounces and complaints fall sharply. Read per subscriber, double opt-in would appear to earn £3.45 against £2.38, which is why the plan reads per submission. Fennolux moves the footer form to double opt-in and leaves its checkout box on single opt-in.

Common mistakes

Opt-in tests usually go wrong in what they divide by and how far they generalise. The experiment library lists the designs beside this one.

  • Comparing subscribers instead of submissions. Double opt-in wins per subscriber because it has fewer of them.
  • Testing against careless single opt-in. Validation belongs in the control, or the test flatters double opt-in.
  • Starting the window at confirmation. The delay before the welcome is part of the cost; start at submission.
  • Changing the consent wording between arms. Only the confirmation step should differ.
  • Ignoring the whole list's complaints. Reputation is shared, so one arm's complaints affect both.
  • Rolling the result out to every form. Each source carries its own risk; the result belongs to this one.

Frequently asked questions

Why divide by submissions rather than by subscribers?

Because the two arms turn submissions into subscribers at different rates, and that difference is the cost being tested. Double opt-in loses everyone who never confirms, so its subscribers are a filtered group and earn more each. Dividing by subscribers would credit double opt-in with the filtering and hide the people it lost. Dividing by submissions compares what the same stream of sign-ups earns under each process.

Is it acceptable to add some people on single opt-in during the test?

Only where single opt-in with clear consent is acceptable for that form and market in the first place. The test compares two processes the business could lawfully choose between; it is not a way to use a weaker process where a stronger one is needed. Both arms show the same consent wording and keep the same record of it. This is general direction, not legal advice, and the rules for each market should be checked before launch.

Can the result be applied to every sign-up form?

No. Each source carries its own risk of typing errors, invented addresses and bots, which is why the usual advice is to choose the process source by source. The test answers the question for the form it ran on. A checkout box, a giveaway and a footer form can each reach a different answer, and a source whose risk is unclear deserves its own test.

Terms used on this page

Tools to try

Frameworks

Sources

  1. Direct marketing guidance — Information Commissioner's Office (UK)
  2. Regulation (EU) 2016/679 (GDPR) — EUR-Lex
  3. Email sender guidelines — Google Workspace Admin Help

Related