Hypothesis
If sign-ups from one web form go through double opt-in instead of single opt-in with validation, the list will gain fewer addresses but revenue from email per form submission over the following weeks will hold, because unconfirmed addresses rarely buy, while bounces and complaints from the form's sign-ups fall. The decision behind it is concrete: confirm sign-ups from a form with double opt-in, or add them on single opt-in with address validation.
The comparison of single and double opt-in sets out the trade-off and chooses per source from the risk each carries: single opt-in where the address has just been proven, as at checkout, and double opt-in on forms exposed to typos, invented addresses and bots. Many forms sit between those cases. A footer form on a site with modest bot traffic, or a sign-up attached to a small incentive, could go either way, and the comparison's reasoning cannot say which without the business's own data. That is the source this design is for.
Write the form, the window, the margin within which revenue counts as holding and the limit on each guardrail into the experiment hypothesis template before launch. The experimentation loop puts this step first because a hypothesis that predicts revenue will hold needs its margin fixed in advance, or any small drop can be argued away afterwards.
Metrics and guardrails
The primary metric is revenue from email per form submission over a fixed window after sign-up. For each arm, take every submission, follow the address for the same number of weeks from the moment it was submitted, add the revenue attributed to email from that address, and divide by the number of submissions in the arm. Where revenue is too sparse, use conversions from email per submission on the same basis. The denominator is fixed at submission, so the people double opt-in loses, the ones who never confirm, count against it with zero revenue. The window starts at submission in both arms, so the delay before a confirmed subscriber's welcome arrives is part of what is measured.
Three guardrails cover what double opt-in is supposed to protect and what it costs:
- Hard bounce rate on the first sends. The high bounce rate guide traces many hard bouncesAn email rejected permanently by the receiving server, usually because the address does not exist or the domain cannot receive mail. back to the form; read them by arm on the welcome message and the first campaigns.
- Spam complaint rate on the first sends. The Spam Complaint RateShare of delivered emails that recipients marked as spam. on the same sends shows whether the arm is mailing people who did not mean to join.
- Share of submissions that become mailable subscribers. In the single opt-in arm, submissions that pass validation; in the double opt-in arm, submissions that confirm. This is the cost side of the trade, and it drives the form's contribution to List Growth RateNet change in an email list over a period as a share of its size at the start..
The guide to valuing an email subscriber explains why attributed revenue is not the same as value, and why a source whose subscribers often complain can carry a negative value. This test compares two arms on the same attribution, so the bias is shared, but the guardrails matter because complaints cost something revenue does not show.
Design
Randomise at the level of the form submission, assigned at the moment it is submitted. The form itself is identical in both arms, with the same fields, the same consent wording and the same record kept. After submission, the single opt-in arm validates the address and adds it to the list at once, and the double opt-in arm sends a confirmation message and adds the address only when its owner clicks. If the same address submits twice, keep its first assignment.
| Design element | Choice | Reason |
|---|---|---|
| Unit of randomisation | Form submission, assigned at submit | The process acts on each submission; the form stays identical |
| Arms | Single opt-in with address validation; double opt-in | Compares double opt-in with the strongest single opt-in, not a careless one |
| Consent | Same wording and the same record in both arms, plus the confirmation time in the double opt-in arm | Both arms must meet the market's rules; only the confirmation step differs |
| Held constant | The form, its placement, the welcome series and the campaign calendar | Isolates the confirmation step |
| Follow-up | Each sign-up followed to the same fixed age from submission | Removes differences in how long each address has been mailed |
| Primary metric | Revenue from email per submission | Counts the addresses double opt-in loses |
Consent comes first. The list building and consent guide explains, from the GDPR and the UK Information Commissioner's Office's direct marketing guidance, that consent must be a clear indication of the person's wishes and that the sender must be able to demonstrate it was given. Double opt-in is one strong way of evidencing it, not the only one, which is why the comparison page treats it as a choice rather than a requirement. Run this test only where both processes are acceptable for the form and the market. This is general direction, not legal advice.
Both arms feed the same welcome series. The welcome series guide covers its content; in the double opt-in arm it starts at the confirmation click, and in the single opt-in arm at once. Design the confirmation message as the comparison page and the list building guide describe: sent immediately, plain about what it is for, one action, with unconfirmed addresses expiring after a stated period.
Sample size reasoning
Revenue per submission is mostly zeros with a few buyers, so the test needs many submissions or a long window. The inputs are the form's submissions per week, the revenue per subscriber from that source and the smallest difference worth changing the process for. Because the hypothesis is that revenue holds, the question is whether the test can show that the difference sits inside the margin set before launch, which usually needs more submissions than showing a difference exists. The sample size and significance guide explains how those inputs combine and why skewed metrics need large samples.
If the form is too small, compare the arms on conversions from email per submission instead, which is less skewed. The conversion rate calculator turns each arm's conversions and submissions into rates to compare.
Duration
Take entries for whole weekly cycles, so that weekday and weekend sign-ups fall equally in both arms. Then follow each sign-up to the same fixed age, long enough to cover the welcome series and several regular campaigns, so that both the early difference in welcome timing and the later difference in buying are counted. The last entries set the reading date: it falls when they reach that age.
Reading the results and the decision rule
Read the test once, on the reading date. Compare revenue from email per submission between the arms, with the interval around the difference, against the margin set before launch. Then check the guardrails. Write the decision rule into the plan in advance:
| Result | Decision |
|---|---|
| Revenue per submission holds within the margin and bounces or complaints fall | Use double opt-in for this form |
| Revenue per submission falls reliably and the guardrails show no problem in the single opt-in arm | Keep single opt-in with validation |
| Revenue holds but the guardrails show no difference | Keep the simpler process to run, which is usually single opt-in with validation |
| Any result | Apply it to this source only |
Pitfalls
Shared sender reputation. Both arms send from the same domain. The deliverability guide explains that mailbox providers judge a sender largely by how recipients react, and it summarises Google's email sender guidelines, which ask senders to keep reported spam low, authenticate their mail and make unsubscribing easy. Complaints from one arm can therefore lower DeliverabilityWhether sent email reaches the inbox rather than spam or rejection, decided by sender reputation. for both, so watch complaints for the whole list, not only by arm, and stop the test if the list-wide figure moves towards its limit.
A confirmation email that silently fails. If the confirmation lands in spam or stops sending, double opt-in loses genuine subscribers and the test measures a broken message. Send to test addresses every day of the entry period and check the confirmation arrives in the inbox.
Results applied to other sources. The comparison page's point stands: the answer depends on the source. Run another test, or apply the comparison's reasoning, for each other form.
Revenue read per subscriber instead of per submission. Double opt-in subscribers are a filtered group, so they earn more each almost by definition. Read per submission.
Two neighbouring designs answer different questions about the same list: the exit-intent pop-up test tests where and when a sign-up form appears, and the inactive subscriber sunset test decides what to do with subscribers who stop engaging after they have joined.
Example: an invented art print shop tests double opt-in on its footer form (illustrative)
Fennolux Prints tests double opt-in on its footer sign-up form
Fennolux Prints, an invented online art print shop, runs single opt-in with validation on its footer form and is unsure whether the form attracts typos and bots. Over eight weeks, 4,800 submissions are split at random, 2,400 to each arm, and each address is followed for ninety days from submission. The plan sets a margin of £0.30: double opt-in counts as holding if its revenue per submission is no more than that below single opt-in.
| Arm | Mailable subscribers | Email revenue | Revenue per submission | Hard bounces, first sends | Complaints, first sends |
|---|---|---|---|---|---|
| Single opt-in with validation | 2,330 (97%) | £5,544 | £2.31 | 3.8% | 0.21% |
| Double opt-in | 1,560 (65%) | £5,376 | £2.24 | 0.3% | 0.05% |
Revenue per submission is £0.07 lower with double opt-in, with an interval of roughly −£0.22 to +£0.08, inside the margin, while bounces and complaints fall sharply. Read per subscriber, double opt-in would appear to earn £3.45 against £2.38, which is why the plan reads per submission. Fennolux moves the footer form to double opt-in and leaves its checkout box on single opt-in.
Common mistakes
Opt-in tests usually go wrong in what they divide by and how far they generalise. The experiment library lists the designs beside this one.
- Comparing subscribers instead of submissions. Double opt-in wins per subscriber because it has fewer of them.
- Testing against careless single opt-in. Validation belongs in the control, or the test flatters double opt-in.
- Starting the window at confirmation. The delay before the welcome is part of the cost; start at submission.
- Changing the consent wording between arms. Only the confirmation step should differ.
- Ignoring the whole list's complaints. Reputation is shared, so one arm's complaints affect both.
- Rolling the result out to every form. Each source carries its own risk; the result belongs to this one.
