Hypothesis
A subject line that names the concrete benefit inside the email will raise clicks per delivered email compared with a curiosity-led line, because readers who open know what they will get. The logic runs through the open: a curiosity line may attract more opens, but some of those readers find nothing they wanted and leave without clicking. A benefit line may attract fewer opens from the wrong people and more clicks from the right ones.
That is why the hypothesis is stated in clicks. It predicts something different from the usual subject line folklore, and it can be wrong in an informative way. If the curiosity line wins on clicks too, your audience rewards intrigue and the benefit line was too dull. Write the hypothesis and the two exact subject lines into the experiment hypothesis template, and keep the preview textThe short line of text an inbox shows after or below an email's subject line, taken from the start of the message. identical in both arms, because preview text is read alongside the subject line and would otherwise be a second variable. The newsletter design and copy guide covers how to write each kind of line; this page only covers how to test them.
Metrics and guardrails
The primary metric is click rate per delivered email: unique recipients who clicked any tracked link, divided by emails delivered in that arm. Using delivered rather than sent removes bounces, which can differ by chance between arms. It is a click-through measure on the whole delivered audience, related to but not the same as CTRShare of impressions that result in a click. on opened emails, which inherits the open-count distortion. The email metrics and testing guide explains how privacy features affect opens and why clicks are the steadier KPIA metric tied directly to an objective. for most tests.
Two guardrails protect the list. Unsubscribe rate rises when a line attracts people to an email that does not match the promise. Spam complaint rateShare of delivered emails that recipients marked as spam. is the more serious signal: complaints harm sender reputation and future delivery, as the deliverability guide explains. Agree beforehand that any noticeable rise in complaints in the challenger arm stops the line from being adopted, whatever happens to clicks. Downstream conversions per delivered email are worth reporting as a secondary read.
Design
Randomise at recipient level within the same send. Split the list into two random halves, send the control line to one and the challenger to the other at the same moment, with identical content, preview text, sender name and send time. Then repeat the comparison on several regular sends, each with its own pair of lines written to the same rule: one names the benefit, one leads with curiosity. A single send tells you about one email; repeated sends tell you about the rule.
| Design element | Choice | Reason |
|---|---|---|
| Unit of randomisation | Recipient, fresh split each send | Avoids one half always getting the same treatment |
| Variants | Benefit-led line versus curiosity-led line | Tests a writing rule, not a single line |
| Held constant | Content, preview text, sender, send time | Only the subject line differs |
| Repetition | Several regular sends | Separates the rule from one-off topics |
| Primary count | Unique clicks per delivered email | Avoids inflated opens |
| Read window | Same number of days after each send | Late clickers counted equally |
Resplit the list at random for each send. If the same half always receives benefit lines, any difference in that half's habits becomes confused with the treatment.
Sample size reasoning
Clicks are rarer than opens, so a click-based test needs a larger audience per arm to detect the same relative change. That is the honest cost of using a trustworthy metric. The inputs are your typical click rate per delivered email, the smallest improvement worth changing your writing rule for, and the confidence you require; the sample size and significance guide explains how to turn those into a required audience. The conversion rate calculator can be used to compute each arm's rate from clicks and deliveries.
On a small list, one send will rarely be enough. That is where repetition helps: pooling the results of several sends that apply the same rule builds the sample over time. Decide before the first send how many sends you will pool, so you are not tempted to stop as soon as the running total looks good.
Reading the results
After the agreed number of sends and the agreed read window after each, pool the clicks and deliveries by arm and compare the click rates. Check the direction send by send as well. A rule that wins on the pooled figure but loses on most individual sends is being carried by one unusual email.
Then check the guardrails. Unsubscribes and complaints should be no worse in the benefit arm; if the curiosity arm shows more complaints, note it even if curiosity wins on clicks, because complaints have a long tail. Report opens if stakeholders expect them, but label them as inflated and never let them override clicks. If the benefit rule wins, update your writing guidelines and keep testing variations of the benefit line. If the result is flat, the subject line style may simply matter less to your audience than content and timing.
A garden centre newsletter tests benefit-led lines
Teaselburrow Plants, an invented garden centre, sends a weekly newsletter to about 24,000 subscribers. Over six sends it splits the list at random each time. One half gets a benefit line such as "Three shrubs that flower all winter, in stock now"; the other gets a curiosity line such as "You won't believe what's blooming". Pooled across the six sends, clicks per delivered email are 2.6% for benefit lines and 2.1% for curiosity lines. Recorded opens favour curiosity lines, but the team treats opens as unreliable. Unsubscribes are similar; spam complaints are slightly higher on curiosity lines. The newsletter adopts benefit-led lines.
Pitfalls
Inflated opens are the pitfall that shaped this design. Privacy features record opens that did not happen, and the share of such recipients varies by audience, so open-based comparisons can be won or lost on mailbox mix rather than on the subject line. Treat opens as a diagnostic, never as the result.
Automatic winner-sends are the second. Many email platforms test two lines on a small sample, then send the "winner" to the rest of the list after a short wait. The sample is small, the wait is short and the choice is often made on opens, which favours curiosity lines and early readers. Use full random halves for a learning test. The third pitfall is a misleading line: a subject line that implies content the email does not contain can attract complaints, and guidance such as the FTC's CAN-SPAM compliance guide expects subject lines not to mislead recipients. Treat this as general direction and check the rules for your own market, not as legal advice.
Common mistakes
Subject line tests are the most frequently run experiment in email and the most frequently misread. Once the writing rule is settled, the email send time test is the usual next question, and the experiment library covers the designs for other channels.
- Judging on opens. Privacy features make opens unreliable; read clicks per delivered email.
- Changing preview text too. Preview text is read with the subject line and becomes a second variable.
- Testing once. One send reflects one topic; repeat the rule across several sends.
- Keeping the same halves. Resplit at random each send so habits do not masquerade as effects.
- Trusting automatic winner picks. Short windows and open-based choices favour the wrong lines.
- Ignoring complaints. A line that wins clicks but raises complaints damages future delivery.
