Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Experiment Library

Email Subject Line Test

A test design for email subject lines judged on clicks, not opens: the hypothesis, unsubscribe and complaint guardrails, split design and reading results.
Experimentbeginner6 min readUpdated

Hypothesis

A subject line that names the concrete benefit inside the email will raise clicks per delivered email compared with a curiosity-led line, because readers who open know what they will get. The logic runs through the open: a curiosity line may attract more opens, but some of those readers find nothing they wanted and leave without clicking. A benefit line may attract fewer opens from the wrong people and more clicks from the right ones.

That is why the hypothesis is stated in clicks. It predicts something different from the usual subject line folklore, and it can be wrong in an informative way. If the curiosity line wins on clicks too, your audience rewards intrigue and the benefit line was too dull. Write the hypothesis and the two exact subject lines into the experiment hypothesis template, and keep the preview text identical in both arms, because preview text is read alongside the subject line and would otherwise be a second variable. The newsletter design and copy guide covers how to write each kind of line; this page only covers how to test them.

Metrics and guardrails

The primary metric is click rate per delivered email: unique recipients who clicked any tracked link, divided by emails delivered in that arm. Using delivered rather than sent removes bounces, which can differ by chance between arms. It is a click-through measure on the whole delivered audience, related to but not the same as CTR on opened emails, which inherits the open-count distortion. The email metrics and testing guide explains how privacy features affect opens and why clicks are the steadier KPI for most tests.

Two guardrails protect the list. Unsubscribe rate rises when a line attracts people to an email that does not match the promise. Spam complaint rate is the more serious signal: complaints harm sender reputation and future delivery, as the deliverability guide explains. Agree beforehand that any noticeable rise in complaints in the challenger arm stops the line from being adopted, whatever happens to clicks. Downstream conversions per delivered email are worth reporting as a secondary read.

Design

Randomise at recipient level within the same send. Split the list into two random halves, send the control line to one and the challenger to the other at the same moment, with identical content, preview text, sender name and send time. Then repeat the comparison on several regular sends, each with its own pair of lines written to the same rule: one names the benefit, one leads with curiosity. A single send tells you about one email; repeated sends tell you about the rule.

Design elementChoiceReason
Unit of randomisationRecipient, fresh split each sendAvoids one half always getting the same treatment
VariantsBenefit-led line versus curiosity-led lineTests a writing rule, not a single line
Held constantContent, preview text, sender, send timeOnly the subject line differs
RepetitionSeveral regular sendsSeparates the rule from one-off topics
Primary countUnique clicks per delivered emailAvoids inflated opens
Read windowSame number of days after each sendLate clickers counted equally

Resplit the list at random for each send. If the same half always receives benefit lines, any difference in that half's habits becomes confused with the treatment.

Sample size reasoning

Clicks are rarer than opens, so a click-based test needs a larger audience per arm to detect the same relative change. That is the honest cost of using a trustworthy metric. The inputs are your typical click rate per delivered email, the smallest improvement worth changing your writing rule for, and the confidence you require; the sample size and significance guide explains how to turn those into a required audience. The conversion rate calculator can be used to compute each arm's rate from clicks and deliveries.

On a small list, one send will rarely be enough. That is where repetition helps: pooling the results of several sends that apply the same rule builds the sample over time. Decide before the first send how many sends you will pool, so you are not tempted to stop as soon as the running total looks good.

Reading the results

After the agreed number of sends and the agreed read window after each, pool the clicks and deliveries by arm and compare the click rates. Check the direction send by send as well. A rule that wins on the pooled figure but loses on most individual sends is being carried by one unusual email.

Then check the guardrails. Unsubscribes and complaints should be no worse in the benefit arm; if the curiosity arm shows more complaints, note it even if curiosity wins on clicks, because complaints have a long tail. Report opens if stakeholders expect them, but label them as inflated and never let them override clicks. If the benefit rule wins, update your writing guidelines and keep testing variations of the benefit line. If the result is flat, the subject line style may simply matter less to your audience than content and timing.

Illustrative example — not a real result

A garden centre newsletter tests benefit-led lines

Teaselburrow Plants, an invented garden centre, sends a weekly newsletter to about 24,000 subscribers. Over six sends it splits the list at random each time. One half gets a benefit line such as "Three shrubs that flower all winter, in stock now"; the other gets a curiosity line such as "You won't believe what's blooming". Pooled across the six sends, clicks per delivered email are 2.6% for benefit lines and 2.1% for curiosity lines. Recorded opens favour curiosity lines, but the team treats opens as unreliable. Unsubscribes are similar; spam complaints are slightly higher on curiosity lines. The newsletter adopts benefit-led lines.

Pitfalls

Inflated opens are the pitfall that shaped this design. Privacy features record opens that did not happen, and the share of such recipients varies by audience, so open-based comparisons can be won or lost on mailbox mix rather than on the subject line. Treat opens as a diagnostic, never as the result.

Automatic winner-sends are the second. Many email platforms test two lines on a small sample, then send the "winner" to the rest of the list after a short wait. The sample is small, the wait is short and the choice is often made on opens, which favours curiosity lines and early readers. Use full random halves for a learning test. The third pitfall is a misleading line: a subject line that implies content the email does not contain can attract complaints, and guidance such as the FTC's CAN-SPAM compliance guide expects subject lines not to mislead recipients. Treat this as general direction and check the rules for your own market, not as legal advice.

Common mistakes

Subject line tests are the most frequently run experiment in email and the most frequently misread. Once the writing rule is settled, the email send time test is the usual next question, and the experiment library covers the designs for other channels.

  • Judging on opens. Privacy features make opens unreliable; read clicks per delivered email.
  • Changing preview text too. Preview text is read with the subject line and becomes a second variable.
  • Testing once. One send reflects one topic; repeat the rule across several sends.
  • Keeping the same halves. Resplit at random each send so habits do not masquerade as effects.
  • Trusting automatic winner picks. Short windows and open-based choices favour the wrong lines.
  • Ignoring complaints. A line that wins clicks but raises complaints damages future delivery.

Frequently asked questions

Why not use open rate to judge a subject line?

Mail privacy features in some email apps load images automatically, which records an open whether or not the person read the message. Open counts are therefore inflated and uneven across audiences, so clicks per delivered email is the more trustworthy signal.

Should we use the platform's automatic winner send?

Only if the winner is chosen on clicks and the test window is long enough. Many automatic winner features pick on opens within a short window, which favours curiosity lines and early readers. For a learning test, send both lines to full random halves instead.

Is a curiosity-led subject line ever misleading?

It can be. Regulators such as the FTC expect subject lines not to mislead recipients about the content of the message. A curiosity line that teases something the email does not contain is both a compliance risk and a source of complaints.

Terms used on this page

Tools to try

Frameworks

Where this page is used

Also related

Sources

  1. CAN-SPAM Act: A Compliance Guide for Business — Federal Trade Commission

Related