Hypothesis
If the headline states the specific outcome the visitor clicked for instead of a generic brand claim, more visitors will complete the primary conversion, because message match improves. The reasoning matters as much as the prediction. A visitor arriving from an ad or a search result carries a promise in their head; the headline is the first place the page either confirms that promise or makes the visitor work to find it. The message match guide explains why that gap costs conversions, and usability research such as the articles published by Nielsen Norman Group consistently describes people scanning rather than reading, so the opening line does a disproportionate share of the work.
Write the hypothesis into the experiment hypothesis template before building anything, and attach the evidence that prompted it: ad copy that promises one thing while the page says another, session recordings of visitors scrolling straight past the headline, or survey answers asking whether the page offers what they searched for. If you cannot point to evidence, the hypothesis-writing tutorial shows how to find some before spending traffic on a guess.
Metrics and guardrails
The primary metric is landing page conversion rate: visitors who complete the page's one primary action divided by visitors who reached the page. Define the action precisely before launch, for example a submitted enquiry form or a completed purchase, and count it from the same source for both variants. Use the conversion rate calculator to check the arithmetic on each arm.
Two guardrails protect the decision. Lead quality rate, the share of conversions that sales or the CRM later mark as genuine, guards against a headline that promises more than the offer delivers and so attracts the wrong people. Bounce rate guards against a headline that confuses or alienates visitors. Set the tolerance for each guardrail in words before launch, such as "no meaningful fall", and agree who decides when a guardrail is breached. A headline that lifts CVRShare of visits or clicks that convert. but drags lead quality down is not a winner; it has moved the problem further down the FunnelModel of the customer journey stages..
Design
Randomise at visitor level, not page view level, so a returning visitor always sees the same headline. Most testing tools do this with a cookie or a user identifier. Run one challenger against the current control; adding more challengers splits the traffic further and lengthens the test. Keep everything else on the page identical, including the subheading, images, form and load speed, because the A/B testing guide explains that each additional change muddies the attribution of the result.
| Design element | Choice | Reason |
|---|---|---|
| Unit of randomisation | Visitor | Stops one person seeing both headlines |
| Variants | Control headline versus one outcome-led challenger | Isolates the single change |
| Traffic split | Even split between arms | Fastest route to the planned sample |
| Duration | Whole weekly cycles until the planned sample is reached | Covers weekday and weekend behaviour |
| Traffic sources | Unchanged during the test | Keeps the visitor mix stable |
| Decision owner | Named before launch | Prevents arguing over the result afterwards |
Duration is set by two conditions, both of which must be met: the planned sample per arm, and a whole number of weeks. Visitors on a Monday morning behave differently from visitors on a Saturday evening, so a test that ends mid-week over-represents one group.
Sample size reasoning
The number of visitors needed depends on three things you know before launch: the baseline conversion rate of the current page, the smallest lift that would be worth rolling out, and how confident you need to be. A low baseline and a small minimum effect both push the required sample up sharply, which is why headline tests on low-traffic pages often cannot detect anything subtle. In that case, test a bolder headline rather than a tweak, or test on a page with more traffic.
Work out the figure with the method in the sample size and significance guide and write it in the hypothesis document. Then convert it into calendar time using your normal weekly traffic and round up to whole weeks. If the answer is longer than the business will tolerate, change the test now rather than stopping it early later.
Reading the results
Read the result once, when the planned sample and duration have both been reached. Look first at the primary metric: is the difference between arms larger than the noise your significance method allows for? Then check each guardrail. Only when the primary metric has improved and the guardrails have held is the challenger a winner.
A flat result means the headline change was not large enough to matter at the effect size you could detect. Record it and move on to a bolder idea. A mixed result, where conversions rise but lead quality falls, usually means the headline over-promised; rewrite it to state the outcome more precisely and test again. Before rolling out, segment the result by traffic source. A headline that wins for paid search visitors and loses for email visitors may belong only on the paid landing page.
A bookkeeping service tests an outcome-led headline
Ledgerly, an invented bookkeeping service for sole traders, runs search ads promising "your tax return done in a week". Its landing page headline reads "Accounting made simple". The team tests the challenger "Your self-assessment filed within seven days" over three full weeks. The control converts at 4.1% and the challenger at 5.0% across roughly 9,000 visitors per arm, which their significance method treats as a real difference. Lead quality, marked by the sales team, holds at about 70% in both arms, and bounce rate falls slightly. They roll the headline out to the paid landing page only, because email visitors showed no difference.
Pitfalls
Novelty is the first trap. Returning visitors may react to anything new, so an early lift can fade within days. Running whole weeks and reading once protects against it. Traffic mix is the second: if the media team doubles spend on one campaign mid-test, the visitor profile shifts and the comparison is no longer like for like, so freeze campaign changes for the test period or agree to restart.
The third trap is a mismatched ad. A headline that matches one ad group's promise will look worse for visitors arriving from a different ad. Check which ads send traffic to the page before writing the challenger, and segment by source when reading. Finally, watch for flicker, where the control headline appears briefly before the challenger loads; it contaminates the challenger arm and can make a good headline look bad. Load the page on a slow phone before launch, so any flickerThe brief flash of the original version before a browser-side test script applies the variant, which turns a result about the change into one about the fault. is found before the first visitor is counted.
The walk-through on sizing a landing page test with the conversion rate calculator shows how to enter each arm separately and read the extra conversions as the value at stake.
Common mistakes
Headline tests look simple, which is why they are run carelessly. Most of the errors below come from treating the test as a copywriting exercise rather than an experiment. If the page also carries a form, the lead form length test is the natural next experiment, and the experiment library lists the other designs.
- Testing a synonym. A small wording change rarely produces an effect large enough to detect. Test a different promise, not a different adjective.
- Changing the subheading too. Two changes mean two possible causes. Keep everything but the headline fixed.
- Stopping on a good day. Peeking and stopping inflates false winners. Read once, at the planned point.
- Ignoring lead quality. A headline that attracts the wrong people looks like a win until sales report back.
- Rolling out everywhere. A headline that matches one ad's promise may hurt pages fed by other ads.
- Skipping the record. Without a written result, the same headline idea is tested again next year.
