Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Experiment Library

Landing Page Headline Test

An experiment design for testing a landing page headline: the hypothesis, the conversion metric and guardrails, randomisation, duration and how to read results.
Experimentbeginner6 min readUpdated

Hypothesis

If the headline states the specific outcome the visitor clicked for instead of a generic brand claim, more visitors will complete the primary conversion, because message match improves. The reasoning matters as much as the prediction. A visitor arriving from an ad or a search result carries a promise in their head; the headline is the first place the page either confirms that promise or makes the visitor work to find it. The message match guide explains why that gap costs conversions, and usability research such as the articles published by Nielsen Norman Group consistently describes people scanning rather than reading, so the opening line does a disproportionate share of the work.

Write the hypothesis into the experiment hypothesis template before building anything, and attach the evidence that prompted it: ad copy that promises one thing while the page says another, session recordings of visitors scrolling straight past the headline, or survey answers asking whether the page offers what they searched for. If you cannot point to evidence, the hypothesis-writing tutorial shows how to find some before spending traffic on a guess.

Metrics and guardrails

The primary metric is landing page conversion rate: visitors who complete the page's one primary action divided by visitors who reached the page. Define the action precisely before launch, for example a submitted enquiry form or a completed purchase, and count it from the same source for both variants. Use the conversion rate calculator to check the arithmetic on each arm.

Two guardrails protect the decision. Lead quality rate, the share of conversions that sales or the CRM later mark as genuine, guards against a headline that promises more than the offer delivers and so attracts the wrong people. Bounce rate guards against a headline that confuses or alienates visitors. Set the tolerance for each guardrail in words before launch, such as "no meaningful fall", and agree who decides when a guardrail is breached. A headline that lifts CVR but drags lead quality down is not a winner; it has moved the problem further down the Funnel.

Design

Randomise at visitor level, not page view level, so a returning visitor always sees the same headline. Most testing tools do this with a cookie or a user identifier. Run one challenger against the current control; adding more challengers splits the traffic further and lengthens the test. Keep everything else on the page identical, including the subheading, images, form and load speed, because the A/B testing guide explains that each additional change muddies the attribution of the result.

Design elementChoiceReason
Unit of randomisationVisitorStops one person seeing both headlines
VariantsControl headline versus one outcome-led challengerIsolates the single change
Traffic splitEven split between armsFastest route to the planned sample
DurationWhole weekly cycles until the planned sample is reachedCovers weekday and weekend behaviour
Traffic sourcesUnchanged during the testKeeps the visitor mix stable
Decision ownerNamed before launchPrevents arguing over the result afterwards

Duration is set by two conditions, both of which must be met: the planned sample per arm, and a whole number of weeks. Visitors on a Monday morning behave differently from visitors on a Saturday evening, so a test that ends mid-week over-represents one group.

Sample size reasoning

The number of visitors needed depends on three things you know before launch: the baseline conversion rate of the current page, the smallest lift that would be worth rolling out, and how confident you need to be. A low baseline and a small minimum effect both push the required sample up sharply, which is why headline tests on low-traffic pages often cannot detect anything subtle. In that case, test a bolder headline rather than a tweak, or test on a page with more traffic.

Work out the figure with the method in the sample size and significance guide and write it in the hypothesis document. Then convert it into calendar time using your normal weekly traffic and round up to whole weeks. If the answer is longer than the business will tolerate, change the test now rather than stopping it early later.

Reading the results

Read the result once, when the planned sample and duration have both been reached. Look first at the primary metric: is the difference between arms larger than the noise your significance method allows for? Then check each guardrail. Only when the primary metric has improved and the guardrails have held is the challenger a winner.

A flat result means the headline change was not large enough to matter at the effect size you could detect. Record it and move on to a bolder idea. A mixed result, where conversions rise but lead quality falls, usually means the headline over-promised; rewrite it to state the outcome more precisely and test again. Before rolling out, segment the result by traffic source. A headline that wins for paid search visitors and loses for email visitors may belong only on the paid landing page.

Illustrative example — not a real result

A bookkeeping service tests an outcome-led headline

Ledgerly, an invented bookkeeping service for sole traders, runs search ads promising "your tax return done in a week". Its landing page headline reads "Accounting made simple". The team tests the challenger "Your self-assessment filed within seven days" over three full weeks. The control converts at 4.1% and the challenger at 5.0% across roughly 9,000 visitors per arm, which their significance method treats as a real difference. Lead quality, marked by the sales team, holds at about 70% in both arms, and bounce rate falls slightly. They roll the headline out to the paid landing page only, because email visitors showed no difference.

Pitfalls

Novelty is the first trap. Returning visitors may react to anything new, so an early lift can fade within days. Running whole weeks and reading once protects against it. Traffic mix is the second: if the media team doubles spend on one campaign mid-test, the visitor profile shifts and the comparison is no longer like for like, so freeze campaign changes for the test period or agree to restart.

The third trap is a mismatched ad. A headline that matches one ad group's promise will look worse for visitors arriving from a different ad. Check which ads send traffic to the page before writing the challenger, and segment by source when reading. Finally, watch for flicker, where the control headline appears briefly before the challenger loads; it contaminates the challenger arm and can make a good headline look bad. Load the page on a slow phone before launch, so any flicker is found before the first visitor is counted.

The walk-through on sizing a landing page test with the conversion rate calculator shows how to enter each arm separately and read the extra conversions as the value at stake.

Common mistakes

Headline tests look simple, which is why they are run carelessly. Most of the errors below come from treating the test as a copywriting exercise rather than an experiment. If the page also carries a form, the lead form length test is the natural next experiment, and the experiment library lists the other designs.

  • Testing a synonym. A small wording change rarely produces an effect large enough to detect. Test a different promise, not a different adjective.
  • Changing the subheading too. Two changes mean two possible causes. Keep everything but the headline fixed.
  • Stopping on a good day. Peeking and stopping inflates false winners. Read once, at the planned point.
  • Ignoring lead quality. A headline that attracts the wrong people looks like a win until sales report back.
  • Rolling out everywhere. A headline that matches one ad's promise may hurt pages fed by other ads.
  • Skipping the record. Without a written result, the same headline idea is tested again next year.

Frequently asked questions

Why test only the headline and not the whole page?

Changing one element tells you what caused the difference. If the headline, image and form all change at once, a win cannot be repeated on other pages because nobody knows which part did the work.

Should a headline test be judged on bounce rate?

No. Bounce rate is a guardrail that warns if the new headline drives people away, but a page can keep more visitors and still convert fewer of them. The decision rests on the conversion rate of the primary action.

How long should a landing page headline test run?

Run it for whole weeks so that weekday and weekend visitors are both represented, and until the sample planned in advance has been reached. Stopping on a good day is the most common way to declare a false winner.

Terms used on this page

Tools to try

Frameworks

Where this page is used

Also related

Sources

  1. Nielsen Norman Group articles — Nielsen Norman Group

Related