Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Conversion Rate Optimisation › Conversion Research

Writing Test Hypotheses

A step-by-step tutorial for writing test hypotheses: starting from evidence, the problem statement, the proposed change, and the expected effect and metric.
Tutorialbeginner12 min readUpdated

Before you start

  • A research finding with a location and a size. A page or funnel step, a device or segment, and an estimate of how many visitors are affected. A vague concern is not enough; go back to the methods guide if that is all you have.
  • Access to the analytics and the recordings that produced the finding, so that the evidence can be quoted, not summarised from memory.
  • The current baseline for the metric you expect to move: the conversion rate (CVR) of the page or step over a normal period, by the segment the finding concerns.
  • A rough sense of traffic, because a hypothesis that predicts an effect smaller than the test could ever detect is not testable; the sample size and significance guide explains the relationship between baseline, effect and visitors needed.
  • Agreement on who reviews hypotheses before they enter the backlog. One reviewer who was not the author is enough.
  • A shared place to store them, such as the research backlog described in the conversion research hub, with the same fields for every entry so that the prioritisation method has something consistent to score.
  1. Gather the evidence

    Collect every piece of research that bears on the finding and write each down as an observation, with its source and its size. "The funnel exploration shows the delivery step loses a much larger share of mobile visitors than of desktop visitors over the last quarter." "In the batch of recordings reviewed, more than half of mobile sessions on that step show the address form reloading." "A third of poll responses on the step mention typing the address twice." Keep observations separate from interpretations: what the data showed, not what you think it means. Note what is missing too. If no attitudinal method has touched the finding, say so, because a hypothesis built on analytics alone is weaker and the reviewer should know. The evidence section is the part of the document that will matter most when the test loses, since it is where the team will look to decide whether the diagnosis or the fix was wrong.

  2. State the problem

    Write one sentence that describes what visitors fail to do, from the visitor's side, without mentioning any solution. "Mobile visitors who reach the delivery step cannot complete the address form in one attempt because the postcode lookup clears fields they have already filled." The test of a good problem statement is that several different changes could address it; if only one change fits, the statement is a solution in disguise. Resist the pull towards the fix at this stage. A team that writes "we need a single-page checkout" as the problem has skipped the diagnosis, and the hypothesis will inherit the skip. Problem statements also expose bugs: if the sentence describes something that is simply broken, stop here, raise it as a defect, and do not spend a test on it.

  3. Name the audience and the page

    Specify exactly who the hypothesis concerns and where. Audience: new or returning, device, traffic source, country, logged in or not, any segment the evidence identified. Page or step: the URL pattern or the funnel step, and whether the change would apply to every visitor to it or only the segment. Being precise here does two jobs. It decides who the experiment will include, which affects the traffic available and therefore whether the test is feasible. And it prevents a common failure in which a finding about mobile visitors from paid social is tested on all visitors to the page, the effect is diluted to nothing, and the hypothesis is wrongly recorded as disproved. If the evidence concerns one segment, the test should target that segment, or the prediction should state the diluted effect expected across everyone.

  4. Propose the change

    Describe the change in enough detail that a designer and a developer could build it without asking questions, and explain why it addresses the problem. The reasoning is the hypothesis's logic and it should be stated, not assumed: preserving entered fields through the postcode lookup removes the repeated entry that recordings show and poll responses describe. One change per hypothesis. A variant that alters the layout, the copy and the form at once can win or lose, but it cannot teach you which part mattered, and the next hypothesis starts from the same ignorance. Where the evidence suggests several possible fixes, write several hypotheses and let prioritisation choose between them. Include a sketch or a reference to a mock-up if one exists, and state what the control will be, which is normally the page as it stands today.

  5. Predict the effect

    State the direction and the rough size of the effect you expect, and why. Direction is non-negotiable: a hypothesis that would count either an increase or a decrease as a success is not a hypothesis. Size is harder, and the honest form is a range anchored to the evidence: if the step loses a given share of visitors to the problem and the change fixes it for most of them, the step's completion rate should rise by something like that share. Writing the prediction before the test is what makes the result informative. A win smaller than predicted says the diagnosis was partly right; a win larger says something else changed too; a loss says the fix, the diagnosis or the segment was wrong, and the evidence section says which. Keep the prediction in relative terms, and let the sample-size work in the testing section translate it into a required run length.

  6. Pick the primary metric and guardrails

    Choose one primary metric: the measure that decides whether the hypothesis held. It should be the closest reliable outcome to the change. For a delivery-step fix, that is the completion rate of the delivery step, or checkout completion if the step is short, rather than site-wide revenue, which the change affects only through a long chain of other steps. Then choose a few guardrails: metrics that must not get worse. Average order value, refund rate, support contacts, page speed and the completion rate of the step after the one being changed are common ones. Name them in advance so that a win on the primary metric cannot be quietly accepted while a guardrail fell. The business key performance indicator (KPI) the programme reports on, revenue or qualified leads, is the reason for the test, but it is rarely the right primary metric for a single experiment. The A/B testing guide covers how unit, metric and guardrails fit together in the test (A/B Test) itself. For pricing, offer and delivery changes that can move both conversion and basket size, the AOV versus revenue per visitor comparison explains which of the two should be primary.

  7. Define scope and risks

    Write down what the test will and will not touch, what could go wrong, and what would make you stop early. Scope covers pages, segments, devices and the share of traffic to include. Risks include technical ones (the change breaks on a browser, the test tool causes flicker on the control), measurement ones (the primary metric's event is not firing reliably on the step), and business ones (a change to delivery messaging has to be checked with the operations team, a claim in new copy needs legal sign-off). For a browser-side test, name the device and connection on which flicker will be checked before launch. Stopping rules are not the same as peeking: they are pre-agreed conditions, such as a sharp fall in a guardrail or an error rate on the variant, under which the test is halted for safety rather than for impatience. Listing them now means nobody has to invent them under pressure in week two.

  8. Write the hypothesis

    Assemble the pieces into the standard form: because we observed the evidence, we believe that the change, for the audience, on the page, will cause the effect on the primary metric, without harming the guardrails. For the running example: because the funnel exploration, recordings and poll responses show mobile visitors re-entering address fields after the postcode lookup, we believe that preserving entered fields through the lookup for mobile visitors on the delivery step will raise delivery-step completion by a noticeable share, without reducing average order value or raising delivery-address errors downstream. Read it aloud. If it needs a paragraph of context to make sense, the earlier steps are incomplete. If it contains two changes joined by "and", split it. If the effect has no direction, go back to the prediction step. A hypothesis that fits in two sentences and could be proved wrong is the goal.

  9. Record and review

    Enter the hypothesis in the shared backlog with every field filled: evidence, problem, audience and page, change, prediction, primary metric, guardrails, scope and risks, author and date. Then have someone who did not write it review it against four questions. Does the evidence support the problem? Does the change address the problem rather than something adjacent? Is the effect predicted with a direction and a plausible size? Could the test, given the traffic, detect that size in an acceptable time? A hypothesis that fails the last question is parked with a note, not deleted, because traffic and priorities change. Once approved, it waits for the prioritisation pass, and when the test runs, the result is written back against the same entry, including what was learned when the prediction was wrong. The entry is then the record a future team member reads before proposing the same test again. When the approved hypothesis becomes a built test, the A/B test launch checklist confirms the hypothesis, metric and guardrails are written down before launch.

Hypothesis template

The template below is the backlog entry in table form. Copy the fields exactly, so that every hypothesis can be compared with every other and scored by the same method.

FieldWhat goes in itExample from the tutorial
EvidenceObservations with source and size, no interpretationFunnel exploration, recordings and poll responses on the mobile delivery step
ProblemOne visitor-side sentence, no solutionMobile visitors cannot complete the address form in one attempt
Audience and pageSegment, device, source; URL or funnel stepMobile visitors on the delivery step of checkout
ChangeBuild-ready description and the reasoning behind itPreserve entered fields through the postcode lookup
Predicted effectDirection and a rough relative size, anchored to evidenceDelivery-step completion rises by a noticeable share
Primary metricOne measure that decides the resultDelivery-step completion rate
GuardrailsMeasures that must not worsenAverage order value, downstream address errors, page speed
Scope and risksWhat is included, what could go wrong, stopping rulesMobile only; lookup vendor change; halt if error rate rises
Status and resultDraft, reviewed, prioritised, testing, concluded; the outcomeReviewed; awaiting prioritisation

The template is deliberately plain. Teams that add a score column at this stage tend to score as they write, which lets enthusiasm leak into the evidence; keep scoring for the prioritisation pass, where every entry is judged against the others at the same time.

Illustrative example — not a real result

A completed hypothesis for a homeware checkout

A homeware retailer's backlog entry reads as follows. Evidence: the funnel exploration for the last 90 days shows the delivery step completing for 46% of mobile visitors against 71% on desktop, a gap of 25 points; of 40 mobile recordings on the step, 23 show the address form reloading after postcode lookup with earlier fields blank; 31 of 90 poll responses mention entering the address twice. Problem: mobile visitors cannot complete the address form in one attempt. Audience and page: mobile visitors, all sources, on the delivery step. Change: preserve all entered fields through the lookup and return focus to the next empty field. Predicted effect: if the bug accounts for most of the 25-point gap and the fix resolves it for most affected visitors, delivery-step completion on mobile should rise from 46% towards 60%, a relative lift of around 30%. Primary metric: delivery-step completion rate, mobile. Guardrails: average order value, address-correction tickets, delivery-step load time. Scope: mobile only, 50/50 split, all traffic. Stopping rule: halt if the variant's form error rate exceeds 5%. Sample: at the baseline and a minimum detectable relative lift of 10%, about 6,500 visitors per arm, or roughly five weeks at current mobile traffic.

For a complete hypothesis carried into a test design, the pricing page layout experiment shows how the prediction, metric and decision rule are set before launch.

Verify it worked

Check the finished entry against the steps before it goes to review.

  • Read the hypothesis aloud. It should fit in two sentences and name one change.
  • Read the evidence. Each observation should carry a source and a size, with no interpretation mixed in.
  • Read the problem statement. It should describe what visitors fail to do, and several different changes should fit it.
  • Check the prediction. It should give a direction and a range anchored to the evidence, as the homeware entry does with its stated start and target.
  • Check the metrics. The primary metric and guardrails are named before the test, and every template field is filled.

If it goes wrong

  • The hypothesis needs a paragraph of context. The earlier steps are incomplete, so return to the evidence and the problem statement and fill the gaps.
  • The sentence joins two changes with "and". The change is bundled, so split it into separate hypotheses, one change each.
  • Any movement in the metric would count as a win. The prediction has no direction, so go back to the prediction step and state one.
  • The problem sentence describes something simply broken. This is a defect, not a hypothesis, so raise it as a defect and do not spend a test on it.
  • The reviewer doubts the traffic could detect the predicted effect. Park the entry with a note rather than deleting it, because traffic and priorities change.

Next steps

Common mistakes

  • Solutions dressed as problems. "We need a progress bar" is a change, not a finding. Write the visitor's difficulty first.
  • No direction, no size. A prediction that any movement would confirm cannot be wrong, and so cannot teach.
  • Bundled changes. New copy, new layout and a new form in one variant produce a result nobody can act on.
  • Site-wide metrics for local changes. Revenue per visitor as the primary metric for a form-field change buries the effect in noise.
  • Guardrails named after the result. Choosing which secondary metrics count once the numbers are in is how losing tests become wins on paper.
  • Testing what the traffic cannot detect. Check feasibility before review, using the sample size and significance guide, and park what does not fit.
  • Skipping the review. The author is the worst judge of whether the evidence supports the change. One independent read catches most of the problems above.

Frequently asked questions

How many hypotheses should come out of one research sprint?

More than the team can test, which is normal and is why prioritisation exists as a separate step. A sprint that finds one problem at the biggest funnel step may produce several hypotheses for it, because the evidence usually supports more than one plausible fix, and each should be written separately so that the test isolates one change. What matters is that every hypothesis traces back to an entry in the backlog with its sources, not how many there are.

Can a hypothesis be written without quantitative data?

It can, and on low-traffic sites and new flows it often has to be, with usability sessions, interviews and support logs as the evidence. What is lost is the sizing: without analytics the prediction cannot be anchored to a measured loss, and the feasibility check cannot be done, so the change is more likely to be shipped on judgement than tested. Write the hypothesis anyway, mark the evidence as qualitative, and record the decision to ship without a test as a decision rather than an accident.

What happens to a hypothesis when the test loses?

It becomes a finding. Go back to the evidence section and decide which part failed: the diagnosis, the fix or the segment. A loss where the evidence was strong usually means the fix did not address the cause and a different change should be written for the same problem; a loss where the evidence was thin means the diagnosis was wrong and the problem goes back to research. Record the reasoning against the same backlog entry so that nobody proposes the identical test next year.

Terms used on this page

Where this page is used

Part of these learning paths

Also related

Related