Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Glossary term

Hypothesis

A testable statement of the change to make, the effect expected on a named metric and the evidence-based reason for expecting it.
StrategyconceptUpdated

Definition

In experimentation, a hypothesis is a falsifiable statement linking a change to an expected effect, with a reason. The usual form reads: because we observed something, we believe that changing one thing for a given audience will move a primary metric in a given direction. Each clause does work. The observation ties the idea to evidence. The change and audience define what will be built and who will see it. The metric and direction define what the result has to show.

The word is sometimes used loosely for any test idea ("let's try a green button"). That is a proposal, not a hypothesis. It becomes one only when the expected effect and the reason for expecting it are stated.

The parts: observation, change, audience, metric, direction

  • Observation. What the research showed, with its source: a drop-off in a funnel report, a pattern in recordings, a recurring answer in a survey.
  • Change. One alteration, described well enough to build. Bundling several changes means a result cannot say which one mattered.
  • Audience. The visitors the evidence concerns, such as mobile users or new customers. Testing a finding about one segment on everyone dilutes the effect.
  • Metric. The single primary measure that decides the outcome, chosen to sit as close to the change as possible.
  • Direction. Up or down, ideally with a rough size. A prediction satisfied by any movement cannot fail.

Many teams add the measures that must not get worse. Naming a Guardrail Metric before launch stops a win on the primary metric being accepted while something more important fell. The experiment hypothesis template gathers these parts, with run length and decision rule, into one record.

In practice: from research evidence to hypothesis

A strong hypothesis starts from research rather than from a preference. Analytics locates where visitors fail; recordings and usability sessions show how; surveys and interviews explain why in the visitor's own words. The conversion research methods guide catalogues those methods, and a hypothesis is strongest when more than one of them points at the same problem. Evidence from a single source is still usable, but the record should say so.

The step from evidence to statement is where most errors creep in: a solution written as the problem, a site-wide metric chosen for a local change, a prediction with no direction. The hypothesis-writing tutorial walks through that step in detail, ending with an independent review before anything is built.

Hypotheses and prioritisation

A research programme produces more hypotheses than any team can test, so they queue. Prioritisation ranks them on expected impact, confidence in the evidence and effort, and it also asks whether the test is feasible at all. A hypothesis predicting a small effect on a low-traffic page may need a Sample Size the site cannot supply in a reasonable time. The guide to prioritising tests treats traffic and the smallest detectable effect as the hard constraint, and parks hypotheses that fail it rather than deleting them.

In the wider cycle, the hypothesis is the hinge between research and the test. The experimentation loop framework places it after research and before scoring, and the experimentation programme playbook makes the record of each hypothesis and its result, kept in a learning log, the main thing a programme produces.

Illustrative example — not a real result

A dog treat shop turns a preference into a test

Pawlicott Dog Treats tested a redesigned homepage because the team preferred it. It lost by a small margin, and the team blamed the season; nobody could say what had been learned. Next, the team ran a 1-question exit survey for 3 weeks and collected 412 answers, of which 157 named the delivery cost, shown only at checkout. The new hypothesis read: because 38% of surveyed leavers cited delivery cost, showing the delivery price and the £30 free-delivery threshold on product pages for all visitors will raise conversion rate, without lowering average order value. The test ran for 4 weeks on 21,000 visitors per arm. Conversion rose from 2.4% to 2.7%, and average order value held. Had it lost, the survey evidence would have told the team whether to doubt the diagnosis or the fix.

An A/B Test is the most common method for checking a hypothesis. A Guardrail Metric is named in the hypothesis to catch harm the primary metric cannot see. Sample Size decides whether the predicted effect can be detected with the traffic available, and Statistical Significance is the threshold the result is read against.

Common confusions

  • A test idea is not a hypothesis. Without a predicted effect and a reason, it cannot be wrong.
  • A hypothesis is not a guess dressed up. It rests on observed evidence, and the record names the source.
  • Losing does not mean the hypothesis was useless. A clear loss against a written prediction says which part of the reasoning failed.
  • The metric is not chosen after the results. Picking whichever number moved turns every test into a win on paper.
  • Statistical significance does not prove the reason. It shows the effect is unlikely to be chance; the observation behind the hypothesis is what explains it.

More terms to look up

Where this term is used

Related