What this template is for
Use the template for any change that will be tested against a control: an A/B TestControlled comparison of two variants. on a page, a pricing or packaging change, an email variant, a new onboarding step, a bid strategy. The fields are the same whatever the channel because the logic is the same: an observation, a cause, a change, a predicted effect, a way of measuring it and a rule for deciding. A hypothesis document written for every test also produces, over time, a learning log that is the main asset of an experimentation programme, since each entry says what was believed, what was tried and what happened.
The template is the artefact at the hypothesis step of the experimentation loop; prioritisation scores can sit in it or in the backlog. How the test is run — the unit of randomisationWhat a test assigns at random to control or variant, such as a visitor, session, account or region; it should match the level at which the metric is counted., allocation, QA, running to the planned sample — is covered by the A/B testing guide, and the arithmetic behind the sample and duration fields by the sample size and significance guide. Where the document sits in a team's weekly rhythm, and how many of them a team can honestly run, is the subject of the experimentation cadence guide.
The template
One document per test. The rows are in the order the reasoning runs, from what was seen to what will be decided, and a row left blank is a reason not to start.
| Field | What to write |
|---|---|
| Name and ID | A short name and the backlog ID, so the test can be referred to in reports and the learning log |
| Evidence | What was observed and where: analytics, session recordings, survey answers, support tickets, past tests. Quote the data, not the interpretation |
| Problem statement | The specific thing that is going wrong for a specific group of people, inferred from the evidence; one or two sentences |
| Proposed change | Exactly what the variant does differently from the control; a sketch or screenshot link if the change is visual |
| Hypothesis | "If we [change], then [metric] will [direction] for [segment], because [mechanism]" |
| Expected effect | The direction and the smallest size of effect worth acting on (the minimum detectable effect) |
| Primary metric | The one metric that decides the test, defined precisely: numerator, denominator, window |
| Guardrail metrics | Metrics that must not worsen beyond a stated bound, and what happens if they do |
| Unit and segment | Who is randomised (user, session, account), who is included, and who is excluded |
| Sample and duration | The sample size each arm needs for the expected effect, the traffic available, and the resulting duration in whole weeks |
| Decision rule | What result leads to ship, iterate or abandon, written before the test starts |
| Risks and dependencies | What could invalidate the test or harm users, and what has to exist for it to run (tracking, engineering, legal) |
| Owner and status | Who owns the test, the planned start, and the status (proposed, approved, running, concluded) |
How to fill each section
Evidence and problem
Begin with what was seen, quoted as data: a drop-off between two steps, a recurring phrase in survey answers, a support ticket theme, a segment that behaves differently. The problem statement is the inference from that evidence — what is going wrong, for whom — and it is a separate field so that the inference can be challenged without arguing about the data. A hypothesis with no evidence field is a preference, and preferences are tested last.
Change and hypothesis
Describe the change precisely enough that an engineer could build it and a reviewer could recognise it in the variant. The hypothesis sentence links change, metric, direction, segment and mechanism; the mechanism is the part people skip, and it is the part that lets the team learn something even when the result is flat. If the mechanism is "it looks better", the test has nothing to teach.
Expected effect, metric and guardrails
The expected effect is the smallest change worth acting on, not the change you hope for; it drives the sample sizeThe number of users or sessions an experiment needs to detect a given effect reliably., so an honest figure here saves weeks. The primary metric is one metric, defined with its numerator, denominator and window — CVRShare of visits or clicks that convert. from landing to purchase within the session, say, rather than "conversions". Guardrails protect what the primary metric does not see: revenue per visitorRevenue divided by the number of visitors in the same period. when the test lifts order count, support contacts when it removes a step, page speed when it adds content. The business KPIA metric tied directly to an objective. the test ultimately serves is worth naming here even when it is too slow to be the primary metric.
Unit, sample and duration
Choose the unit of randomisation so that one person sees one variant consistently, and state who is excluded (internal traffic, existing customers, bot traffic). Sample size per arm follows from the baseline rate and the expected effect; duration follows from the sample and the eligible traffic, rounded up to whole weeks so that weekday and weekend behaviour are both covered. A test whose duration comes out at several months is telling you the effect is too small to detect with this traffic, and this field is where that is discovered.
Decision rule and risks
Write the rule before the test starts, and tie it to the interval around the result rather than the point estimate alone: ship if the interval for the improvement clears zero, the estimate reaches the stated floor (normally the smallest effect worth acting on) and the guardrails are intact; iterate if the direction is right but the interval is too wide to be sure, meaning it includes zero and still allows an effect of the size worth acting on; abandon if the interval rules out an effect of that size, or if the variant is worse. A flat result whose interval still includes the smallest effect worth acting on does not disprove an effect: it says the test was too small to tell, and the rule says what happens next. A flat result whose interval rules that size out narrows the cause, since an effect of the planned size is unlikely, though a smaller one is not excluded. If the interval clears zero but the estimate falls short of the floor, do not ship, and log the size so a later decision can weigh it; if a guardrail fails, the variant does not ship whatever the primary metric did. Risks cover both validity (a sale during the test, a tracking change) and users (a variant that could mislead or harm). Dependencies say what must exist before the start date, because the usual reason a test slips is tracking that was assumed.
Filled example
The document below is completed for an invented software business so that a runnable test plan is visible from end to end; its figures are illustrative.
Ledgerline: pricing page plan comparison test
| Field | Ledgerline's hypothesis document |
|---|---|
| Name and ID | Pricing page: plan comparison above the fold — backlog item EXP-114 |
| Evidence | Over the last 90 days, 46% of pricing page visitors scroll past the three plan cards without clicking any plan; session recordings (n = 60) show 38 visitors scrolling up and down between the cards, and 11 of those expanding the "compare all features" accordion. Exit survey answers (n = 212) include "not sure which plan I need" or similar in 27% of responses. Pricing-to-trial conversion is 4.1% overall and 2.6% for visitors arriving from the "freelance invoicing software" search campaign |
| Problem statement | First-time visitors cannot tell which of the three plans fits a sole trader, and the comparison that would tell them is hidden in an accordion below the cards |
| Proposed change | Replace the accordion with an always-visible comparison table directly under the plan cards, with a "best for" row at the top (sole trader / small team / agency) and the two differentiating features (recurring invoices, multi-currency) highlighted. Control: the current page with the accordion |
| Hypothesis | If we show the plan comparison without a click, then pricing-to-trial conversion will increase for new visitors to the pricing page, because they can match their situation to a plan without leaving the page or guessing |
| Expected effect | Relative increase of at least 12% in pricing-to-trial conversion (from 4.1% to about 4.6%); anything smaller is not worth the maintenance of a longer page, so 12% is also the floor in the decision rule |
| Primary metric | Trial starts divided by unique pricing page visitors, within the same session, attributed to the variant seen |
| Guardrail metrics | Share of trials on the lowest plan must not rise by more than 5 points (the fear: the table pushes everyone to the cheapest option); pricing page load time must stay under 2.0 s on mobile; support tickets tagged "which plan" must not increase |
| Unit and segment | Randomised by user (first-party cookie), 50/50. Included: all new visitors to /pricing. Excluded: logged-in users, internal IP ranges, traffic from the partner referral programme (different pricing) |
| Sample and duration | Baseline 4.1% (the overall rate, used as the estimate for new visitors), minimum detectable relative effect 12%, power 80%, significance 5%: about 27,000 visitors per arm. Eligible traffic is 29,000 visitors a week, so two full weeks, extended to three to cover one month-end when invoicing traffic peaks; planned 23 March – 12 April 2026. The search-campaign segment (2.6% baseline) would need about 43,000 visitors per arm on its own, so it is read afterwards as an unpowered secondary cut and does not decide the test |
| Decision rule | Ship if the 95% confidence interval for the lift has a lower bound above zero, the estimate is at least 12% relative (the floor in the Expected effect row) and the guardrails hold. If the lower bound is above zero but the estimate is below 12%, the lift is real but smaller than the longer page is worth: do not ship, and log the size. Iterate (test a shorter table) if the interval includes zero but its upper bound is still 12% or more, because a flat result like that does not disprove an effect of the size worth acting on. Abandon if the interval includes zero and its upper bound is below 12%, or if the variant is worse; the log then says that an effect of the planned size is unlikely, not that visibility of the comparison is never a barrier |
| Risks and dependencies | Risk: the spring promotion planned for April would change the plan prices mid-test; agreed with marketing to start it on 13 April. Dependencies: the experiment flag in the page framework; the trial_start event firing with the variant parameter, verified in the tracking QA on 20 March |
| Owner and status | Owner: growth product manager. Analyst: marketing analytics lead. Status: approved, build in sprint 14, start 23 March |
Common mistakes
- Starting from the change. A redesign in search of a justification produces a hypothesis written backwards. Start with the evidence and let the change follow from the problem.
- No mechanism. "Because it looks cleaner" cannot be wrong, so it cannot teach. Say why the change would alter behaviour, and a flat result becomes a finding.
- The hoped-for effect as the expected effect. An optimistic effect size shrinks the sample and the duration, and the test ends underpowered. Use the smallest effect worth acting on.
- Metrics named but not defined. "Conversions" means something different in each tool. Write the numerator, the denominator and the window.
- Guardrails added after the result. A guardrail chosen once the data is in is an excuse. List them with the hypothesis.
- Deciding the rule after seeing the data. The decision ruleA rule written before a test starts that says which result ships the change, which leads to another round and which abandons it. exists to be written before the first visitor is assigned; afterwards it is a rationalisation.
- Stopping at the first significant day. Peeking and stopping early inflates false positives. Run to the planned sample and duration, whatever the interim read says.
- Flat results left unrecorded. A test with no clear effect still narrowed the possible causes, and its interval shows which effect sizes are ruled out. It goes into the learning log with the same care as a winner.
The templates library holds the monthly report in which concluded tests and their decisions are recorded.
