Why intuition fails on test results
The human eye reads a difference between two numbers as information. Shown a variant converting slightly above the control after a few days, almost everyone concludes the variant is better; shown it slightly below, that it is worse. Both readings are usually premature, because the eye has no sense of how much two random samples from the same population differ on their own.
The variability is large at the sizes most tests reach early on. A conversion rate (CVRShare of visits or clicks that convert.) is a proportion, and a proportion measured on a few hundred visitors wanders by amounts that dwarf the improvements most changes produce. The gap between two arms in the first days of a test is mostly this wandering. As the samples grow the wandering shrinks, slowly — halving the noise takes several times the visitors, not twice as many — and the true difference, if there is one, emerges from under it.
Intuition fails in a second way: it treats a result as either real or not. Statistics offers something more useful and less comfortable — a statement about how surprising the data would be if there were no real difference, and a range within which the true difference plausibly lies. Learning to read those two statements, rather than a single headline rate, is most of what this guide is for. The procedure that surrounds them is in the A/B testing guide, and the experimentation hub sets out where statistics sits in the practice.
Baseline, effect size and sample size
Three quantities are decided before a test starts, and the third follows from the first two.
The baseline rate is the current conversion rate of the thing being tested, measured on the audience the test will include, over a period long enough to average out weekly cycles. It comes from analytics, and it has to be the rate for the test's unit of randomisationWhat a test assigns at random to control or variant, such as a visitor, session, account or region; it should match the level at which the metric is counted.: a per-visitor rate if visitors are assigned, a per-session rate if sessions are. The conversion rate calculator turns counts into the rate.
The minimum detectable effect is the smallest improvement the test is designed to find. It is a business decision dressed as a statistical one: how small a gain would still be worth shipping, maintaining and explaining? Smaller effects need far more visitors to detect, because the noise has to shrink below the effect. A team that chooses a large effect because it makes the test short will find that most real improvements are smaller than that, and the test will miss them while reporting "no significant difference".
Sample size is then the number of visitors per arm needed so that an effect of the chosen size, if it exists, would be detected with the agreed power at the agreed significance level. It depends on the baseline (rarer events need more visitors), on the effect size (smaller effects need many more), and on the two risk tolerances described in the next section. Calculators exist for the arithmetic; the judgement is in the inputs. The output — visitors per arm, and therefore days at the site's traffic — is what tells a team whether a test is feasible at all, which is why the prioritising tests guide treats detectable effect at available traffic as the hard constraint on a backlog.
Two consequences follow. Low-traffic pages cannot test small changes, however well reasoned, and should test bold ones or not at all. And a test's sample size is fixed before launch and then reached; it is not a target to be revised when the result looks promising.
Significance and power
A test can be wrong in two directions, and each has a name and a tolerance.
A false positive is declaring a difference when there is none: the variant is no better, but the samples happened to land apart. The significance level is the tolerance for this error — the share of tests with no real effect that would nonetheless be declared winners. The p-value a test reports is the probability of seeing a difference at least as large as the observed one if there were no real difference; the result is called statistically significant when the p-value falls below the agreed level. It is not the probability that the variant is better, and a result just over the line is not meaningfully different from one just under it.
A false negative is missing a difference that exists: the variant is better, but the test did not have enough visitors to show it. Power is the probability of detecting an effect of the chosen minimum size if it is real, and the complement of the tolerance for false negatives. Low power is the silent killer of testing programmes: tests that could never have detected a realistic effect are run, come back inconclusive, and are read as evidence that the change does not work.
| Concept | Plain-language meaning | Set by | What goes wrong if ignored |
|---|---|---|---|
| Significance level | How often you accept declaring a winner that is not one | Convention and risk appetite, before the test | Looser levels ship more false wins; the conventional level is a convention, not a law |
| p-value | How surprising the observed gap would be if nothing had changed | The data, after the test | Read as "probability the variant is better", which it is not |
| Power | How likely the test is to find an effect of the minimum size, if real | The sample size, given baseline and effect | Underpowered tests report "no difference" for changes that work |
| Minimum detectable effect | The smallest improvement the test is built to find | Business value of the change | Set too high to shorten the test; real but modest effects are missed |
| One-sided or two-sided | Whether the test asks "is B better" or "is B different" | The hypothesis, before the test | A one-sided test chosen after seeing the direction inflates false positives |
| Multiple comparisons | Several variants, metrics or segments tested at once | The design | Every extra comparison is another chance for noise to look like a result |
The significance level and power are chosen before the test, and most teams keep the conventional values rather than tuning them per test. What should vary is the minimum detectable effect, which is where the business judgement lives.
Confidence intervals
A p-value answers one question: is the difference surprising under no effect? A confidence intervalA range of values for an effect, such as a lift in conversion rate, that is consistent with the data at a chosen confidence level. answers the more useful one: given the data, what range of true effects is consistent with what was observed? It is reported as a lower and an upper bound around the observed difference, and three readings cover most situations.
If the whole interval lies above zero, the variant is better, and the interval says by how much at least and at most. If the whole interval lies below zero, the variant is worse. If the interval straddles zero, the test has not separated the effect from noise — and the width of the interval says whether that is because the effect is small or because the test was too small. A narrow interval around zero is evidence of no meaningful effect. A wide interval that spans both a loss and a large gain is evidence of nothing except that more data was needed.
The interval also disciplines the headline. The observed lift is the single most likely value, but it is not the result; the result is the range. A win reported as "between a small and a moderate improvement" is honest; the same win reported as the point estimate will disappoint when it is rolled out and the true effect turns out to be near the lower bound, which, for tests stopped the moment they became significant, it usually does. Decisions about whether to ship compare the interval with the minimum worth shipping, not the point estimate with zero. The experiment results report template puts the interval ahead of the point estimate for that reason.
Peeking and sequential testing
The classical test assumes the sample size was fixed in advance and the data was read once, at the end. Reading it repeatedly while the test runs, and stopping the first time the result looks significant, breaks that assumption in a way that is easy to underestimate. Each look is another opportunity for noise to cross the line; across many looks the chance that at least one crosses it rises far above the nominal level. Evan Miller's essay on how not to run an A/B test works through this and shows that a test peeked at frequently will declare winners among variants that do nothing at a rate several times the one the significance level promises. The "winners" then fail to deliver after rollout, and the programme's credibility goes with them.
The fix is procedural first: decide the sample, run to it, read once. Monitoring for defects — broken tracking, sample ratio problems, guardrail breaches — is not peeking, because no decision about the primary metric is made from it.
Where early stopping is genuinely needed, sequential methods exist that plan for it. They set stricter thresholds at each planned look so that the overall false-positive rate stays at the agreed level, or they use an always-valid approach that allows continuous monitoring at the cost of needing more visitors overall. Some testing tools implement one of these; what matters is to know which method the tool uses and to use it as designed, not to apply a fixed-sample reading to a test that was watched daily. Bayesian reporting, which gives a probability that the variant is better, is popular partly because its outputs read naturally; it is not immune to the same problem when the decision rule is "stop when the probability looks high", and it needs its own stopping ruleA rule set in advance that says when to stop: when a test has run long enough or must halt, or when a flexible budget line stops increasing.. The testing pitfalls guide treats peeking as the diagnosable fault it is. The same trap catches weekly reports read for anything unusual, and reading small numbers in marketing reports handles it outside tests with counts, ranges and pooled views.
When a test cannot answer the question
Some tests cannot succeed at any duration a business will tolerate, and the time to find out is before launch. The arithmetic of sample size is unforgiving on pages with low traffic or rare conversions: an improvement that would matter to the business may require more visitors than the page sees in a year. The honest responses are to test a bolder change whose effect would be larger, to test earlier in the journey where the event is more common and accept the weaker link to revenue, to measure something else — a usability study, a before-and-after with a long window and modest claims — or to not test and decide on judgement, saying so. What does not work is running the test anyway and reading noise. The same constraint governs channel-level experiments: the incrementality testing guide sizes holdout and geo designs with the same logic and runs into the same wall when the outcome is rare. The guide to conversion work without an A/B test turns each of those responses into a method, from testing a template across many pages to shipping a change with a written prediction and a reading date fixed in advance.
A software firm's demo-request test
A software company's pricing page receives about 9,000 visitors a month and converts 1.8% of them to demo requests, roughly 160 a month. The team wants to test a reworked pricing table and would ship it for a relative improvement of 15%, which would lift the rate to about 2.1%. At the conventional significance level and the customary power, detecting a change of that size needs on the order of 45,000 visitors per arm — about ten months of traffic split two ways. A test planned for one month could only reliably detect something like a 50% relative lift, far larger than any pricing-table change in the team's log has produced. They choose not to run it. Instead they run a usability study with eight prospects on the two layouts, make the change on judgement, and test the demo form itself on a later page that sees four times the traffic, where a 15% effect is detectable within six weeks.
When qualified leads are too few to power a test, the experimentation loop for lead generation shows how to validate a proxy and schedule a later pipeline readout.
To see what a planned minimum detectable effect means in enquiries or orders, enter the baseline and the effect in the conversion rate calculator set up for landing page tests.
For a test that has already run and still has no verdict, the diagnosis for an A/B test not reaching significance checks the plan's sample and effect, the size of the change, the metric, the number of variants and the page's traffic.
Common mistakes
- Sizing the test after it starts. Sample size is an input, not a conclusion.
- Choosing a large detectable effect to shorten the test. The test then misses the modest gains that most changes actually produce.
- Reading the p-value as the probability the variant is better. It is the probability of the data under no effect.
- Reporting the point estimate as the result. The interval is the result; the point estimate is its middle.
- Calling a wide, straddling interval "no effect". It is "could not tell".
- Peeking with a fixed-sample method. Either run to the planned sample or use a sequential method as designed.
- Switching to a one-sided test after seeing the direction. That is peeking by another name.
- Ignoring the base rate of true effects. Most changes do little; a programme with many tests and few real winners will see a meaningful share of its significant results turn out false, and the KPIA metric tied directly to an objective. after rollout is the only check.
