Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Glossary term

Confidence Interval

A range of values for an effect, such as a lift in conversion rate, that is consistent with the data at a chosen confidence level.
AnalyticsconceptUpdated

Definition

Any figure measured on a sample, a conversion rate, an average order value or the difference between two of them, would come out slightly differently if the test were run again on different visitors. A confidence interval puts bounds on that uncertainty. It is calculated from the observed value, the variability of the data and the sample size, at a confidence level chosen in advance. Repeated over many tests, intervals built this way contain the true value at the stated rate; for any single test, the interval is the set of effects the data does not rule out.

In survey and market research the same idea is usually reported as a margin of error: a proportion plus or minus a figure. The comparison of qualitative and quantitative research lists intervals among the outputs that distinguish quantitative work, because a number from a sample is only complete with its uncertainty attached.

Confidence interval versus p-value

The two come from the same calculation and answer different questions. A p-value asks: if the change had no effect at all, how surprising would a difference this large be? A small p-value says the difference is unlikely to be pure noise, and that is all it says. It does not say how big the effect is or whether it is worth having.

The interval answers the size question directly. An interval for the difference between variant and control that lies wholly above zero corresponds to a significant positive result; one that straddles zero corresponds to a non-significant one. But the interval also shows what the p-value hides: whether a significant result is plausibly large or plausibly trivial, and whether a non-significant result is a near-certain absence of effect or simply a test too small to tell. The guide to sample size and statistical significance covers how intervals are built and why they are reported ahead of the p-value.

In practice: reading width and whether it crosses zero

Three readings cover most decisions.

  • Wholly on one side of zero. The direction is established. The bound nearest zero is the most cautious estimate of the effect, and it is often the figure a forecast should use.
  • Straddling zero and wide. The test could not tell. The honest label is inconclusive, not "no effect", and the width shows how much larger the test would have needed to be.
  • Straddling zero and narrow. The effect, if any, is small. This is real evidence that the change does little, and it is a useful result.

Width is driven mainly by sample size and by how variable the metric is. Revenue per visitor, which swings with occasional large orders, gives wider intervals than a conversion rate on the same traffic. The experiment results report template asks for the interval beside every primary result for exactly this reason, and the email frequency test reads its revenue difference from the interval rather than the point estimate before checking guardrails.

Practical versus statistical significance

A result can be statistically significant and still not worth acting on. With enough traffic, an interval can sit wholly above zero while its lower end is a lift so small that the engineering and maintenance would cost more than it returns. The opposite also happens: an interval that crosses zero may still be mostly made up of worthwhile effects, which argues for a longer test rather than abandoning the idea.

The practical test is to compare the interval with the smallest effect the business decided in advance was worth shipping. The checkout test programme case study shows that reading in action, with one test whose interval justified shipping and another whose interval could not separate a gain from a loss. The experimentation section sets out how that minimum effect is fixed in the plan before any data arrives.

Illustrative example — not a real result

A mattress retailer stops booking the headline lift

Slumberley Mattress Co. tested a shorter checkout with about 9,000 checkout starts in each arm. Checkout completion was 41.0% in control and 43.6% in the variant, a lift of 2.6 points, and the test was declared a winner on a p-value below 0.05. The 95% interval on the difference ran from 0.2 to 5.0 points. The team shipped the change, because every value in the interval was positive and the change was cheap to maintain, but the finance forecast used the lower half of the range, about 1 point, rather than 2.6. Three months after rollout, completion had settled at 42.3%, inside the interval and well below the headline.

Statistical Significance is the yes-or-no reading of the same calculation, and the interval adds the size the significance test leaves out. Sample Size is the main lever on width: quadrupling the sample roughly halves it. An A/B Test reports its result most usefully as an interval on the difference between arms. A Novelty Effect can produce an interval that excludes zero during the early part of a test and then fades, which is why a narrow interval from a short run still needs the trend checked.

Common confusions

  • The observed lift is the result. It is the single most likely value inside a range; the range is the result.
  • An interval that crosses zero means no effect. A wide one means the test could not tell.
  • There is a fixed chance the true value lies in this particular interval. The confidence level describes how the method performs across many tests, not a probability attached to one interval.
  • Significant means important. Compare the interval with the smallest effect worth shipping.
  • A margin of error is something different. It is the half-width of a confidence interval, usually quoted for survey proportions.

More terms to look up

Where this term is used

Related