Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Glossary term

Sample Size

The number of users or sessions an experiment needs to detect a given effect reliably.
AnalyticsconceptUpdated

Definition

In an A/B Test two groups see different versions of a page or message and their outcomes are compared. Because each group is a sample, its measured rate wobbles from one week to the next even when nothing has changed. The sample size is the number of observations per group needed for a real difference of a stated size to stand out from that wobble with acceptable confidence.

Four inputs set it:

  • Baseline rate. The current CVR or other rate of the metric being tested. Rare outcomes need larger samples than common ones.
  • Minimum detectable effect. The smallest change the test is designed to notice, usually stated relative to the baseline. Shrink it and the required sample grows far faster than the effect shrinks.
  • Significance level. The tolerance for a false positive: declaring a winner when the versions are really the same.
  • Power. The probability of detecting the effect if it is real; low power produces confident reports of "no difference" for changes that work.

The guide to sample size and significance explains each input, confidence intervals and the conventions most teams use, so this page stays with what the term means and how it is misused.

In practice

Sample size is the reason testing feels effortless on a busy checkout and frustrating on a niche site. With little traffic or a low baseline, the visitors needed to detect a modest improvement may take months to arrive. Teams in that position have honest options: test bigger, bolder changes that would produce a larger effect; test higher in the funnel where the metric is more frequent; or accept that some decisions will be made on judgement and research instead. The conversion rate calculator for landing page tests helps establish the baseline that every sample calculation starts from, and the experiment hypothesis template gives the sample and run time a field of their own so they are written down before the build.

What the minimum detectable effect implies

The minimum detectable effect is a business decision dressed as a statistical one. It should be the smallest improvement that would justify shipping and maintaining the change. Choosing a tiny one feels rigorous but makes many tests impossible to finish; choosing a huge one lets tests end quickly but means any realistic, modest gain is likely to be missed and reported as a failure. A test that ends with "no significant difference" has only shown that the effect, if any, was probably smaller than the minimum it was designed to see.

Peeking and early stopping

Checking results repeatedly and stopping the first time a version looks significant inflates the false positive rate well beyond the level the team believes it is using. Evan Miller's essay on how not to run an A/B test sets out why: random fluctuation guarantees that an unremarkable test will cross the line at some point if it is watched often enough. The remedy is to fix the sample in advance and read the result once it is reached, or to use a sequential method designed for repeated looks. The experimentation programme playbook builds that rule into governance so nobody stops a test early on enthusiasm.

Illustrative example — not a real result

A fencing club calls its test too early

Vesper Fencing Club tests a new trial-class page against its old one. After a weekend the new page shows 14 bookings from 210 visitors against 8 from 205, and the club declares a winner. Its baseline booking rate is 4%, and the change it hoped for was a lift to 5%. At conventional significance and power, detecting that lift needs roughly 6,700 visitors per version, which at its traffic is about nine weeks. The club restarts the test with that sample fixed. When it is reached the new page converts at 4.3% and the old at 4.1%, a difference well inside the noise, so the club keeps the old page and moves on to testing the booking form.

An A/B Test is the experiment whose sample is being sized. A Holdout Group is sized by the same logic, since the withheld group must be large enough for its outcome to be compared with the exposed group's. CVR is the most common baseline, and Incrementality studies face the sample problem in its hardest form, because the lift they look for is often small. The testing hub places sample size between hypothesis design and reading results, and the ecommerce checkout test case study shows tests run to a sample registered before launch.

Common confusions

  • Sample size is not a duration. Time follows from traffic; the requirement is a count of observations. Full weekly cycles still matter so that weekday and weekend visitors are both represented.
  • Significance is not size. A significant result can be a trivially small lift, and a large observed lift in a small sample can be noise.
  • "No difference" is not "no effect". An underpowered test cannot rule out a change smaller than its minimum detectable effect.
  • More variants need more traffic. Every extra version splits the sample further and adds comparisons that raise the chance of a false winner.
  • Peeking is not monitoring. Watching for broken tracking is wise; deciding on the result before the planned sample is reached is not.

More terms to look up

Calculators

Where this term is used

Sources

  1. How Not To Run an A/B Test — Evan Miller

Related