Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Glossary term

Statistical Significance

A test result unlikely enough under 'no real difference' to pass a threshold chosen in advance.
AnalyticsconceptUpdated

Definition: null hypothesis, p-value and significance level

Three ideas fit together.

  • The null hypothesis is the assumption that there is no real difference: the variant converts at the same underlying rate as the control, and any gap in the sample is chance.
  • The p-value is the probability of seeing a difference at least as large as the observed one if the null hypothesis were true. A small p-value means the result would be unusual under "no difference".
  • The significance level is the threshold the p-value must fall below for the result to be declared significant. It is the tolerance for false positives: the share of tests with no real effect that would nonetheless be declared winners.

So a significant result means the p-value fell below the agreed level. The sample size and significance guide places these alongside power and confidence intervals, which complete the picture.

In practice: setting the threshold before the test

The threshold is a decision about risk and belongs in the test plan, written down before any data arrives, together with the primary metric, the sample needed and the planned duration. Most teams keep the conventional level rather than tuning it per test. Choosing it afterwards, or switching from a two-sided to a one-sided test once the direction of the result is known, makes the threshold mean nothing.

Significance is computed from counts, not from rates on a dashboard. A conversion rate calculator set up for landing page tests describes each arm and values the effect being hunted, but it does not test significance; that comes from the testing method. The A/B testing guide covers the plan that the threshold sits in.

What significance does not tell you

Size. A tiny, commercially meaningless difference becomes significant with enough traffic. The effect size, and the confidence interval around it, says how big the difference is likely to be.

Value. Whether the difference pays for the change, the maintenance and the risk is a business judgement the p-value cannot make.

Probability that the variant is better. The p-value is computed assuming no difference; it is not the chance that the variant wins. A result just past the line is not meaningfully different from one just short of it.

Generality. A result holds for the population, period and conditions tested. A different season, traffic mix or audience may behave differently, which is why research outside testing, covered in the marketing research methods hub, still matters.

Peeking and multiple comparisons

The classical calculation assumes the sample size was fixed in advance and the result read once at the end. Evan Miller's essay on how not to run an A/B test shows what happens otherwise: a test checked repeatedly and stopped the first time it looks significant declares winners among changes that do nothing far more often than the significance level promises. Each look is another chance for noise to cross the line. The fix is procedural: decide the sample, run to it, read once, or use a sequential method designed for early looks.

Multiple comparisons inflate false positives the same way. Testing several variants, several metrics or many segments gives noise many chances to look like a result, so the primary comparison is named in advance and the rest are treated as leads for the next test. The testing section and the site's experiments library apply these rules to specific tests.

Illustrative example — not a real result

A checkout test stopped on its best day

Pekoeloft Tea Merchants tested a one-page checkout against its three-step checkout. The team checked the results every morning, and on day nine the variant showed 2.6% conversion against 2.2% for the control, a p-value of 0.04 on about 12,000 visitors per arm, so they stopped the test and called it a win. A month after launch, overall conversion had not moved. They reran the test with a sample of 24,000 visitors per arm fixed in advance and a single read at the end. The variant converted at 2.31% and the control at 2.27%, a p-value of 0.77: no significant difference. The early "win" had been noise caught on a good day. The tea merchant kept the one-page checkout because customers preferred it, but stopped reporting it as a conversion gain.

An A/B Test is the most common setting for a significance test in marketing. Sample Size is planned so that a test has a fair chance of reaching significance for an effect worth finding. A Multivariate Test compares many combinations at once, which multiplies the comparisons and the risk of false positives. A Holdout Group applies the same logic to campaigns, comparing exposed and unexposed groups.

Common confusions

  • Significant means important. It means unlikely under no difference; the effect may be trivial.
  • The p-value is the chance the variant is better. It is computed assuming there is no difference.
  • Not significant means no effect. It may mean the test was too small to detect one.
  • Checking daily is harmless. Stopping on a good day inflates false positives.
  • More metrics give more insight. Each extra comparison is another chance for noise to win.

More terms to look up

Calculators

Where this term is used

Sources

  1. How Not To Run an A/B Test — Evan Miller

Related