Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Glossary term

Stopping Rule

A rule set in advance that says when to stop: when a test has run long enough or must halt, or when a flexible budget line stops increasing.
AnalyticsconceptUpdated

Definition

A stopping rule is a condition, agreed in advance, that ends an activity whose results arrive gradually. It answers "when do we stop?" without reference to whether the results so far are pleasing. That independence is the whole point. A rule invented while the data is visible is not a rule at all; it is a decision dressed up as one.

The term turns up in two places on this site, and the two senses share a logic. In experimentation, the rule governs when a test ends or halts. In a target-led paid media budget, it governs when a flexible spending line stops increasing. In both, the rule exists so that the stop is triggered by something planned rather than by whoever is watching the dashboard.

Stopping rules in tests: duration, sample and halt conditions

For a classical test, the rule is usually a duration or a sample: run until the planned number of visitors has been reached, covering whole business cycles, then read once. The guide to sample size and significance explains why. The usual significance test assumes the data is read a single time at the end; checking repeatedly and stopping the first time the result crosses the line gives noise many chances to look like a win, so the real false-positive rate climbs well above the one the team thinks it is running.

Two kinds of early stop are legitimate. The first is a halt for safety: the guide to writing hypotheses asks for pre-agreed conditions, such as a sharp fall in a guardrail or an error appearing in the variant, under which the test stops to protect the business. Watching for broken tracking or a lopsided traffic split is monitoring, not peeking, because no conclusion about the primary metric is drawn from it. The second is a method designed for early looks. Sequential approaches plan their looks and set stricter thresholds at each, and Bayesian reporting still needs a rule of its own; what matters is following the method the tool actually uses.

The A/B test launch checklist treats a missing rule as a reason to hold the launch, and the experimentation programme playbook fixes the duration and the rule at design, then reviews live tests for health only.

In practice: stopping rules in budgets and the marginal cost rule

The second sense belongs to budgets that are allowed to follow results. The tutorial on agreeing a target-led paid media budget writes the stopping rule as a marginal cost rule: a flexible line may keep spending while the estimated cost of the extra results it buys is at or below the target, and it stops increasing when that is no longer true. The estimate comes from budget steps in the business's own data, which are noisy, so the rule asks for a sustained reading rather than one week, and it says what happens on a breach: the line returns to the last level at which the marginal cost met the target. The rule decides how much is spent between a ceiling and a floor that finance sets separately.

Illustrative example — not a real result

A bicycle shop lets its checkout test run its full course

Nirkoft Bicycles tested a shorter checkout against its existing one. The design, written before launch, set a sample of 24,000 visitors per variant, a duration of 4 full weeks and one halt condition: stop if payment errors in the variant rose above 1%. After 5 days the variant showed a 9% lift in completed orders, and the owner asked to switch everyone over before the weekend. The analyst pointed to the rule. The test ran on, payment errors never approached the halt line, and at the end of week 4 the lift had shrunk to 0.6%, with an interval comfortably spanning zero. The shop kept its existing checkout and spent the build time on delivery messaging instead.

The stopping rule sits beside the Decision Rule: one says when the test ends, the other says what each possible result will lead to, and both are written before launch. The planned Sample Size is often the stopping rule's main content in a fixed-sample A/B Test, and reading Statistical Significance only once, at that point, is what keeps its error rate honest.

Common confusions

  • Stopping when the result is significant is a stopping rule. It is the habit the rule exists to prevent. Significance reached on an early look is far more likely to be noise than the stated level suggests.
  • Any early stop breaks the rule. A halt for a safety condition named in advance is part of the rule. What breaks it is stopping because the primary metric looks good or bad.
  • The stopping rule and the decision rule are the same thing. One fixes when to read the result; the other fixes what to do with it.
  • A budget stopping rule caps spend. It decides where a flexible line stops increasing; the ceiling that caps total spend is a separate limit set from cash and payback.
  • Bayesian tools do not need a stopping rule. They report results differently, but a team that stops as soon as the probability looks high inflates its errors in the same way.

More terms to look up

Where this term is used

Related