Definition
A decision rule states, in advance of a test, the result that leads to each of the possible actions. The experiment hypothesis template gives it its own row, beside the primary metric and the guardrails: what result leads to ship, iterate or abandon, written before the test starts. Its common shape is to ship if the primary metric improves by at least the smallest effect worth acting on with every guardrail intact, to iterate if the direction is right but the size is not, and to abandon if the result is flat or negative.
The same idea works outside page tests. In the tutorial on testing demand before launch the rule has go, change and stop lines for the commitment rate on a waitlist or deposit page, and every threshold is worked back from the business's own economics rather than borrowed from someone else's test.
What a usable decision rule states
A rule that only says "ship if it wins" decides nothing. A usable one names:
- The metric that decides. One primary metric, defined with its numerator, denominator and window, so that nobody can switch to a friendlier measure afterwards.
- The line for each action. The result that ships, the result that earns another round and the result that ends the idea, with the gap between them stated rather than left to judgement.
- The guardrails that can veto. A win on the primary metric that damages refunds or support contacts is not a win, and the rule says so before anyone sees the numbers.
- How uncertainty is read. A result whose plausible range straddles a line is evidence too; the rule says in advance whether that means a second test or a smaller step.
- Who makes the call. Ideally someone other than the person who built the test.
In practice: calling a finished test against its rule
The experimentation cadence guide puts the rule at the centre of the weekly read-out: each finished test is called win, loss or inconclusive against the rule written before it started, by someone other than its builder, and the call is logged the same day. An inconclusive result is recorded as inconclusive rather than recast as a narrow win.
The experiment results report makes this mechanical. Its decision field quotes the rule from the plan and then applies it. If the team dislikes the outcome, the report still records what the rule says, and any override is written separately with the name of the person who made it and the reason. That separation keeps a record of how often judgement overruled the plan, which is itself worth reading.
The rule also guards the step after the call. Before a winner is built for real, the guide to shipping a winning variant checks that the result meets the decision rule quoted from the plan, alongside the planned sample, the sample ratio, the tracking and the guardrails. A result that fails any of those checks is a lead for a new test, not a change to ship.
A tea merchant calls a subscription test against the rule it wrote first
Zuldeth Tea Merchants tested a subscribe-and-save option on 6 of its loose-leaf product pages. Before launch the plan recorded the rule: ship if subscription sign-ups per product page visitor rise by at least 15% relative with the refund rate no higher than its usual 3%; retest a different discount if the lift is positive but under 15%; abandon if flat or negative. After the planned 4 weeks and roughly 22,000 visitors, the variant showed a 9% lift with refunds steady. The founder, who had designed the offer, wanted to ship it. The analyst quoted the rule, recorded the result as iterate, and the next round tested a deeper first-order discount. That round cleared 15% and shipped.
Related terms
A decision rule completes a HypothesisA testable statement of the change to make, the effect expected on a named metric and the evidence-based reason for expecting it.: the hypothesis predicts the direction and the reason, and the rule says what each result will lead to. In an A/B TestControlled comparison of two variants. it is applied once the planned sample has been reached, and a Guardrail MetricA metric an experiment must not worsen beyond an agreed bound, watched beside the primary metric that decides the test. gives it the power to veto a win that costs too much elsewhere. The Stopping RuleA rule set in advance that says when to stop: when a test has run long enough or must halt, or when a flexible budget line stops increasing. is its companion: it fixes when the test ends or must halt, while the decision rule fixes what the result means once it has ended. How the test itself is designed and run is the subject of the A/B testing guide.
Common confusions
- A decision rule is the hypothesis. The hypothesis is the claim; the rule is the commitment about what to do with each result. A team can have a sharp hypothesis and still argue for weeks without a rule.
- A decision rule is the stopping rule. One says when to stop collecting data, the other what the data means. Stopping early when results look good is a stopping-rule failure, even if the decision rule is sound.
- The rule can be adjusted once the data is in. Afterwards it is a rationalisation. If the rule was wrong, record the override openly and write a better rule for the next test.
- Significance is the rule. A statistically significant result can still fall below the smallest effect worth shipping, or break a guardrail. Significance is one input the rule reads, not the rule itself.
