Definition
Every experiment is decided by one primary metric, chosen because it measures the effect the change is meant to have. That focus is deliberate, and it is also a blind spot. A change that pushes the primary metric up can push something else down: a discount prompt lifts conversion and lowers margin, a shorter form lifts leads and lowers their quality, a heavier page lifts engagement and slows loading for everyone.
A guardrail metric is the safeguard against that blind spot. It is a measure the test does not aim to move, with a stated limit on how far it may move in the wrong direction. The variant may leave it flat or improve it; what it may not do is worsen it past the limit and still be shipped as if nothing happened.
Primary metric versus guardrails
The two play different roles in the decision. The primary metric is tested for an improvement, with the sample sized to detect the smallest lift worth having. A guardrail is tested for harm, and the question asked of it is narrower: is there evidence it got worse by more than the bound? A test can have one primary metric and several guardrails, and adding guardrails does not change what the test is trying to prove.
The A/B testing guide lists typical guardrails beside the primary metric when the test is designed and puts them second in the reading order, after the primary result and before any segments. The same split exists outside experiments. The KPI tree framework uses guardrails for metrics that sit outside a tree's arithmetic and limit how its drivers may be pushed, which is the same idea applied to ongoing targets rather than a single test.
In practice: choosing guardrails and setting bounds
Good guardrails come from asking what the change could plausibly break, not from a fixed list.
- Money that the primary metric ignores. Revenue or margin per visitor when the primary metric is conversion; order value when testing a delivery threshold.
- Downstream outcomes. Refunds, cancellations, lead qualification or repeat purchase when the test is judged on a first action.
- Audience health. Unsubscribes and spam complaints when testing email frequency or content.
- Experience and technical quality. Page speed, error rates and support contacts when the change adds code or complexity.
Each guardrail then needs a bound: the size of deterioration the business will not accept, such as a stated drop in margin per visitor or a rise in unsubscribes per send. The bound is a business judgement, often set at the level where a past change was regretted. Writing it down before launch matters because a guardrail chosen after the data is in becomes an excuse to ship or to stop. The experiment hypothesis template gives guardrails and their bounds their own field, and the A/B test launch checklist will not let a test start until they are written.
What to do when a guardrail is breached
A breach does not automatically kill the variant, and a held guardrail does not automatically ship it. A breach means the result is no longer a simple win: the business must weigh the gain on the primary metric against the cost on the guardrail and decide whether to ship, redesign or abandon. Often the useful outcome is a redesign that keeps what worked and removes what did the damage.
Two reading points apply. A guardrail is measured with the same noise as any metric, so a small adverse move inside its interval is not proof of harm. And guardrails read late, such as refunds or cancellations, may need the test or a holdout to run longer than the primary metric alone requires. The experiment results report template records each guardrail with its bound, its observed value in each arm and whether it held, so the decision is visible to anyone reading later, and the guide to experimentation cadence keeps guardrail results in the experiment log beside the primary result.
A pet supplies shop tests a pop-up discount
Wagbrook Pet Supplies tested a pop-up offering 15% off a first order against no pop-up, with conversion rate as the primary metric and margin per visitor as the guardrail, bounded at a fall of no more than 3%. Over four weeks and about 52,000 visitors, conversion rose from 2.4% to 2.7%, a clear win on the primary metric. Margin per visitor fell by 6%, beyond the bound: most buyers who used the code had items already in their basket and would probably have ordered anyway. The variant was not shipped. A redesigned pop-up, shown only to visitors leaving with an empty basket, was tested next and lifted conversion to 2.5% with margin per visitor flat.
Related terms
An A/B TestControlled comparison of two variants. is the usual setting for guardrails, which sit beside its primary metric throughout the run. A KPIA metric tied directly to an objective. can carry guardrails too, limiting how a target may be chased outside any single experiment. Statistical SignificanceA test result unlikely enough under 'no real difference' to pass a threshold chosen in advance. applies to guardrails as to any metric, so a breach is judged against noise rather than on a raw difference. The HypothesisA testable statement of the change to make, the effect expected on a named metric and the evidence-based reason for expecting it. written before a test names the primary metric and is the right place to name its guardrails and their bounds as well.
Common confusions
- A guardrail is a second primary metric. It is tested for harm against a bound, not for improvement.
- Guardrails can be picked after the result. Chosen late, they become a way to rationalise a decision.
- A breach means the test failed. It means the win has a cost the business must weigh, and often a redesign follows.
- More guardrails are safer. Each one adds a chance of a false alarm; choose the few the change could plausibly harm.
- Flat guardrails prove there was no harm. They show no harm beyond the bound that the test could detect.
