Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Glossary term

Test Plan

The written design of a test, fixed before launch: hypothesis, unit of randomisation, primary metric, guardrails, planned sample, duration and decision rule.
AnalyticsconceptUpdated

Definition

A test plan is the written design of an experiment, fixed before launch. The A/B testing guide names three things that make a test a test rather than a comparison: random assignment, both versions running at the same time, and a plan written in advance, in which the metric, the sample and the decision rule are fixed before launch so that the result cannot be steered after the fact. The plan is the third of those.

It is a design, not a schedule. A project plan says when a test will be built and launched; the test plan says what the test is and how its result will be read. A test can be on time and still have no plan.

What a plan contains

The GA4 A/B test analysis tutorial lists what it expects to find in the plan before any analysis starts:

  • The hypothesis. The change, the effect expected on a named metric and the reason to expect it.
  • The unit of randomisation. What is assigned at random, such as a user, a session, a recipient or a region, which decides what the analysis must count.
  • The primary metric. The single measure the decision rests on.
  • Guardrail metrics. The measures the change must not worsen beyond an agreed bound.
  • The planned sample and duration. How much data the test needs to detect the effect that would matter, and for how many whole weeks it runs.
  • The analysis method. How the difference between arms will be judged against noise.
  • The decision rule. Which result leads to shipping, which to another round and which to abandoning the idea.

Experiment designs add whatever their question needs. The discount depth test owns a test plan with three arms, unique codes per arm, gross profit per recipient as the measure and full-price sales afterwards among its guardrails, all inside fair pricing rules.

In practice: analysis, shipping, experiment designs and programmes

The plan is read more often than it is written. The analysis tutorial says plainly that it implements the plan and does not write it. The tutorial on shipping a winning variant starts from the test plan and the results report, so that what ships is what was tested and the decision matches the rule set before launch. When a result looks wrong, the plan is the reference the guide to A/B test pitfalls checks against: visitors per arm against the planned sample, and how the test was stopped.

Programmes give plans to things that run for months. The lifecycle email programme playbook expects each live journey to have a test plan, testing things that could change behaviour, such as timing, offer, number of messages or content angle, with a holdout where volumes allow. The creative production playbook has channel managers launch approved assets according to the test plan, keeping asset names intact so results can be traced back to the concepts being compared.

Illustrative example — not a real result

A coffee subscription stops on a good day, then reruns from a plan

Hobzarnet Coffee tested a new pricing page with a monthly plan shown first. It had no written plan. The team checked results every morning and stopped the test on day 6, when the new page showed a 14% higher subscription rate. Rolled out to everyone, the subscription rate did not move. The team reran the test from a written plan: the user as the unit, subscription starts within 7 days as the primary metric, cancellations within 30 days as a guardrail, 9,000 visitors per arm and a fixed run of 4 whole weeks, read once at the end. The final difference was under 1%, inside the range the plan treated as no effect, and the current page stayed.

The Hypothesis is the plan's first line, and everything after it serves the test of that statement. The Decision Rule is the plan's last line, deciding in advance what each result leads to. A Guardrail Metric is the plan's protection against a change that wins on the primary metric while damaging something else. The Sample Size sets how much data the plan commits to before reading, and with it how long the test must run. The Unit of Randomisation sits between them, because the sample is counted in whatever unit is randomised.

Common confusions

  • A plan is a launch schedule. Dates and build tasks belong to the project. The test plan is the design and the rule for reading it.
  • The plan can be finished after the test starts. Any choice made after the data is visible can follow the data. The plan is fixed before launch, and a change to it is dated and explained.
  • Small tests do not need one. A short plan is still a plan. Without one, even a simple split can be stopped on a good day or read on whichever metric moved.
  • The testing tool's settings are the plan. A tool records the split and its own goal; the hypothesis, the guardrails, the sample, the duration and the decision rule are the team's to write down.
  • A plan guarantees a valid result. It prevents steering. Broken tracking, uneven splits and interacting tests can still spoil a test that was planned well, which is why the plan is checked again when the result is read.

More terms to look up

Where this term is used

Related