Definition
The unit of randomisation is the thing that gets assigned to a group in a test. The A/B testing guide puts it first among the three decisions that shape every test, ahead of the primary metric and the guardrails, and names the wrong unit among the procedural failures that sink tests before the statistics are reached. Each unit is assigned once, by the testing tool or the application, and stays in its group for the life of the test.
The unit is not the same as the audience. The audience is who is eligible to enter; the unit is the level at which entry is decided. Two tests on the same audience of shoppers can randomise people in one case and shopping sessions in the other, and they will not measure the same thing.
Choosing the unit: visitors, sessions, accounts, recipients, products, regions
- Visitor or user. The usual choice on websites, identified by a cookie or a login so that a person sees the same version on every visit.
- Session. Simpler to implement, but one person can see both versions on different visits, which matters whenever the outcome takes more than one visit.
- Account. For products used by teams, so that colleagues share one experience.
- Recipient, customer, quote or product. The library of experiment designs states the unit at the top of each design section, and the list runs well beyond visitors: email recipients for a send test, quotes for a pricing test, products for a merchandising test.
- Region. Where individuals cannot be separated, such as a broadcast or local campaign, whole areas are assigned instead, with far fewer units to compare.
Larger units keep experiences consistent but give fewer units to count, so the sample takes longer to fill.
In practice: matching the unit to the metric and the baseline rate
The unit has to match the metric. A visitor-level metric, such as purchases per visitor, needs visitor-level assignment, or the calculation uses the wrong denominator. The same rule reaches back into planning: the sample size and significance guide requires the baseline rate to be the rate for the test's unit, per visitor if visitors are assigned and per session if sessions are. It reaches forward into reading the result too. The tutorial on analysing an A/B test in GA4 counts users, not sessions, when the tool assigns visitors, because a session-level denominator lets one person count several times.
The experiment hypothesis template gives the unit its own row, with who is included and excluded, so the choice is written down before anyone sees a number.
A meal kit company changes the unit and reruns
Ixbridle Meal Kits tests a shorter checkout and assigns by session, because its tool does that by default. The primary metric is orders per customer over 4 weeks. Most customers order weekly, so each makes around 4 visits in the test, and by the end about 70% of returning customers have seen both checkouts. The variant shows a lift of 3%, but nobody can say which version earned each order. The team reruns the test with the customer account as the unit, so each customer sees one checkout throughout, recalculates the sample size from the per-customer order rate, and adds a weekly comparison of group sizes. The rerun needs 6 weeks rather than 4 and shows no difference worth shipping.
Related terms
An A/B TestControlled comparison of two variants. is the commonest test the unit is chosen for. Sample SizeThe number of users or sessions an experiment needs to detect a given effect reliably. is worked out in units, so a larger unit needs more time to reach the same sample. A Sample Ratio MismatchA persistent gap between the share of visitors an A/B test was meant to assign to each variant and the share each actually received., a persistent gap between the planned and the actual share of units in each group, often points to a problem in how units are assigned or counted. A Test PlanThe written design of a test, fixed before launch: hypothesis, unit of randomisation, primary metric, guardrails, planned sample, duration and decision rule. is where the unit is fixed before launch, beside the hypothesis, the primary metric and the analysis method.
Common confusions
- The unit is whatever the tool defaults to. The default is a convenience. The unit should follow the metric and the experience being tested.
- Assigning by session and reporting per customer. The counts no longer describe independent units, and the statistics are computed on the wrong denominator.
- The unit and the audience are the same choice. The audience is who may enter; the unit is what is assigned.
- A bigger unit is always safer. It protects consistency but shrinks the number of units, so the test needs more time or can detect only larger effects.
