Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Glossary term

Unit of Randomisation

What a test assigns at random to control or variant, such as a visitor, session, account or region; it should match the level at which the metric is counted.
AnalyticsconceptUpdated

Definition

The unit of randomisation is the thing that gets assigned to a group in a test. The A/B testing guide puts it first among the three decisions that shape every test, ahead of the primary metric and the guardrails, and names the wrong unit among the procedural failures that sink tests before the statistics are reached. Each unit is assigned once, by the testing tool or the application, and stays in its group for the life of the test.

The unit is not the same as the audience. The audience is who is eligible to enter; the unit is the level at which entry is decided. Two tests on the same audience of shoppers can randomise people in one case and shopping sessions in the other, and they will not measure the same thing.

Choosing the unit: visitors, sessions, accounts, recipients, products, regions

  • Visitor or user. The usual choice on websites, identified by a cookie or a login so that a person sees the same version on every visit.
  • Session. Simpler to implement, but one person can see both versions on different visits, which matters whenever the outcome takes more than one visit.
  • Account. For products used by teams, so that colleagues share one experience.
  • Recipient, customer, quote or product. The library of experiment designs states the unit at the top of each design section, and the list runs well beyond visitors: email recipients for a send test, quotes for a pricing test, products for a merchandising test.
  • Region. Where individuals cannot be separated, such as a broadcast or local campaign, whole areas are assigned instead, with far fewer units to compare.

Larger units keep experiences consistent but give fewer units to count, so the sample takes longer to fill.

In practice: matching the unit to the metric and the baseline rate

The unit has to match the metric. A visitor-level metric, such as purchases per visitor, needs visitor-level assignment, or the calculation uses the wrong denominator. The same rule reaches back into planning: the sample size and significance guide requires the baseline rate to be the rate for the test's unit, per visitor if visitors are assigned and per session if sessions are. It reaches forward into reading the result too. The tutorial on analysing an A/B test in GA4 counts users, not sessions, when the tool assigns visitors, because a session-level denominator lets one person count several times.

The experiment hypothesis template gives the unit its own row, with who is included and excluded, so the choice is written down before anyone sees a number.

Illustrative example — not a real result

A meal kit company changes the unit and reruns

Ixbridle Meal Kits tests a shorter checkout and assigns by session, because its tool does that by default. The primary metric is orders per customer over 4 weeks. Most customers order weekly, so each makes around 4 visits in the test, and by the end about 70% of returning customers have seen both checkouts. The variant shows a lift of 3%, but nobody can say which version earned each order. The team reruns the test with the customer account as the unit, so each customer sees one checkout throughout, recalculates the sample size from the per-customer order rate, and adds a weekly comparison of group sizes. The rerun needs 6 weeks rather than 4 and shows no difference worth shipping.

An A/B Test is the commonest test the unit is chosen for. Sample Size is worked out in units, so a larger unit needs more time to reach the same sample. A Sample Ratio Mismatch, a persistent gap between the planned and the actual share of units in each group, often points to a problem in how units are assigned or counted. A Test Plan is where the unit is fixed before launch, beside the hypothesis, the primary metric and the analysis method.

Common confusions

  • The unit is whatever the tool defaults to. The default is a convenience. The unit should follow the metric and the experience being tested.
  • Assigning by session and reporting per customer. The counts no longer describe independent units, and the statistics are computed on the wrong denominator.
  • The unit and the audience are the same choice. The audience is who may enter; the unit is what is assigned.
  • A bigger unit is always safer. It protects consistency but shrinks the number of units, so the test needs more time or can detect only larger effects.

More terms to look up

Where this term is used

Related