Definition
A cool-down period is a piece of the test calendar that sits between two moments: the end of the treatment and the final reading. The treatment is whatever the test does to the treated group, such as running ads there, pausing them or excluding a customer list. When it ends, the counting does not. Both groups, treated and held back, keep being recorded, and only when the stretch is over is the outcome compared with what the plan said to compare it against.
The experiments and the checklist call it a cool-down. This page uses the longer name from the incrementality guide.
Why it exists: late orders and lasting effects
Two things make a reading on the day the treatment stops misleading.
Orders still on their way. Someone who saw an ad late in the test may not order until afterwards. If the reading is taken at the moment the treatment ends, those orders have not been counted in either group. The cool-down gives the slower orders room to arrive, in both groups.
Effects that outlast the treatment. An effect may not switch off with the treatment. The brand search holdout brings the ads back in this stretch and watches the regions for any lasting effect.
The cool-down is separate from the length of the test itself. The guides ask for a full buying cycle and a cool-down, and neither stands in for the other.
In practice: incrementality tests, regional holdouts, exclusion tests and the readiness checklist
- Incrementality tests. The incrementality testing guide places it in its advice on duration, after the full buying cycle and the treatment, and before the reading. That guide owns test design, cells, size and duration; this page only names the stretch.
- Regional holdouts. The guide to measuring offline media points to the incrementality guide's regional design, which covers a full purchase cycle and then a cool-down, for media that no platform can split person by person.
- Pausing brand search. The brand search holdout test pauses the ads for a full buying cycle, then keeps a short cool-down with the ads back, and reads the test once, after both.
- Excluding existing customers. The existing customer exclusion test leaves a short cool-down after the exclusion before it reads repeat orders, since an order that followed an advert seen near the end may not have arrived yet.
- Before the start. The holdout test readiness checklist lists the cool-down among the things fixed in advance with the test period, the outcome and the reading date.
A sailmaker reads its pause after the late orders arrive
Ganutras Sailmakers, an invented maker of custom sails, pauses its ads in 5 of its 15 regions for 9 weeks, about one buying cycle for its customers, and leaves the other 10 untouched. The plan sets a 3-week cool-down with the ads back on and both groups of regions still counted, and fixes the reading for the end of it. A look at the end of week 9 shows orders in the paused regions 14% below the prediction from the matched regions, and one colleague wants to report that as the cost of switching the ads off. During the cool-down, orders still on their way arrive in both groups and the ads are back in the paused regions. Read on the planned date, the shortfall is 6%. The team reports that figure, notes that the 3 weeks were chosen before the test began, and keeps the week-9 figure in the log as an early reading.
Related terms
A Holdout GroupA randomly chosen part of an audience kept from exposure to measure what exposure caused. is the part of the audience kept from the treatment; the cool-down keeps counting it, alongside the treated group, until the reading. IncrementalityConversions caused by marketing. is the effect the test is trying to isolate, and reading it too soon can mis-state it. A Geo ExperimentA test that runs a campaign in some regions and holds it back in comparable ones, then compares total outcomes, where people cannot be randomised one by one. uses areas as the unit rather than individuals, and its glossary entry lists a cool-down beside the full cycle. A BaselineAn estimate of what a metric would have done without a given change, campaign or promotion, against which the effect is measured. is the estimate the actual figure is read against; the cool-down changes when that reading is made, not what the baseline is. Conversion LagThe time between an ad interaction and the conversion it leads to, which leaves the most recent reporting days incomplete. is the delay between an interaction and its conversion, which is the reason the stretch is needed.
Common confusions
- A cool-down is a cooling-off period for customers. It is a stretch of measurement. No customer is being paused, and a pause between sends is another thing.
- It is the platform settling after a change. A bidding strategy finding its feet is a different question; this stretch is about counting a test's orders.
- A long enough test makes it unnecessary. The guides name both a full buying cycle and a cool-down.
- It guarantees every late order is in. The guides do not say so; its length is a decision, fixed before the start, and a reading is still read as a range.
- It is more time to look. The reading is planned for once, after the treatment and the cool-down; stopping when the number looks good is a separate habit the guides warn against.
