Hypothesis
Sending the regular campaign when recipients have historically engaged will raise clicks per delivered email over the current fixed send time, because the message lands when people read. Behind the hypothesis is a simple observation about inboxes: an email that arrives hours before someone checks their mail sits under newer messages by the time they look. Arriving close to the moment a person usually reads should keep it near the top.
The hypothesis can fail in two ways, and both are useful. The engagement-based time may make no difference because your audience reads email in batches whatever time it arrives. Or it may help only for some segments, such as people who read on a phone in the evening. Write down which segments you expect to benefit before launch, using the segmentation guide to define them consistently, and record the hypothesis and the method for calculating each recipient's time in the experiment hypothesis template. The method matters: "most frequent click hour over recent months" is a different treatment from "most recent open hour".
Metrics and guardrails
The primary metric is click rate per delivered email, pooled across the campaigns in the test: unique clickers divided by deliveries in each arm. It is preferred to opens because privacy features record opens that did not happen, and those false opens are often timestamped at delivery, which would bias a timing test towards whatever time the email arrived. The email metrics and testing guide explains this distortion and why clicks are the steadier KPIA metric tied directly to an objective..
Unsubscribe rate is the first guardrail; an email arriving at an odd hour, such as late at night, can feel intrusive. Conversions per delivered email is the second, and it matters more than it looks. A send time that produces more clicks from people browsing idly on a phone may produce fewer purchases than a time when people sit at a desk. If conversions fall while CTRShare of impressions that result in a click. rises, the timing is attracting attention without intent.
Design
Randomise at recipient level once, at the start, and keep each recipient in their arm for every campaign in the test. The control arm receives each campaign at the current fixed time. The test arm receives the same campaign at each recipient's engagement-based time, calculated by the agreed method. Everything else is identical: subject line, preview text, content, sender name and the day of sending.
| Design element | Choice | Reason |
|---|---|---|
| Unit of randomisation | Recipient, assigned once for the whole test | Timing effects build through habit |
| Variants | Fixed send time versus engagement-based send time | One timing change |
| Held constant | Subject line, content, sender, send day | Only the hour differs |
| Recipients without history | Fixed time in both arms, reported separately | Avoids testing a guess |
| Time zones | Engagement time computed in each recipient's local time | Prevents mis-timed sends |
| Duration | Several regular campaigns | Separates timing from topic |
Schedule all sends within the same day so that the test arm's emails do not drift into the following day for some recipients. A send that moves to a different day is testing day of week as well as time.
Sample size reasoning
Timing effects, where they exist, tend to be modest, and modest effects need large audiences to detect. Start from the click rate per delivered email you usually see, decide the smallest improvement that would justify the operational effort of per-recipient timing, and work out the audience needed using the sample size and significance guide. The conversion rate calculator turns your clicks and deliveries into the baseline rate.
Because recipients stay in their arm, each campaign adds a fresh round of observations from the same people. Pooling helps, but the observations are not fully independent: a heavy clicker clicks in every campaign. Treat the number of recipients, not the number of sends, as your sample, and plan enough campaigns to see whether the effect is stable rather than to inflate the count.
Reading the results
When the planned campaigns have all been sent and the read window after the last one has closed, compare pooled clicks per delivered email between arms. Then look at each campaign in turn. A real timing effect should point the same way in most campaigns; a pooled win driven by one campaign is more likely to reflect that campaign's topic.
Check conversions per delivered email next. If clicks rose and conversions held or rose, the new timing is a win. If clicks rose but conversions fell, the engagement-based time may be catching people in a browsing moment, not a buying one; consider a time based on purchase history instead of clicks. Finally, segment by device and by the recipient's engagement level. Many teams find that engaged readers open whenever the email arrives and the gain, if any, comes from occasional readers.
A recipe subscription tests engagement-based timing
Pantry Club, an invented recipe box company, sends a weekly menu email at 9am to about 40,000 customers. It randomises customers into two arms and, for eight consecutive weeks, sends the test arm the same email at each customer's most frequent click hour over the past three months. Pooled clicks per delivered email are 5.8% in the test arm against 5.3% for the fixed time, and the test arm wins in six of the eight weeks. Menu changes per delivered email, the conversion that matters, rise slightly. Unsubscribes are unchanged. The company adopts engagement-based timing for this email only.
Pitfalls
Time zones are the first trap. If engagement times are computed in the sender's time zone, recipients abroad receive emails at the wrong local hour, and the test arm is penalised for a data error. Compute and send in local time, or exclude recipients whose time zone is unknown.
One-off results are the second. Teams that run one campaign, see a lift and switch everyone to the new timing often find the gain vanishes, because the lift belonged to that campaign's topic or that day's news. Plan several campaigns from the start. The third pitfall is content changing between sends in a way that interacts with timing: a breakfast recipe email may suit morning sends and a weekend offer may suit evenings. Keep the campaign type steady during the test, and if your campaigns vary a lot, read each type separately rather than pooling everything.
Common mistakes
Send time tests are cheap to set up and easy to misread, because timing effects are modest and campaigns vary. Settle the subject line question first with the email subject line test, and browse the experiment library for designs beyond email.
- Judging on opens. Machine-recorded opens cluster at delivery time and bias timing tests.
- Reassigning recipients every send. People need to experience one schedule consistently.
- Ignoring time zones. An engagement hour in the wrong zone is the wrong hour.
- Letting sends drift into the next day. That tests day of week as well as hour.
- Stopping after one campaign. One topic and one day can swamp a modest timing effect.
- Dropping conversions from the read. More clicks at a browsing hour can mean fewer purchases.
