Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Experiment Library

Email Send Time Test

A test design for email send timing: the hypothesis, clicks per delivered email as the metric, guardrails, recipient randomisation and reading results.
Experimentbeginner6 min readUpdated

Hypothesis

Sending the regular campaign when recipients have historically engaged will raise clicks per delivered email over the current fixed send time, because the message lands when people read. Behind the hypothesis is a simple observation about inboxes: an email that arrives hours before someone checks their mail sits under newer messages by the time they look. Arriving close to the moment a person usually reads should keep it near the top.

The hypothesis can fail in two ways, and both are useful. The engagement-based time may make no difference because your audience reads email in batches whatever time it arrives. Or it may help only for some segments, such as people who read on a phone in the evening. Write down which segments you expect to benefit before launch, using the segmentation guide to define them consistently, and record the hypothesis and the method for calculating each recipient's time in the experiment hypothesis template. The method matters: "most frequent click hour over recent months" is a different treatment from "most recent open hour".

Metrics and guardrails

The primary metric is click rate per delivered email, pooled across the campaigns in the test: unique clickers divided by deliveries in each arm. It is preferred to opens because privacy features record opens that did not happen, and those false opens are often timestamped at delivery, which would bias a timing test towards whatever time the email arrived. The email metrics and testing guide explains this distortion and why clicks are the steadier KPI.

Unsubscribe rate is the first guardrail; an email arriving at an odd hour, such as late at night, can feel intrusive. Conversions per delivered email is the second, and it matters more than it looks. A send time that produces more clicks from people browsing idly on a phone may produce fewer purchases than a time when people sit at a desk. If conversions fall while CTR rises, the timing is attracting attention without intent.

Design

Randomise at recipient level once, at the start, and keep each recipient in their arm for every campaign in the test. The control arm receives each campaign at the current fixed time. The test arm receives the same campaign at each recipient's engagement-based time, calculated by the agreed method. Everything else is identical: subject line, preview text, content, sender name and the day of sending.

Design elementChoiceReason
Unit of randomisationRecipient, assigned once for the whole testTiming effects build through habit
VariantsFixed send time versus engagement-based send timeOne timing change
Held constantSubject line, content, sender, send dayOnly the hour differs
Recipients without historyFixed time in both arms, reported separatelyAvoids testing a guess
Time zonesEngagement time computed in each recipient's local timePrevents mis-timed sends
DurationSeveral regular campaignsSeparates timing from topic

Schedule all sends within the same day so that the test arm's emails do not drift into the following day for some recipients. A send that moves to a different day is testing day of week as well as time.

Sample size reasoning

Timing effects, where they exist, tend to be modest, and modest effects need large audiences to detect. Start from the click rate per delivered email you usually see, decide the smallest improvement that would justify the operational effort of per-recipient timing, and work out the audience needed using the sample size and significance guide. The conversion rate calculator turns your clicks and deliveries into the baseline rate.

Because recipients stay in their arm, each campaign adds a fresh round of observations from the same people. Pooling helps, but the observations are not fully independent: a heavy clicker clicks in every campaign. Treat the number of recipients, not the number of sends, as your sample, and plan enough campaigns to see whether the effect is stable rather than to inflate the count.

Reading the results

When the planned campaigns have all been sent and the read window after the last one has closed, compare pooled clicks per delivered email between arms. Then look at each campaign in turn. A real timing effect should point the same way in most campaigns; a pooled win driven by one campaign is more likely to reflect that campaign's topic.

Check conversions per delivered email next. If clicks rose and conversions held or rose, the new timing is a win. If clicks rose but conversions fell, the engagement-based time may be catching people in a browsing moment, not a buying one; consider a time based on purchase history instead of clicks. Finally, segment by device and by the recipient's engagement level. Many teams find that engaged readers open whenever the email arrives and the gain, if any, comes from occasional readers.

Illustrative example — not a real result

A recipe subscription tests engagement-based timing

Pantry Club, an invented recipe box company, sends a weekly menu email at 9am to about 40,000 customers. It randomises customers into two arms and, for eight consecutive weeks, sends the test arm the same email at each customer's most frequent click hour over the past three months. Pooled clicks per delivered email are 5.8% in the test arm against 5.3% for the fixed time, and the test arm wins in six of the eight weeks. Menu changes per delivered email, the conversion that matters, rise slightly. Unsubscribes are unchanged. The company adopts engagement-based timing for this email only.

Pitfalls

Time zones are the first trap. If engagement times are computed in the sender's time zone, recipients abroad receive emails at the wrong local hour, and the test arm is penalised for a data error. Compute and send in local time, or exclude recipients whose time zone is unknown.

One-off results are the second. Teams that run one campaign, see a lift and switch everyone to the new timing often find the gain vanishes, because the lift belonged to that campaign's topic or that day's news. Plan several campaigns from the start. The third pitfall is content changing between sends in a way that interacts with timing: a breakfast recipe email may suit morning sends and a weekend offer may suit evenings. Keep the campaign type steady during the test, and if your campaigns vary a lot, read each type separately rather than pooling everything.

Common mistakes

Send time tests are cheap to set up and easy to misread, because timing effects are modest and campaigns vary. Settle the subject line question first with the email subject line test, and browse the experiment library for designs beyond email.

  • Judging on opens. Machine-recorded opens cluster at delivery time and bias timing tests.
  • Reassigning recipients every send. People need to experience one schedule consistently.
  • Ignoring time zones. An engagement hour in the wrong zone is the wrong hour.
  • Letting sends drift into the next day. That tests day of week as well as hour.
  • Stopping after one campaign. One topic and one day can swamp a modest timing effect.
  • Dropping conversions from the read. More clicks at a browsing hour can mean fewer purchases.

Frequently asked questions

Why keep recipients in the same arm for every campaign?

Send time effects build through habit, so a recipient who flips between times never experiences either schedule properly. Keeping each person in one arm across campaigns also lets you see whether the effect holds or fades.

What if many recipients have no engagement history?

Give them the fixed time in both arms, or exclude them from the test and report them separately. Guessing a time for people with no history adds noise and tests the guess, not the idea.

Can one campaign settle the question?

Rarely. One campaign carries one topic, one day and one set of news events, any of which can swamp a timing effect. Plan for several campaigns and read the pooled result alongside the campaign-by-campaign direction.

Terms used on this page

Tools to try

Frameworks

Where this page is used

Also related

Related