Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Experiment Library

Onboarding Email Cadence Test

A test design comparing behaviour-triggered onboarding emails with a fixed schedule: activation as the metric, unsubscribe guardrails and cohort design.
Experimentintermediate6 min readUpdated

Hypothesis

Sending onboarding emails triggered by what a new user has not yet done, instead of on a fixed schedule, will raise the share of users who activate, because each message arrives when it is relevant. A fixed schedule sends the same message on the same day to everyone, so a user who has already set up their account still receives the set-up email, and a user who is stuck on the first step may not hear about it until days later. A behaviour-triggered flow sends the set-up nudge only to those who have not set up, at the point they stall.

The test compares two whole flows, which is legitimate here because the unit of change is the cadence logic rather than a single message. Keep the content of equivalent messages as close as possible between arms, so that the difference is when and to whom, not what. The welcome series guide covers what each message should do, and the nurture sequences guide covers trigger and branching logic. Record both flows, message by message, in the experiment hypothesis template. The tutorial on building onboarding emails for a software product builds the behaviour-triggered arm, from the definition of activation to the exits.

Metrics and guardrails

The primary metric is activation rate: the share of users in each arm who complete the activation action within the cohort window. Define the action from product data before launch and agree the window, which should be long enough for most users who will ever activate to do so. Reading activation instead of email engagement keeps the test focused on the product outcome. It also sidesteps the open-count distortion that privacy features cause.

Unsubscribe rate is the first guardrail. A triggered flow may send more emails to users who are stuck, and if that feels like nagging, unsubscribes will rise and the business loses the channel for later lifecycle messages. Spam complaint rate is the second and more serious guardrail, because complaints affect delivery for every email the business sends. Report emails sent per user in each arm as context. Later, compare retention and LTV by arm, since activation is only valuable if it leads to lasting use.

Design

Randomise each new user at sign-up, before the first onboarding email, and keep them in their arm for the whole cohort window. The control arm receives the current fixed schedule. The test arm receives the behaviour-triggered flow. Both arms receive any transactional messages, such as password resets, in the same way.

Design elementChoiceReason
Unit of randomisationUser, at sign-upOnboarding happens once and must start in one arm
VariantsFixed-schedule flow versus behaviour-triggered flowTests cadence logic
ContentEquivalent messages kept as similar as possibleIsolates timing and targeting
Cohort windowAgreed period from sign-upGives users time to activate
Enrolment periodWhole weeks of sign-upsBalances weekday and weekend sign-ups
Read pointAfter the last enrolled user's window closesEvery user observed for the full window

There are two periods. During enrolment, new sign-ups enter the test. After enrolment ends, the test continues until the last user enrolled has completed the cohort window. Only then is every user observed for the same length of time.

Sample size reasoning

The sample is the number of new users enrolled, and it is limited by your sign-up rate. Start from the current activation rate, decide the smallest improvement that would justify building and maintaining triggered logic, and work out the users needed per arm with the sample size and significance guide. The conversion rate calculator gives the baseline activation rate from your own counts.

Turn the required users into an enrolment period using your weekly sign-ups, then add the cohort window to find the total time to a result. That total is often longer than teams expect, and it is worth sharing at the start so nobody presses for an early read when the first cohorts look promising.

Reading the results

Once the last cohort window closes, compare activation rates between arms. Then check the guardrails and the emails-per-user figure. A higher activation rate with steady unsubscribes and complaints is a clear win for the triggered flow.

Look at when activation happened as well as whether. A triggered flow may speed activation without changing its final level, which is still worth having because earlier value tends to support retention; the retention and payback guide explains why the timing of value matters to payback. If activation is flat but the triggered arm sends fewer emails, you have the same outcome with less noise, which is a modest win for the user experience. If unsubscribes rose sharply, review the trigger rules: users stuck at one step may be receiving too many nudges. Check the result for each sign-up source, because users from different channels arrive at different points in the Funnel.

Illustrative example — not a real result

A budgeting app tests triggered onboarding

Coinwise, an invented budgeting app, sends a five-email onboarding sequence on fixed days. It builds a triggered flow that sends the "connect your bank" email only to users who have not connected an account after a day, and so on for each step. Over four weeks, about 8,000 new users are randomised. With a fourteen-day cohort window, 38% of the triggered arm activate against 33% on the fixed schedule. Emails per user fall, and unsubscribes are slightly lower. Coinwise adopts the triggered flow and plans to compare ninety-day retention by arm.

Pitfalls

Cohort timing is the first pitfall. If the test is read while recent sign-ups are still inside their window, those users have had less time to activate, and whichever arm enrolled more users late looks worse. Wait until every enrolled user has completed the full window.

Product changes mid-test are the second. A new onboarding screen, a pricing change or a bug fix in the middle of the test affects both arms, but not necessarily equally, because the triggered flow reacts to behaviour that the product change alters. Freeze onboarding-related product changes during enrolment or note them and read before and after separately. The third pitfall is counting emails rather than outcomes: reporting opens, clicks or sends per arm as the result misses the point of the test. Those numbers explain the result; activation is the result.

For the wider practice behind this design, the experimentation loop for retention covers persistent holdouts, cohort-age readouts, novelty effects and testing whole programmes.

Common mistakes

Cadence tests run for a long time, so mistakes made at the start are discovered late and cost a full cycle. For regular campaigns rather than onboarding, the email send time test is the closer design, and the experiment library covers other channels.

  • Reading on opens or clicks. The test is about activation, not email engagement.
  • Splitting existing users. Onboarding happens once; randomise new users at sign-up.
  • Reading before cohort windows close. Recent users have not had time to activate.
  • Changing message content between arms. Keep equivalent messages alike so cadence is the variable.
  • Ignoring unsubscribes. A flow that activates more users but loses the channel can cost more later.
  • Shipping product changes mid-test. They interact with triggers and confuse the comparison.

Frequently asked questions

Why randomise at sign-up rather than splitting existing users?

Onboarding only happens once per user, at the start. Randomising at sign-up means each new user experiences one flow from the beginning, and the two groups are comparable because they joined over the same period.

What counts as activation?

Activation is the first action that shows a user has received the product's core value, such as creating a first project or completing a first order. Define it before launch, from product data, and use the same definition for both arms.

Does the behaviour-triggered flow need different consent?

Onboarding emails are often service messages, but the line between service and marketing content varies. Guidance such as the ICO's direct marketing guidance describes the distinction in general terms; check how it applies to your messages with someone qualified to advise.

Terms used on this page

Tools to try

Frameworks

Where this page is used

Also related

Sources

  1. Direct marketing guidance — Information Commissioner's Office (UK)

Related