Hypothesis
Sending onboarding emails triggered by what a new user has not yet done, instead of on a fixed schedule, will raise the share of users who activate, because each message arrives when it is relevant. A fixed schedule sends the same message on the same day to everyone, so a user who has already set up their account still receives the set-up email, and a user who is stuck on the first step may not hear about it until days later. A behaviour-triggered flow sends the set-up nudge only to those who have not set up, at the point they stall.
The test compares two whole flows, which is legitimate here because the unit of change is the cadence logic rather than a single message. Keep the content of equivalent messages as close as possible between arms, so that the difference is when and to whom, not what. The welcome series guide covers what each message should do, and the nurture sequences guide covers trigger and branching logic. Record both flows, message by message, in the experiment hypothesis template. The tutorial on building onboarding emails for a software product builds the behaviour-triggered arm, from the definition of activation to the exits.
Metrics and guardrails
The primary metric is activation rateThe share of new users or sign-ups who complete the activation event, the action marking first real value, within a set window.: the share of users in each arm who complete the activation action within the cohort window. Define the action from product data before launch and agree the window, which should be long enough for most users who will ever activate to do so. Reading activation instead of email engagement keeps the test focused on the product outcome. It also sidesteps the open-count distortion that privacy features cause.
Unsubscribe rate is the first guardrail. A triggered flow may send more emails to users who are stuck, and if that feels like nagging, unsubscribes will rise and the business loses the channel for later lifecycle messages. Spam complaint rate is the second and more serious guardrail, because complaints affect delivery for every email the business sends. Report emails sent per user in each arm as context. Later, compare retention and LTVTotal value a customer brings over time. by arm, since activation is only valuable if it leads to lasting use.
Design
Randomise each new user at sign-up, before the first onboarding email, and keep them in their arm for the whole cohort window. The control arm receives the current fixed schedule. The test arm receives the behaviour-triggered flow. Both arms receive any transactional messages, such as password resets, in the same way.
| Design element | Choice | Reason |
|---|---|---|
| Unit of randomisation | User, at sign-up | Onboarding happens once and must start in one arm |
| Variants | Fixed-schedule flow versus behaviour-triggered flow | Tests cadence logic |
| Content | Equivalent messages kept as similar as possible | Isolates timing and targeting |
| Cohort window | Agreed period from sign-up | Gives users time to activate |
| Enrolment period | Whole weeks of sign-ups | Balances weekday and weekend sign-ups |
| Read point | After the last enrolled user's window closes | Every user observed for the full window |
There are two periods. During enrolment, new sign-ups enter the test. After enrolment ends, the test continues until the last user enrolled has completed the cohort window. Only then is every user observed for the same length of time.
Sample size reasoning
The sample is the number of new users enrolled, and it is limited by your sign-up rate. Start from the current activation rate, decide the smallest improvement that would justify building and maintaining triggered logic, and work out the users needed per arm with the sample size and significance guide. The conversion rate calculator gives the baseline activation rate from your own counts.
Turn the required users into an enrolment period using your weekly sign-ups, then add the cohort window to find the total time to a result. That total is often longer than teams expect, and it is worth sharing at the start so nobody presses for an early read when the first cohorts look promising.
Reading the results
Once the last cohort window closes, compare activation rates between arms. Then check the guardrails and the emails-per-user figure. A higher activation rate with steady unsubscribes and complaints is a clear win for the triggered flow.
Look at when activation happened as well as whether. A triggered flow may speed activation without changing its final level, which is still worth having because earlier value tends to support retention; the retention and payback guide explains why the timing of value matters to payback. If activation is flat but the triggered arm sends fewer emails, you have the same outcome with less noise, which is a modest win for the user experience. If unsubscribes rose sharply, review the trigger rules: users stuck at one step may be receiving too many nudges. Check the result for each sign-up source, because users from different channels arrive at different points in the FunnelModel of the customer journey stages..
A budgeting app tests triggered onboarding
Coinwise, an invented budgeting app, sends a five-email onboarding sequence on fixed days. It builds a triggered flow that sends the "connect your bank" email only to users who have not connected an account after a day, and so on for each step. Over four weeks, about 8,000 new users are randomised. With a fourteen-day cohort window, 38% of the triggered arm activate against 33% on the fixed schedule. Emails per user fall, and unsubscribes are slightly lower. Coinwise adopts the triggered flow and plans to compare ninety-day retention by arm.
Pitfalls
Cohort timing is the first pitfall. If the test is read while recent sign-ups are still inside their window, those users have had less time to activate, and whichever arm enrolled more users late looks worse. Wait until every enrolled user has completed the full window.
Product changes mid-test are the second. A new onboarding screen, a pricing change or a bug fix in the middle of the test affects both arms, but not necessarily equally, because the triggered flow reacts to behaviour that the product change alters. Freeze onboarding-related product changes during enrolment or note them and read before and after separately. The third pitfall is counting emails rather than outcomes: reporting opens, clicks or sends per arm as the result misses the point of the test. Those numbers explain the result; activation is the result.
For the wider practice behind this design, the experimentation loop for retention covers persistent holdouts, cohort-age readouts, novelty effectsA temporary change in behaviour caused by a change being new, which fades as users get used to it. and testing whole programmes.
Common mistakes
Cadence tests run for a long time, so mistakes made at the start are discovered late and cost a full cycle. For regular campaigns rather than onboarding, the email send time test is the closer design, and the experiment library covers other channels.
- Reading on opens or clicks. The test is about activation, not email engagement.
- Splitting existing users. Onboarding happens once; randomise new users at sign-up.
- Reading before cohort windows close. Recent users have not had time to activate.
- Changing message content between arms. Keep equivalent messages alike so cadence is the variable.
- Ignoring unsubscribes. A flow that activates more users but loses the channel can cost more later.
- Shipping product changes mid-test. They interact with triggers and confuse the comparison.
