Hypothesis
If demo requesters can book a slot on a calendar immediately after the form instead of waiting for a sales callback, more requests will turn into held demos, because the commitment is made while intent is highest. The decision behind it: let demo requesters book a slot immediately after the form, or have sales call them back to arrange it.
The callback has strengths the calendar lacks. A rep can qualify the requester before committing time, steer an unsuitable prospect elsewhere and build some rapport before the demo. The calendar removes a round of phone or email tag and lets the requester commit at the moment they asked. Which effect is larger depends on the buyers, the sales team's speedAverage time between a new lead arriving and the first contact from sales. and how many unsuitable companies request demos. The inbound lead handling playbook describes how demo requests should be handled in general; this test settles the specific choice of mechanism. Record the arms, the contact rules for each and the guardrail limits in the experiment hypothesis template before launch.
Metrics and guardrails
The primary metric is demos held per demo request: of all requests assigned to an arm, the share that results in a demo that actually took place within the follow-up window. Bookings per request and the confirmation step's CVRShare of visits or clicks that convert. are reported as context, but they are what the calendar arm would naturally win, so they do not decide the test. The denominator is every request, including those the callback arm never reached.
Three guardrails catch the ways instant booking can disappoint. The no-show rate among booked demos shows how many easy bookings were never meant seriously; a rise is expected, and the question is whether held demos still gain. The sales-qualified rate of held demos shows whether the calendar fills sales time with weaker prospects. Bookings from companies outside the target profile, judged against the profile sales already uses, catch the unsuitable demos a callback would have screened out. Agree with sales, through the shared definitions described in the sales and marketing alignment guide, which records count as held and qualified before launch.
Design
Randomise at the level of the demo requester, at form submission. In the calendar arm, the confirmation step shows a booking calendar with the next available slots and a clear CTAThe prompt that tells a reader the one action to take next, such as a button or link. to confirm one; an automated confirmation and reminder follow. In the callback arm, the confirmation step says sales will be in touch, and the standard callback process runs. The form, its fields and its wording stay identical. The forms guide covers form design; usability research such as the Nielsen Norman Group's recommendations for web forms is a useful check that the form itself is not the bottleneck before this test begins. The calendar arm is a small booking flow, and the guide to booking and appointment flows covers how such a flow shows availability, confirms the slot and makes changes easy.
| Design element | Choice | Reason |
|---|---|---|
| Unit of randomisation | Demo requester, assigned at form submission | The step after the form is what differs |
| Arms | Booking calendar on the confirmation step versus standard callback | One change: how the demo gets arranged |
| Form | Identical in both arms | Isolates the post-form step |
| Contact rules | Calendar arm gets confirmation and reminders, a call only if missed | Prevents the calendar arm getting both treatments |
| Availability | Same reps and similar slot availability for both arms each week | Calendar gaps would penalise one arm |
| Primary metric | Demos held per request within a fixed window | Counts conversations, not bookings |
Sample size reasoning
Demo requests are scarce, so the sample is the main constraint. The inputs are the current share of requests that end in a held demo, the smallest gain that would justify changing the process and the confidence required. The sample size and significance guide shows how they set the requests each arm needs, and the conversion rate calculator set up for lead generation helps derive the baseline from your own form and CRM figures. A small team receiving few requests may need months; if the planned sample is out of reach, consider whether a larger difference, such as instant booking against a callback that is slower than usual, is the question worth testing.
Duration
Enrol across full weekly cycles until the planned sample is reached, then allow a window long enough for booked demos to take place and for the callback arm to arrange and hold its demos. The callback arm's demos usually happen later, so reading before the window closes favours the calendar. Avoid enrolling across holidays or events that change reps' availability for part of the test.
Reading the results and the decision rule
Read once, after the follow-up window closes for the last enrolled request. Compare demos held per request with its confidence interval, then each guardrail. The rule, written before launch:
| Result | Decision |
|---|---|
| Demos held per request rise and qualification does not fall | Adopt instant booking |
| Demos held rise but out-of-profile bookings grow | Adopt instant booking with a qualifying step before the calendar |
| Demos held rise but the qualified rate of held demos falls | Keep the callback, or offer the calendar only to requests matching the target profile |
| No reliable difference | Keep whichever process sales prefers to run |
Once the test has finished, a significance reader for two conversion rates confirms the primary read. Enter requests in each arm as visitors and demos held as conversions. It reads a completed test; it does not size one.
Demo scheduling significance checker
Held demos lead to customers at the lead-to-customer rate the business already tracks, which turns the result into revenue terms for the decision.
A clinic scheduling software firm tests instant booking
Brightquay, an invented shift-scheduling software company for clinics, splits 1,000 demo requests over fourteen weeks: 500 see a booking calendar after the form and 500 receive the usual callback. After a three-week window, demos held per request are 52% for the calendar arm and 41% for the callback arm, with a 95% interval for the difference of roughly 5 to 17 points. No-shows among bookings are 18% against 9%, and the sales-qualified rate of held demos is 62% against 66%, inside the agreed limit. But out-of-profile companies rise from 6% to 11% of held demos. Brightquay adopts instant booking with one qualifying question before the calendar.
Pitfalls
Reading bookings rather than held demos is the first pitfall. The calendar arm almost always books more, and the gap shrinks or disappears once no-shows are counted. Calendar availability differing by week is the second. If the calendar shows few slots in a week when reps are at an event, requesters in that arm see a poorer offer, while the callback arm is unaffected; keep slot availability broadly steady and log the weeks when it was not. Reps calling the booking arm too is the third. If they phone everyone who booked, the calendar arm receives the calendar and the callback, and the test no longer compares the two mechanisms.
Time zones matter for teams that sell across regions: show slots in the requester's local time, and make sure the callback arm is not disadvantaged by calls placed outside the requester's working hours.
Common mistakes
Scheduling tests are lost in the gap between booking and holding. The lead response time test changes how fast the callback happens rather than replacing it, and the experiment library lists every design.
- Declaring the calendar a winner on bookings. Count demos that were held.
- Changing the form at the same time. Only the step after submission should differ.
- Letting reps call both arms. Agree contact rules for each arm before launch.
- Running through weeks with thin availability. Empty calendars penalise the booking arm.
- Reading before callback demos have happened. The callback arm's demos arrive later.
- Ignoring who books. Out-of-profile bookings cost sales time even when they are held.
