Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Experiment Library

Quote Follow-Up Test

An experiment design for open quotes: an automated follow-up sequence against owner-only follow-up, read on quotes accepted per quote sent by cohort.
Experimentintermediate11 min readUpdated

Hypothesis

If open quotes enter an automated follow-up sequence that answers the questions buyers ask after a quote and stops on any reply, more quotes will be accepted by the decision horizon, because many quotes go quiet over an unanswered question, unless the messages irritate buyers or prompt discounts that erase the gain. The decision behind it: put open quotes into an automated follow-up sequence that answers common questions, or leave follow-up to the quote's owner.

The tutorial on automating quote follow-ups builds the sequence: the quote as a stage with its own fields, messages written around the questions buyers ask after a quote, timing from the business's own history, a task for the owner before anything automated, exits on any reply or decision, and a lost reason when a quote expires. It then reports acceptance by the month quotes were sent. What it does not do is compare the sequence with the follow-up the business would have done anyway. Acceptance can rise after a sequence goes live because the season turned or prices changed, and it can stay flat while the sequence quietly recovers quotes that a slow month would otherwise have lost. A random split separates the sequence from everything else.

The sequence is a narrow form of nurturing: one person, one quote, one decision. The nurture sequences guide covers the broader pattern of moving leads that are not ready; this test is about the short stretch between a price and a yes or no. Write the arms, the messages, the exits and the limits on each guardrail into the plan before the first quote is assigned, following the experimentation loop for lead generation, which asks for the measure further down the funnel than the first response.

Metrics and guardrails

The primary metric is quotes accepted per quote sent, read for each send-month cohort. The denominator is every quote assigned to the arm in that month, whether or not the buyer ever replied. The numerator is the quotes accepted by the time the cohort closes, which is when every quote in it has been accepted, declined or passed its expiry date. This is the quote-to-order rate the tutorial reports, and the conversion rate calculator turns quotes sent and accepted into that rate for each arm and month. Win Rate, as the tutorial and the glossary define it, counts decided quotes only; read it alongside, but keep the primary metric on quotes sent, so an arm that turns more quotes into decisions of either kind is not hidden.

Three guardrails catch a sequence that wins for the wrong reasons:

  • Unsubscribes and complaints from follow-up messages. A chaser that buyers treat as nagging costs more than the quote: it costs the relationship and, where the follow-up is marketing, the right to send to that person. Count opt-outs and complaints per quote in the sequence arm against a limit set before launch.
  • Discount given per accepted quote. A sequence can win by inviting buyers to ask for a lower price, or by owners using it to offer one. Record the discount on every accepted quote in both arms; a rise in acceptance paid for with a matching fall in price is not a win.
  • Messages sent after a reply or a decision. The tutorial calls a follow-up after the buyer has said yes the most damaging thing the workflow can do. Any such message is a failure of the exits, not a statistic to be weighed, and the decision rule treats it as a reason to stop.

Read the lost reasons by arm as well. A sequence that shifts lost quotes from "no response" to "price" or "timing" has done something useful even before acceptance moves, because the business now knows why.

Design

Randomise at quote level, at the moment the quote is sent. When the quote enters the quote stage, the workflow assigns it to an arm and stores the arm on the deal. Before assigning, check whether the same buyer already has a quote in the test; if so, the new quote takes the same arm, so no buyer receives automated follow-ups on one quote and none on another. Revised quotes for the same job keep the original arm and the original send date.

Design elementChoiceReason
Unit of randomisationQuote, assigned when sent; quotes to the same buyer share one armThe sequence acts on the quote, and a buyer must not see both treatments
Control armOwner-only follow-up, as todayThe decision is whether automation adds to what owners already do
Treatment armOwner follow-up plus the automated sequence, stopping on any reply, acceptance, decline, booked call or opt-outIsolates the automated messages
Held constantOwner tasks, prices, quote format, expiry period and the closing message at expiryOnly the sequence differs between arms
Primary metricQuotes accepted per quote sent, per send-month cohortCounts quiet quotes and waits for each cohort to finish
Reading pointWhen every quote in a cohort has expired or been decidedOpen quotes never count as lost

The owner task comes first in both arms, as the tutorial asks, and the workflow waits for it to be done or for its due date to pass before the sequence arm's first message. Owners must not know which arm a quote is in, or at least must be asked to follow up exactly as they do today; an owner who chases control quotes harder to beat the machine contaminates the comparison. The lead response test faces the same issue for first contact, and its design for randomising lead response time checks that the standard arm does not slip; here the check runs the other way.

Which follow-ups are service messages and which are marketing is settled before launch, not by arm. The promotional versus transactional guide places sending the quote the buyer asked for on the service side, a transactional message, and a chaser whose purpose is to prompt a decision closer to marketing. The quote follow-up tutorial, drawing on the UK Information Commissioner's Office's direct marketing guidance and the FTC's CAN-SPAM guide, treats marketing follow-ups as needing a lawful basis, an honest sender and a working unsubscribe. So the sequence arm sends automated messages only to buyers whose consent covers them, and buyers whose consent does not cover them are excluded from the test before assignment rather than placed in the control arm by default. This is general direction, not legal advice.

Sample size reasoning

The sample is quotes, and most businesses send few of them, so this test is usually slow. The inputs are the current quote-to-order rate, the number of quotes sent per month and the smallest rise in acceptance worth the cost of building and maintaining the sequence. The sample size and significance guide shows how those combine into the number of quotes each arm needs, and dividing by the monthly volume gives the number of send-month cohorts the test must run. Splitting the quotes evenly between two arms is the fastest way to reach that number.

Where the volume cannot answer within a reasonable period, say so before launch rather than running the test and reading noise. The fallback is to run the sequence for every quote and compare send-month cohorts before and after it started, using the same cohort reading. Label that comparison as weaker evidence, because season, prices and the job mix change alongside the sequence, and record what else changed in those months.

Duration

Run enough send-month cohorts to reach the sample, and hold each cohort open until its quotes have expired or been decided. The reading stage therefore extends beyond the last quote sent by at least the expiry period. The business's own sales cycle sets how long acceptances keep arriving; the expiry period should already cover it, as the tutorial sets it from the quote history. Keep any seasonal peak in quote requests wholly inside the run, so that both arms meet it, and do not change prices, the quote format or the expiry period while a cohort is open.

Reading the results and the decision rule

Read each cohort once it closes, and the test once the last cohort closes. Compare quotes accepted per quote sent between arms with the interval around the difference, not just the point estimate. Then check the guardrails by arm: unsubscribes and complaints per quote, discount per accepted quote and any message sent after a reply or decision. Only then look at segments named before launch, such as the source of the original enquiry or the size of the quote. Write the rule into the plan in advance:

ResultDecision
Acceptance rises reliably and unsubscribes, complaints and discounts stay within their limitsKeep the sequence for every quote
Acceptance rises but discounts per accepted quote rise with itRewrite the messages so they answer questions rather than invite price talk, then retest
Acceptance does not moveRewrite the messages around the lost reasons collected, or drop the sequence
Any message reaches a buyer after a reply or decisionStop the sequence at once and fix the exits before any further reading

Pitfalls

Open quotes counted as lost are the commonest error. A test read on the last day of sending treats every quote still in the buyer's inbox as a failure, and because the cohorts are read by send month this would penalise whichever arm happened to receive more late quotes. Wait for each cohort to close.

Exits that fail after acceptance are the most damaging. Quotes are often accepted by phone, in person or by a signed form that never touches the CRM's email thread, so the exit only fires when someone updates the deal. Test every exit with test contacts before launch, as the tutorial asks, and audit the guardrail weekly during the run.

Owners following up harder in the control arm is the subtlest. If owners can see the arm, some will compensate, calling control quotes more often, and the test then measures owner effort rather than the sequence. Hide the arm from the deal view where the CRM allows it, and compare owner tasks completed per quote by arm at the end.

Discounts used to close are the last trap. A message that asks "is price the issue?" invites a price conversation; an owner who knows the sequence is running may use it to offer a reduction. Record discounts in both arms and read them as a guardrail. The demo scheduling test faces a similar question at an earlier stage, where an easier booking can add bookings that never become held demos, and the remedy is the same: read the outcome that pays, not the step that is easy to move.

Example: an invented roofing contractor tests its quote follow-ups (illustrative)

Illustrative example — not a real result

Hobrinske Roofing tests an automated quote follow-up

Hobrinske Roofing, an invented roofing contractor, sends around 80 quotes a month after roof surveys, and its quotes expire after 45 days. Its history shows a quote-to-order rate of about 30%. Owners already call each buyer 5 days after sending. The firm builds a short sequence of two emails, one answering questions about scaffolding and access, the other setting out staged payment, sent on days 10 and 20 to buyers whose consent covers them, stopping on any reply, acceptance, decline, booked call or opt-out. Quotes are split at random as they are sent, with repeat buyers kept in one arm.

After four send-month cohorts, each read 45 days after its last quote, the arms stand as follows:

ArmQuotes sentAcceptedAccepted per quote sentDiscount per accepted quoteUnsubscribes
Owner only1584629.1%£1200
Owner plus sequence1615735.4%£1152

The difference of about six points has an interval that runs from roughly minus four to plus sixteen points, so with four cohorts the result is promising but not reliable. No message reached a buyer after a decision, discounts held level, and "no response" fell as a lost reason in the sequence arm. Hobrinske keeps the split running for a further four cohorts, as its plan allowed when the first reading was inconclusive.

Common mistakes

Quote follow-up tests fail on their reading point and their exits more often than on their statistics. The experiment library holds the other lead-handling designs.

  • Reading before cohorts close. Open quotes are not lost quotes; wait for every quote in a cohort to expire or be decided.
  • Dropping the owner's call in the sequence arm. The question is whether automation adds to the owner, not whether it replaces them.
  • Splitting a buyer's quotes across arms. A buyer with two quotes should not see both treatments.
  • Counting win rate instead of acceptance per quote sent. Win rate leaves out quotes that never reach a decision, which are exactly the quotes the sequence is meant to move.
  • Ignoring discounts. Acceptance bought with a lower price is a pricing decision, not a follow-up result.
  • Untested exits. One message after a yes outweighs every quote the sequence recovers.

Frequently asked questions

Why randomise quotes rather than compare the months before and after the sequence started?

Because the months differ in more than the sequence. Season, the mix of jobs quoted, prices and the owners' own workload all move from one month to the next, and any of them can change how many quotes are accepted. Assigning quotes at random as they are sent means both arms face the same months, the same prices and the same owners, so the automated messages are the only systematic difference.

Do owners stop calling buyers in the sequence arm?

No. The owner's call or note comes first in both arms, exactly as it does today, and the automated messages are added only in the sequence arm. The test asks whether the automation adds accepted quotes on top of the owner's own follow-up, which is the decision the business actually faces, not whether automation can replace a person.

Can a quote still open when the test ends be counted as lost?

No. A quote that has neither expired nor been decided has not finished, and counting it as lost would favour whichever arm happened to have more recent quotes. Each send-month cohort is read only once every quote in it has been accepted, declined or passed its expiry date, which is why the reading comes some time after the last quote is sent.

Terms used on this page

Tools to try

Frameworks

Also related

Sources

  1. Direct marketing guidance — Information Commissioner's Office (UK)
  2. CAN-SPAM Act: A Compliance Guide for Business — Federal Trade Commission

Related