Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Marketing Research › Troubleshooting

Survey Response Rate Low

Why few invited people complete a survey: a questionnaire longer than the decision needs, a vague invitation, a stale frame and customers asked too often.
Troubleshootingintermediate16 min readUpdated

First, read the completion funnel from invited to finished

Before touching the questionnaire, write down four counts: how many people were invited, how many reached the first page, how many answered the first question that matters to the decision, and how many finished. The survey and questionnaire design guide says to track the completion rate as you would any conversion rate (CVR), and that reading works at every stage, not only the last. Each pair of counts is a rate, and the conversion rate calculator turns the pairs into comparable figures, so the stage with the steepest fall is visible at a glance rather than buried in a platform summary.

Where the fall sits points at the cause. A gap between invited and arrived is the invitation or its delivery: either the message gave no reason to click, or it never reached an inbox. A gap between arrived and started is the first screen: a length the invitation did not admit to, a consent page that reads like a contract, or a screener doing its job and sending people who do not qualify away, which is not a response problem at all. A gradual decline from the first question to the last is length. A single steep step at one question is that question. And a shortfall concentrated in one group of the people invited, with the rest answering as expected, is the frame or fatigue in that group rather than anything in the survey. Whether the people who did answer can speak for those who did not is a separate question with its own page, the diagnosis for survey results that are not representative; this one is about getting the answers in.

Two comparisons sharpen the reading. The first is the business's own history: the sampling tutorial takes past surveys of similar customers, not a figure from elsewhere, as the starting point for how many invitations a group needs, and the same history says whether this survey is unusual or merely ordinary. The second is time. The design guide asks for problems to be fixed in the first hours of fieldwork rather than after it closes, so a funnel read on the first morning is worth more than one read at the end. If the fix is a change to the invitation, treat it as a test rather than a rewrite: the design guide allows a split test of two invitations on a portion of the list, and the experimentation loop gives it a shape, with an observation from the funnel, a hypothesis about what the new wording will change, a send to a random part of the remaining invitations, and a written learning before the rest go out.

Diagnosis table

Diagnosis

SymptomCheckLikely causeFix
A steady decline from the first question to the last, and a completion time longer than the invitation promisedEvery question against the analysis plan, and where the questions the decision needs sit in the orderA questionnaire longer than the analysis plan needs, with the important questions lateCut what feeds no output, route people past what does not apply to them, move the decision's questions up and state the length honestly
Invitations delivered and opened, few people reaching the first pageThe invitation's text for who is asking, why, how long it takes and what happens to the answers; the sender name and subject lineAn invitation that does not say who is asking, why, how long it takes or what will happen to the answersRewrite the invitation on a research footing and test it against the old one on part of the list
Bounces and unknown addresses, replies saying they were asked already, one group answering far below the restThe frame against the cleaning and contact rules, and the bounce log of the list's last mailingA frame full of out-of-date addresses, duplicates and people surveyed recently by another teamClean the frame, apply the contact rule across teams and release a reserve in the groups that answer less
Response falling wave on wave, complaints about surveys, the same names answering every timeEvery survey any team has sent to these people in the periodCustomers asked too often, so the willing stop answering and the rest never didAgree a contact rule every team follows, send fewer and better-timed surveys and tell respondents what changed
One steep step in the funnel at a single questionThat question on a phone, its wording with people from the audience, and the "other" answers around itA steep drop at one question that is confusing, intrusive or broken on a phoneRebuild or move the question, retest it, log the change and analyse answers before and after it separately
Few arrivals from one mailbox provider while the rest look normal, or low opens everywhere with "delivered" reported as fineBounces by type, engagement by provider and the sending domain's authentication and reputationInvitations that never arrive because the sending domain or list is in poor healthFix placement before resending, and send the reminder from a healthy domain to a cleaned list

Fix, verify and prevent each cause

The questionnaire is longer than the decision needs

The survey design guide is blunt about length: each question added lowers both the share of people who finish and how carefully they answer towards the end, and tired respondents start choosing the first or middle option. Nielsen Norman Group's articles on survey design, as the site's research guides cite them, make the same point about length costing completion. The usual way a survey grows is that the questions were written before the analysis plan, so every interesting question looked useful and the ones the decision needs ended up on the last page. Grids that put many items on one scale are efficient to build and invite straight-lining.

Fix: write or reopen the analysis plan, mark the output each question feeds, cut every question that feeds nothing, add routing so people only see what applies to them, and move the questions the decision depends on to the front. Time the result at a respondent's pace and put that time in the invitation.

Verify: the funnel on the next batch of invitations declines gently rather than steadily, and the completion time matches what the invitation promised.

Prevent: write the analysis plan before the questionnaire on every study and check each question against it before the pretest, so length is argued about on paper rather than discovered in the field.

The invitation gives no reason to answer

The design guide sets out what an invitation has to say: who is asking, why, how long it takes, what will happen to the answers and whether there is an incentive. An invitation that skips those is asking a stranger for a favour without saying who is asking or what for. The professional codes the site's research guides describe from the Market Research Society expect participation to be voluntary and informed, with respondents told who is asking and why, so the honest invitation is also the proper one; and the marketing research hub notes that an invitation that promotes a product is marketing whatever its subject line says. Channel matters too: the guide sends email to customers who opted in, an in-product or on-site intercept to current users and a panel to non-customers, and treats an email invitation as subject to the same subject line, sender name and timing effects on open and click-through rate (CTR) as any campaign.

Fix: rewrite the invitation with the five things it must say, in the words customers use, from a sender name people recognise, and send one reminder to non-responders, which the guide says usually lifts response without annoying people. Where two versions are plausible, send each to a random portion of the remaining list and read arrivals before the rest go out.

Verify: the share of delivered invitations that reach the first page rises on the test portion, and the reminder adds arrivals without a rise in unsubscribes or complaints.

Prevent: keep the invitation in the survey launch checklist as a gate, with its lines on a research footing and on stating who is asking, why, how long and what happens to the answers ticked by someone other than the writer.

The frame is stale, doubled or already asked

A customer list is a frame, not the population, and the sampling tutorial lists what sits in an uncleaned one: duplicate accounts for the same person, staff and test records, trade accounts in a consumer list, out-of-date contact details, and people surveyed recently by another team. Out-of-date details mean invitations that never arrive, and the tutorial notes that the loss falls mostly on long-standing or lapsed customers, which is why a shortfall often lands in exactly the group the decision most needs. Groups that answer less, often lapsed and infrequent customers, need proportionally more invitations in the first place.

Fix: rebuild the frame as the tutorial draws it: export only the fields the draw needs, remove duplicates, staff, test records and anyone who must not be contacted, take out everyone surveyed within the contact rule's period by any team, and count what each rule removed. Then release reserve invitations only in the strata that are answering below plan, and record when and how many.

Verify: bounces on the reserve batch fall to the level of the list's normal mailings, and the thin strata fill towards the counts the analysis plan asked for.

Prevent: keep the draw record with the brief, add every respondent's survey date to the contact history so the rule protects them next time, and treat a large removal during cleaning as a finding about the list that the report should state.

Customers have been asked too often

The voice of the customer guide describes survey fatigue plainly: customers asked too often stop answering, and those who keep answering become less representative. Its answer is a contact rule, under which anyone surveyed recently is left alone for a set period, no matter which team wants to ask, and it names falling response rates and complaints about surveys as the signs that frequency is too high. The guide also sends relationship surveys a few times a year at most and transactional surveys only after interactions that matter.

Fix: list every survey the service, product, marketing and research teams have sent to these customers in the period, agree one contact rule across all of them, and take the people inside the rule's window out of this draw. Where the survey must go ahead now, invite the people the rule allows and say in the report who it left out.

Verify: the share of invited customers who reach the first page recovers on the next wave, and the replies that say "you asked me already" stop.

Prevent: close the loop with the people who did answer, because the guide says telling customers what changed raises future response rates, and keep the contact rule where every team that sends surveys can see it.

One question loses most of the people who reach it

A single steep step in the funnel is the signature of one question, and the design guide names the usual reasons: it is confusing, intrusive or broken on a phone. The tutorial on pretesting a questionnaire describes what breaks on phones, such as grids that need sideways scrolling and long option lists that push the next button off the screen, and what only people from the audience catch, such as a word that means something else to customers or a question they would rather not answer.

Fix: open the question on the device most respondents use, then put it in front of a few people from the audience and ask them to think aloud as they answer, without explaining it to them. Rebuild a grid as single questions, move an intrusive question towards the end with a "prefer not to say" option, or reword it in the audience's own phrase. Because answers before and after a change are answers to different questions, record the change and the point it took effect, and analyse the two parts separately.

Verify: on the next batch the fall at that question shrinks to the gentle decline the rest of the survey shows, and the "other" answers around it stop repeating the same missing option.

Prevent: run the pretest the tutorial sets out before every launch, including a soft launch to a small random part of the sample, read its funnel for drop-off and speeding, and freeze the questionnaire before full fieldwork.

The invitations never arrived

A platform's "delivered" count means a receiving server accepted the message, and the email deliverability guide is clear that it says nothing about whether the message reached an inbox, a promotions tab or a spam folder. A survey invitation sent through a marketing domain inherits that domain's authentication, reputation and sending pattern, so a list that has not been mailed for months, a batch of addresses that bounce, or a domain with an authentication fault can leave most invitations unseen while the platform reports success. The guide's own diagnosis reading applies: engagement falling at one provider while the others hold steady points to placement at that provider.

Fix: read bounces by type and engagement by mailbox provider for the invitation send, remove hard bounces at once, check that the sending domain is authenticated and that its reputation in the provider's postmaster view has not fallen, and work through the placement causes in the diagnosis for an open rate that dropped before sending anything else. Then send the reminder from the healthy domain to the cleaned list.

Verify: opens and clicks on the reminder sit at the level of the list's normal mailings across every provider, and arrivals at the survey's first page rise with them.

Prevent: send research invitations from a domain that is kept in good health, never from one that has been idle, and read the provider split before fieldwork is judged a failure.

Prevention: a pretest, an honest invitation and a contact rule every team follows

Every cause above is visible before the first invitation goes out, and the cheapest place to catch it is the gate the survey launch checklist describes, run before each survey and again before each wave. Its sections follow the causes: the decision and the analysis plan, so the questionnaire holds only what the decision needs; the questionnaire read for bias, pretested with the audience and frozen; the sample drawn from a cleaned frame under the contact rule and recorded; the invitation on a research footing with consent agreed; and a plan for the answers. A team that ticks those lines has already ruled out most of this page.

Plan the numbers from the smallest group, not the total. The design guide asks for the sample size to be planned for the smallest subgroup the report must read, and the sample size entry summarises the idea, while the sampling tutorial takes the share who will answer from the business's own past surveys rather than from a figure found elsewhere. Invitations per stratum then follow, with a reserve held back for the groups that answer less. A survey planned this way rarely surprises anyone, because the expected funnel was written down before fieldwork.

Pretest as the tutorial does: a check of every question against the plan, a read-through on respondents' devices at a respondent's pace, think-aloud sessions with people from the audience, a test of every route, a soft launch and a change log, and then a freeze. The time the pretest produces goes into the invitation, which states who is asking, why, how long, what happens to the answers and whether there is an incentive, with one reminder planned from the start. Where the invitation is the thing being improved, the experimentation loop keeps the change honest: one hypothesis, one portion of the list, one reading, one written learning.

Keep the frame and the contact rule as standing records rather than study-by-study tasks. The draw record lives with the brief, the contact history gains every respondent's survey date, and the contact rule is agreed across the teams that send surveys and kept where they can all see it. The voice of the customer guide's point about closing the loop belongs here too: people who hear what changed because they answered are more willing to answer again, and a programme that never tells them trains them to stop. Where surveys run continuously, the customer research section sets out the rest of the practice around them. And where the real question is why people do something rather than how many do it, more invitations are the wrong answer: a handful of interviews, recruited and screened as the tutorial on recruiting research participants describes, will answer it better than a larger survey, and if those interviews are already running and producing nothing usable, the diagnosis for interviews that produce no insight is the page for that.

Finally, treat the sending domain as part of the research instrument. Email invitations are email, and the deliverability guide's habits apply: authentication in place, hard bounces removed, engagement watched by provider, and no sudden whole-list send from a domain that has been quiet. A funnel that starts at "delivered" rather than "arrived" hides the last cause on this page, so build the funnel from the survey platform's arrivals, not the email platform's acceptances. Other research symptoms have their own pages in the marketing research troubleshooting section.

A pottery school finds a question that lost half its respondents (illustrative)

Illustrative example — not a real result

Yolandrick Pottery Studio

Yolandrick Pottery Studio, an invented pottery school, surveys past students about new course formats and plans on 400 completed answers from 2,400 invitations. After the first week it has 190. The email platform reports 590 bounces, about a quarter of the frame, almost all from students who last booked more than 2 years ago, which is the lapsed group the decision most needs; nobody had removed the addresses that bounced on the last newsletter. Of the 1,810 delivered, 1,160 were opened and 620 people clicked through, so the invitation is doing its job; 580 start the survey.

The question-by-question funnel shows 540 people reaching question 7, a grid of 11 rating rows, and 280 leaving it. On a phone the grid needs sideways scrolling and the next button sits below the fold. A further 230 invitations went to students the front desk had surveyed about parking 4 weeks earlier; 6 of them answered.

The studio rebuilds the grid as 11 single questions, each with a labelled scale, and pretests the rewritten section with 3 former students on their own phones; all 3 finish in under 7 minutes, which replaces the "about 5 minutes" the invitation had promised. The change and the date it took effect go into the change log, and answers before and after it are analysed separately. The studio removes the 590 bounced addresses, takes the parking respondents out of the frame under a new rule that nobody is surveyed twice within 3 months, sends one reminder to the people who were delivered but did not start, and releases 350 reserve invitations in the lapsed stratum only. Over the next 10 days the reminder and the reserve produce 540 starts and 300 completions, 64 of them from lapsed students against 9 before, which takes the total past the 400 planned. The draw record and the contact rule go into the studio's research folder, and the next wave is planned from this funnel rather than from hope.

Common mistakes

  • Adding an incentive before reading the funnel. An incentive changes who answers; it does not fix a broken question or a stale frame.
  • Reading "delivered" as "invited". Acceptance by a server is not an invitation anyone saw.
  • Shortening the survey in the field without a log. Answers before and after the change are answers to different questions.
  • A third reminder to everyone. More invitations bring more of the same kinds of people; release the reserve in the thin strata instead.
  • Pretesting with colleagues. They know what every question means, so the question that loses respondents is never found.
  • One team's survey at a time. Without a shared contact rule, the service team's survey last month is this survey's shortfall.
  • Judging fieldwork on the last day. The first morning's funnel is when a fix is still cheap.

Frequently asked questions

Should we add an incentive to lift the response?

Not until the funnel has said where people are being lost. An incentive cannot rescue a question that breaks on a phone, a frame full of dead addresses or an invitation that never arrived, and it changes who answers, which is why the survey design guide asks the invitation to say whether there is one and the recruitment guidance asks for a form that compensates people for their time without biasing the result, such as a voucher rather than a discount on the product being researched. If the funnel points at a weak invitation, test an incentive against a clearer invitation on part of the list and read which one brings in the groups that were missing.

How many reminders can we send?

The survey design guide's advice is one reminder to non-responders, which usually lifts response without annoying people. A further reminder mostly brings more of the same kinds of people, and the sampling tutorial makes the same point about reserves: extra invitations add answers but do not remove non-response bias. If particular groups are short, release reserve invitations in those strata rather than reminding everyone again, and record what was released and when.

Can we cut the survey while it is still in the field?

Only as a last resort, when the funnel shows a single question losing most of the people who reach it. The pretesting tutorial treats the answers gathered before and after any change as separate sets, so note the moment the change went live and read the two parts apart rather than pooling them. If the shortfall comes from overall length rather than one question, leave the questionnaire as it is for this wave, finish fieldwork and cut against the analysis plan before the next one.

Terms used on this page

Tools to try

Frameworks

Where this page is used

Sources

  1. Market Research Society — MRS
  2. Nielsen Norman Group articles — Nielsen Norman Group

Related