Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Marketing Research › Customer Research

Survey and Questionnaire Design

How to design surveys that produce usable answers: question types and scales, wording and order bias, length and invitations, sampling, consent, and analysis.
Guideintermediate11 min readUpdated

A survey is a measurement instrument

A survey is not a list of things the team would like to know. It is an instrument that converts people's experiences and opinions into numbers, and like any instrument it can be badly calibrated. A thermometer that reads high is worse than no thermometer, because people act on it. A survey with leading questions or a skewed sample produces confident numbers that point the wrong way.

Start with the analysis, not the questions. Write down the decision, the few figures that would inform it, the comparisons the analysis will make (between segments, over time, or against a target), and the tables or charts the report will contain. Every question must feed one of those outputs. Questions that are merely interesting make the survey longer, which costs completions and attention, and they are rarely analysed. The research brief template is where the decision and those outputs are written down before the questionnaire is drafted.

Surveys measure well what people can report accurately: recent behaviour, awareness, satisfaction with an experience they have had, preferences between concrete options, and characteristics used for segmentation. They measure badly what people cannot know about themselves, such as why they really chose something or what they would do in a situation they have not faced. Those questions belong in customer interviews, which should usually come first and supply the options and the language the survey uses. The tutorial on turning interview findings into survey questions walks through that hand-off. Ongoing customer experience metrics, such as satisfaction and loyalty scores tracked over time, are a programme of their own and are covered in the voice of the customer guide.

Question types and scales

The question type decides what the analysis can do with the answer. Choose it by working backwards from the output.

TypeWhat it measuresGood forWatch for
Single choiceOne option from a listBehaviour, categories, segmentationOptions must be exhaustive and mutually exclusive; include "other" and "none"
Multiple choiceEvery option that appliesChannels used, features used, sources consultedLong lists favour items near the top; randomise order
Rating scale (agreement, satisfaction, likelihood)Intensity of an attitudeComparing items, segments or wavesLabel every point; keep the same direction throughout
RankingRelative priorityA short list of options people can genuinely compareHard on long lists; the bottom ranks are noisy
Constant sum or trade-offRelative importance under constraintPrioritising features or benefitsCognitively demanding; needs a clear instruction
Numeric entryA count or amountFrequency, spend, team sizeRanges are easier to answer and less intrusive
Open textUnprompted reasons and languageCatching what the options missedSlow to analyse; use sparingly and code the answers

For rating scales, choose a length and keep it consistent, label every point rather than just the ends, include a neutral midpoint unless there is a strong reason not to, and offer "don't know" or "not applicable" where some people genuinely cannot answer. Keep scales running in the same direction across the survey, so that respondents do not have to re-read each one. Grids that put many items on one scale are efficient but invite straight-lining, where tired respondents click the same column down the page.

Wording, order and bias

Small wording changes move results, which is why most survey bias is introduced by the people writing the questions rather than by the people answering them. Nielsen Norman Group's articles on survey design cover many of these effects in a user research context; the principles carry across.

Leading questions contain the expected answer: "How much did you enjoy our new checkout?" assumes enjoyment. Ask neutrally: "How would you rate the checkout?" with a balanced scale.

Double-barrelled questions ask two things at once: "How satisfied are you with our price and delivery?" Split them.

Loaded or unfamiliar terms confuse or steer. Use the words customers use, which interviews supply, and avoid internal product names and jargon.

Acquiescence is the tendency to agree with statements. Prefer questions that ask for a rating or a choice over agree-or-disagree statements, or balance positively and negatively worded statements.

Social desirability pushes people towards answers that make them look good, such as reading more or spending less. Make answering honestly easy: neutral wording, anonymity where possible, and ranges rather than exact figures for sensitive amounts.

Order effects arise because earlier questions frame later ones. Ask general questions before specific ones, unprompted awareness before prompted, and overall satisfaction before questions about individual aspects. Randomise the order of options in lists, except where the order is natural, such as scales or age bands. A brand tracker depends on this ordering more than most surveys, and the tutorial on setting up a brand tracking survey builds a complete questionnaire map around it.

Test every survey before launch with a few people from the target audience, ideally while watching them answer and asking them to think aloud. Pretesting finds confusing questions, missing options and broken routing faster than any amount of internal review. The tutorial on pretesting a questionnaire runs the think-aloud sessions, the route checks and a soft launch step by step.

Length, invitations and response

Every additional question reduces the share of people who finish and the care they take near the end. Cut ruthlessly against the analysis plan, use routing so people only see questions that apply to them, and put the most important questions early. Tell people honestly how long the survey takes; an estimate that proves too short damages trust and completion.

The invitation decides who responds. State who is asking, why, how long it takes, what will happen to the answers and whether there is an incentive. Send it through the channel the audience uses: email for customers who opted in, an in-product or on-site intercept for current users, a panel for non-customers. Invitations by email are subject to the same deliverability and timing realities as any other email campaign: subject line, sender name and timing all affect the open and click-through rate (CTR), and one reminder to non-responders usually lifts response without annoying people.

Watch the completion funnel as data comes in. A steep drop at one question usually points to a problem with that question: it is confusing, intrusive or broken on mobile. Track the completion rate as you would any CVR, and fix problems in the first hours rather than after fieldwork closes. When a survey already in the field has far fewer answers than planned, the diagnosis for a low survey response rate reads that funnel stage by stage to separate the questionnaire, the invitation, the frame, fatigue, a single broken question and delivery. Where two versions of an invitation or an incentive are plausible, an A/B Test on a portion of the list can settle which recruits better, as long as both versions reach comparable people.

Sampling, weighting and representativeness

A survey describes the people who answered it. Whether it describes anyone else depends on how they were selected and who chose not to respond.

Define the population the decision concerns: all active customers, recent buyers, people in the category who have never bought from you. Then identify a sample frame that reaches them, such as a customer list, a panel or a site intercept, and note who the frame misses. A customer email list misses non-customers, customers who did not opt in, and people whose addresses are out of date. When the frame is the business's own records, the tutorial on drawing a survey sample from a customer list cleans the list, stratifies it and compares the people who answered with the frame.

Probability samples, where everyone in the frame has a known chance of selection, support stronger claims about the population. Most marketing surveys use non-probability samples, such as opt-in panels, email lists and intercepts, which are quicker and cheaper but rely on assumptions about whether respondents resemble the population. Those assumptions should be checked, not taken on trust.

Non-response bias arises when the people who answer differ from those who do not. Engaged and satisfied customers tend to respond more. Compare respondents with the frame on what you already know, such as tenure, spend, region or product, and look for gaps. When a finding has already been challenged, the diagnosis for survey results that are not representative works back through the frame, non-response, wording, design, weighting and the pretest to say what the survey can still claim.

Weighting adjusts results so that groups count in proportion to their share of the population, using known figures for the population as targets. It corrects imbalances on the variables weighted, not on others, and heavy weights make results less stable. Weight on a few variables that matter to the answers, and report both weighted and unweighted figures when they differ.

Sample size determines how precisely results can be read, especially for subgroups. The arithmetic of precision is covered in statistical texts and in the conversion testing material; the design point is to plan for the smallest subgroup you need to report, not the total.

Surveys collect personal data whenever responses can be linked to a person, which includes many surveys that do not ask for a name: an email invitation with a tracked link, or a customer ID passed into the survey, is enough. The professional codes published by ESOMAR and the MRS set the standard for research: participation is voluntary and informed, respondents are told who is asking and why, research is never used as a cover for selling, and extra care is taken with children and vulnerable people.

Data protection law, including the GDPR where it applies, adds legal duties: a lawful basis for processing, a clear notice of what is collected and why, minimising what is collected, keeping it secure and deleting it when no longer needed. In practice that means asking only what the analysis needs, saying whether responses are anonymous, confidential (linked but not reported by name) or attributable, separating identifiers from answers as early as possible, and not using research responses for marketing follow-up unless respondents agreed to that separately. Sensitive topics such as health need extra care and, often, explicit consent. Treat this as general direction and agree the specifics with whoever handles data protection. A survey fielded in another country and language needs adapting, translating and a back-translation check before any of this, which the guide to researching customers in a new market works through.

Analysing and reporting survey results

Clean the data before analysing it: remove test responses, duplicates, people who failed screening questions, and respondents who straight-lined or finished implausibly fast. The tutorial on cleaning and tabulating survey responses applies those rules as written before anyone reads the results, logs each exclusion and builds every table with its base. Code open-text answers into themes, using a codeframe built from a sample and then applied to the rest; when a language model applies the codeframe, the AI research synthesis guide sets out the verbatim-excerpt and blind-check rules that keep the coding honest. Then run the analysis plan written at the start: the figures for the decision, the comparisons between segments, and the trends against earlier waves. The survey analysis plan template lays that plan out field by field, from the question-to-output map to the rules for small groups.

Report with restraint. Give the base for every figure, so readers know how many people answered each question. Show differences between groups only where the sample supports them, and say when a subgroup is too small to read. Present the exact question wording alongside the result, because "satisfied" means little without the scale it came from. Separate what the data shows from what the team concludes, and state the sample's limits plainly near the top, not in a footnote.

Connect results to action. Where a finding should change a target, a message or a KPI, say so. Where a finding raises a question the survey cannot answer, propose the follow-up, which is often an interview round or a behavioural check. For satisfaction or effort scores linked to customer records, the tutorial on testing whether survey scores predict churn runs that behavioural check against what the same customers did afterwards. The conversion research methods guide covers on-site polls and other research aimed at specific pages and funnels, which complement broader customer surveys. Where the survey is a single question on one page of the website, the tutorial on on-page polls and post-purchase surveys applies the same wording rules and reads the answers for themes and phrases rather than proportions.

Illustrative example — not a real result

A garden centre chain fixes a skewed survey

Bramblewick Garden Centres surveys its loyalty-card members about a planned café refurbishment. The first draft asks, "How much would you enjoy a new, modern café?" and takes 18 minutes in pretesting. The research lead rewrites the question neutrally, cuts the survey to 7 minutes, and randomises the list of café features. The invitation goes to 24,000 members; 2,600 complete it. Respondents skew older than the membership: 62% are over 55 against 41% of members. After weighting by age band to membership figures, support for a larger seating area falls from 71% to 58%, and interest in a children's menu rises from 19% to 27%. The report shows both weighted and unweighted figures, states that non-members are not covered, and recommends a short intercept survey in two stores to reach them.

Common mistakes

  • Writing questions before the analysis plan. Long surveys full of interesting questions nobody analyses.
  • Leading and double-barrelled wording. Questions that suggest the answer or ask two things at once.
  • Asking people to predict themselves. Hypothetical purchase intent and willingness to pay read as facts.
  • Skipping the pretest. Confusing questions and broken routing found only after the data is in.
  • Treating a list as the market. Results from engaged email subscribers reported as if they described all customers or all buyers.
  • Reporting subgroups that are too small. Differences presented as findings when they are noise.
  • Mixing research and marketing. Using survey responses for sales follow-up without separate consent.
  • Hiding the method. Figures without bases, question wording or sample caveats.

Frequently asked questions

How long should a customer survey be?

As short as the decision allows. Every question beyond what the analysis plan needs costs completions and lowers the quality of answers at the end of the survey, because tired respondents start choosing the first or middle option. Write the analysis plan first, cut any question that does not feed it, and test how long the survey takes a real person to complete before sending it.

Should rating scales have a neutral midpoint?

Usually yes. Removing the midpoint forces people with no real opinion to pick a side, which adds noise rather than information. The exception is when you have good reason to believe people are hiding behind the middle on a topic they do have a view on; even then, offering a separate 'don't know' or 'not applicable' option is more honest than forcing a choice.

Can I trust the results of a survey sent to my email list?

You can trust them as a description of the people on your list who chose to answer, which is a useful but narrow group. They will usually be more engaged and more positive than customers as a whole, and they say nothing about people who are not on the list. Compare the respondents with the list on what you already know, such as tenure or purchase history, and weight or caveat the results where they differ.

Terms used on this page

Where this page is used

Part of these learning paths

Also related

Sources

  1. ESOMAR — ESOMAR
  2. Market Research Society — MRS
  3. Regulation (EU) 2016/679 (GDPR) — EUR-Lex
  4. Nielsen Norman Group articles — Nielsen Norman Group

Related