A survey is a measurement instrument
A survey is not a list of things the team would like to know. It is an instrument that converts people's experiences and opinions into numbers, and like any instrument it can be badly calibrated. A thermometer that reads high is worse than no thermometer, because people act on it. A survey with leading questionsA survey or interview question whose wording contains or suggests its expected answer, producing measurement bias rather than evidence. or a skewed sample produces confident numbers that point the wrong way.
Start with the analysis, not the questions. Write down the decision, the few figures that would inform it, the comparisons the analysis will make (between segments, over time, or against a target), and the tables or charts the report will contain. Every question must feed one of those outputs. Questions that are merely interesting make the survey longer, which costs completions and attention, and they are rarely analysed. The research brief template is where the decision and those outputs are written down before the questionnaire is drafted.
Surveys measure well what people can report accurately: recent behaviour, awareness, satisfaction with an experience they have had, preferences between concrete options, and characteristics used for segmentation. They measure badly what people cannot know about themselves, such as why they really chose something or what they would do in a situation they have not faced. Those questions belong in customer interviews, which should usually come first and supply the options and the language the survey uses. The tutorial on turning interview findings into survey questions walks through that hand-off. Ongoing customer experience metrics, such as satisfaction and loyalty scores tracked over time, are a programme of their own and are covered in the voice of the customer guide.
Question types and scales
The question type decides what the analysis can do with the answer. Choose it by working backwards from the output.
| Type | What it measures | Good for | Watch for |
|---|---|---|---|
| Single choice | One option from a list | Behaviour, categories, segmentation | Options must be exhaustive and mutually exclusive; include "other" and "none" |
| Multiple choice | Every option that applies | Channels used, features used, sources consulted | Long lists favour items near the top; randomise order |
| Rating scale (agreement, satisfaction, likelihood) | Intensity of an attitude | Comparing items, segments or waves | Label every point; keep the same direction throughout |
| Ranking | Relative priority | A short list of options people can genuinely compare | Hard on long lists; the bottom ranks are noisy |
| Constant sum or trade-off | Relative importance under constraint | Prioritising features or benefits | Cognitively demanding; needs a clear instruction |
| Numeric entry | A count or amount | Frequency, spend, team size | Ranges are easier to answer and less intrusive |
| Open text | Unprompted reasons and language | Catching what the options missed | Slow to analyse; use sparingly and code the answers |
For rating scales, choose a length and keep it consistent, label every point rather than just the ends, include a neutral midpoint unless there is a strong reason not to, and offer "don't know" or "not applicable" where some people genuinely cannot answer. Keep scales running in the same direction across the survey, so that respondents do not have to re-read each one. Grids that put many items on one scale are efficient but invite straight-lining, where tired respondents click the same column down the page.
Wording, order and bias
Small wording changes move results, which is why most survey bias is introduced by the people writing the questions rather than by the people answering them. Nielsen Norman Group's articles on survey design cover many of these effects in a user research context; the principles carry across.
Leading questions contain the expected answer: "How much did you enjoy our new checkout?" assumes enjoyment. Ask neutrally: "How would you rate the checkout?" with a balanced scale.
Double-barrelled questions ask two things at once: "How satisfied are you with our price and delivery?" Split them.
Loaded or unfamiliar terms confuse or steer. Use the words customers use, which interviews supply, and avoid internal product names and jargon.
Acquiescence is the tendency to agree with statements. Prefer questions that ask for a rating or a choice over agree-or-disagree statements, or balance positively and negatively worded statements.
Social desirability pushes people towards answers that make them look good, such as reading more or spending less. Make answering honestly easy: neutral wording, anonymity where possible, and ranges rather than exact figures for sensitive amounts.
Order effects arise because earlier questions frame later ones. Ask general questions before specific ones, unprompted awareness before prompted, and overall satisfaction before questions about individual aspects. Randomise the order of options in lists, except where the order is natural, such as scales or age bands. A brand tracker depends on this ordering more than most surveys, and the tutorial on setting up a brand tracking survey builds a complete questionnaire map around it.
Test every survey before launch with a few people from the target audience, ideally while watching them answer and asking them to think aloud. Pretesting finds confusing questions, missing options and broken routing faster than any amount of internal review. The tutorial on pretesting a questionnaire runs the think-aloud sessions, the route checks and a soft launchReleasing a survey or test to a small share of its audience first, then pausing to read the data for problems before the full launch. step by step.
Length, invitations and response
Every additional question reduces the share of people who finish and the care they take near the end. Cut ruthlessly against the analysis plan, use routing so people only see questions that apply to them, and put the most important questions early. Tell people honestly how long the survey takes; an estimate that proves too short damages trust and completion.
The invitation decides who responds. State who is asking, why, how long it takes, what will happen to the answers and whether there is an incentive. Send it through the channel the audience uses: email for customers who opted in, an in-product or on-site intercept for current users, a panel for non-customers. Invitations by email are subject to the same deliverability and timing realities as any other email campaign: subject line, sender name and timing all affect the open and click-through rate (CTRShare of impressions that result in a click.), and one reminder to non-responders usually lifts response without annoying people.
Watch the completion funnel as data comes in. A steep drop at one question usually points to a problem with that question: it is confusing, intrusive or broken on mobile. Track the completion rate as you would any CVRShare of visits or clicks that convert., and fix problems in the first hours rather than after fieldwork closes. When a survey already in the field has far fewer answers than planned, the diagnosis for a low survey response rate reads that funnel stage by stage to separate the questionnaire, the invitation, the frame, fatigue, a single broken question and delivery. Where two versions of an invitation or an incentive are plausible, an A/B TestControlled comparison of two variants. on a portion of the list can settle which recruits better, as long as both versions reach comparable people.
Sampling, weighting and representativeness
A survey describes the people who answered it. Whether it describes anyone else depends on how they were selected and who chose not to respond.
Define the population the decision concerns: all active customers, recent buyers, people in the category who have never bought from you. Then identify a sample frame that reaches them, such as a customer list, a panel or a site intercept, and note who the frame misses. A customer email list misses non-customers, customers who did not opt in, and people whose addresses are out of date. When the frame is the business's own records, the tutorial on drawing a survey sample from a customer list cleans the list, stratifies it and compares the people who answered with the frame.
Probability samples, where everyone in the frame has a known chance of selection, support stronger claims about the population. Most marketing surveys use non-probability samples, such as opt-in panels, email lists and intercepts, which are quicker and cheaper but rely on assumptions about whether respondents resemble the population. Those assumptions should be checked, not taken on trust.
Non-response biasThe gap between the people a survey or study invited and those who answered; engaged, satisfied customers tend to reply more, so results lean their way. arises when the people who answer differ from those who do not. Engaged and satisfied customers tend to respond more. Compare respondents with the frame on what you already know, such as tenure, spend, region or product, and look for gaps. When a finding has already been challenged, the diagnosis for survey results that are not representative works back through the frame, non-response, wording, design, weighting and the pretest to say what the survey can still claim.
Weighting adjusts results so that groups count in proportion to their share of the population, using known figures for the population as targets. It corrects imbalances on the variables weighted, not on others, and heavy weights make results less stable. Weight on a few variables that matter to the answers, and report both weighted and unweighted figures when they differ.
Sample size determines how precisely results can be read, especially for subgroups. The arithmetic of precision is covered in statistical texts and in the conversion testing material; the design point is to plan for the smallest subgroup you need to report, not the total.
Consent, anonymity and data handling
Surveys collect personal data whenever responses can be linked to a person, which includes many surveys that do not ask for a name: an email invitation with a tracked link, or a customer ID passed into the survey, is enough. The professional codes published by ESOMAR and the MRS set the standard for research: participation is voluntary and informed, respondents are told who is asking and why, research is never used as a cover for selling, and extra care is taken with children and vulnerable people.
Data protection law, including the GDPR where it applies, adds legal duties: a lawful basis for processing, a clear notice of what is collected and why, minimising what is collected, keeping it secure and deleting it when no longer needed. In practice that means asking only what the analysis needs, saying whether responses are anonymous, confidential (linked but not reported by name) or attributable, separating identifiers from answers as early as possible, and not using research responses for marketing follow-up unless respondents agreed to that separately. Sensitive topics such as health need extra care and, often, explicit consent. Treat this as general direction and agree the specifics with whoever handles data protection. A survey fielded in another country and language needs adapting, translating and a back-translationA translation of the target-language version back into the source language, made to check what risky lines say; it shows meaning, not how the text reads. check before any of this, which the guide to researching customers in a new market works through.
Analysing and reporting survey results
Clean the data before analysing it: remove test responses, duplicates, people who failed screening questions, and respondents who straight-lined or finished implausibly fast. The tutorial on cleaning and tabulating survey responses applies those rules as written before anyone reads the results, logs each exclusion and builds every table with its base. Code open-text answers into themes, using a codeframeThe defined set of codes open-text answers are tagged with, built from a sample, applied consistently, checked by a second coder and counted against a base. built from a sample and then applied to the rest; when a language model applies the codeframe, the AI research synthesis guide sets out the verbatim-excerpt and blind-check rules that keep the coding honest. Then run the analysis plan written at the start: the figures for the decision, the comparisons between segments, and the trends against earlier waves. The survey analysis plan template lays that plan out field by field, from the question-to-output map to the rules for small groups.
Report with restraint. Give the base for every figure, so readers know how many people answered each question. Show differences between groups only where the sample supports them, and say when a subgroup is too small to read. Present the exact question wording alongside the result, because "satisfied" means little without the scale it came from. Separate what the data shows from what the team concludes, and state the sample's limits plainly near the top, not in a footnote.
Connect results to action. Where a finding should change a target, a message or a KPIA metric tied directly to an objective., say so. Where a finding raises a question the survey cannot answer, propose the follow-up, which is often an interview round or a behavioural check. For satisfaction or effort scores linked to customer records, the tutorial on testing whether survey scores predict churn runs that behavioural check against what the same customers did afterwards. The conversion research methods guide covers on-site polls and other research aimed at specific pages and funnels, which complement broader customer surveys. Where the survey is a single question on one page of the website, the tutorial on on-page polls and post-purchase surveys applies the same wording rules and reads the answers for themes and phrases rather than proportions.
A garden centre chain fixes a skewed survey
Bramblewick Garden Centres surveys its loyalty-card members about a planned café refurbishment. The first draft asks, "How much would you enjoy a new, modern café?" and takes 18 minutes in pretesting. The research lead rewrites the question neutrally, cuts the survey to 7 minutes, and randomises the list of café features. The invitation goes to 24,000 members; 2,600 complete it. Respondents skew older than the membership: 62% are over 55 against 41% of members. After weighting by age band to membership figures, support for a larger seating area falls from 71% to 58%, and interest in a children's menu rises from 19% to 27%. The report shows both weighted and unweighted figures, states that non-members are not covered, and recommends a short intercept survey in two stores to reach them.
Common mistakes
- Writing questions before the analysis plan. Long surveys full of interesting questions nobody analyses.
- Leading and double-barrelled wording. Questions that suggest the answer or ask two things at once.
- Asking people to predict themselves. Hypothetical purchase intent and willingness to pay read as facts.
- Skipping the pretest. Confusing questions and broken routing found only after the data is in.
- Treating a list as the market. Results from engaged email subscribers reported as if they described all customers or all buyers.
- Reporting subgroups that are too small. Differences presented as findings when they are noise.
- Mixing research and marketing. Using survey responses for sales follow-up without separate consent.
- Hiding the method. Figures without bases, question wording or sample caveats.
