What a pretest catches that a review does not
The survey design guide says to test every survey before launch with a few people from the target audience, ideally watching them answer and asking them to think aloud, because pretesting finds confusing questions, missing options and broken routing faster than any amount of internal review. The reason is simple. The people who wrote the survey know what each question is for, so they read it as intended. Respondents read it as written, on a small screen, after a day's work, with their own meaning for every word.
An internal review is still worth doing, and it comes first. It catches what the writers can see: a question that feeds no output, a typing error, a scale that runs the wrong way. What it cannot catch is the gap between what the business means and what the audience understands. Nielsen Norman Group's articles on survey design cover many of the wording effects involved, in a user research context, and the survey design guide sets out the wording and order rules the pretest checks against.
A pretest also protects something less obvious. The guide asks for a soft check on completion as data arrives, reading the completion rate as you would a CVRShare of visits or clicks that convert.. A pretest makes that check deliberate: it runs the survey on a small slice of the sample first, while there is still time to act on what it shows. If a survey is already in the field with far fewer answers than planned, the diagnosis for a low survey response rate works back from that funnel to the questionnaire, the invitation, the frame, fatigue, a single broken question or delivery.
Pretesting has a close relative in conversion research. The usability testing tutorial runs a pilot session before a round of usability testsA research session in which a person attempts a realistic task on a site, app or prototype while an observer notes where they struggle. (the usability test entry defines the method), and its moderating rules carry over to the think-aloud sessions below. The brand tracker tutorial pilots its questionnaire before the baseline wave, because after that every fix is a break in the series.
Before you start
Have the analysis plan beside the questionnaire: the decision, the figures that will inform it, the comparisons the analysis will make and the tables the report will contain. The research brief template is where those are written down. Without them, the first step has nothing to check against. The survey analysis plan template holds the question-to-output map this first step checks against. The analysis planA survey's plan, written before the questionnaire: the decision, the figures and comparisons the report needs, which question feeds each, and small-group rules. comes before the questionnaire, which is what lets it act as the standard every question is checked against.
Build the survey in the tool it will run in, with all routing, randomisation and validation switched on, and set up a test link whose answers are kept apart from real data or can be removed cleanly. A pretest on a document or a slide misses the faults that live in the build.
Recruit a few people from the audience for think-aloud sessions, screened as the recruitment tutorial describes, and include people from each group whose routes differ. Colleagues are not a substitute, for the reason the FAQ gives.
Pretest participants are research participants. The principles the site's research guides describe from the MRS and ESOMAR codes apply: they take part voluntarily, are told who is asking and why, and their answers stay confidential unless they agree otherwise. Sessions are often recorded, and recordings are Personal DataAny information relating to an identified or identifiable living person, such as a name, email address, location data or an online identifier.; agree what is recorded, who sees it and when it is deleted with whoever owns data protection, as the customer interviews tutorial does for its sessions.
Open a change log: a simple table with the question, the problem found, the evidence, the change made and the date.
The steps
Check every question against the analysis plan
Go through the questionnaire with the analysis plan open and mark, for each question, the output it feeds. Cut any question that feeds nothing, because every extra question costs completion and attention. For the rest, check that the answer format can produce the output: a comparison between groups needs the grouping question; a trend needs the wording of the earlier wave, unchanged; a ranking needs options people can genuinely compare. Then check the survey design guide's wording rules question by question: no leading or double-barrelled wording, no internal product names, scales labelled and running in the same direction, a "don't know" or "not applicable" where some people cannot answer, and option lists that are complete and do not overlap.
Read it as respondents will see it, on the devices they use
Take the survey yourself through the test link on each kind of device the audience uses, with the most common one first. Most survey problems on phones are layout problems: grids that need sideways scrolling, long option lists that push the "next" button off the screen, sliders that are hard to set, text boxes too small to show what was typed. The guide to forms covers the field-level friction a survey shares with any form: labels, input types, error messages and accessibility. Time the survey at a respondent's pace, not a writer's, and make sure the time the invitation promises is honest.
Run think-aloud sessions with people from the audience
Ask each participant to complete the survey while saying aloud what they are thinking: what they read, what they think it means, why they choose an answer. Borrow the usability testing tutorial's rules for moderating: say that the survey is being tested, not the person; then stay quiet. Do not explain a question that confuses them, because respondents in the field will have no one to ask; the moment they hesitate is the finding. Note what each participant did and said, where they paused, what they reread and what they skipped, keeping observations apart from your interpretation of them.
Probe what each question meant to them
At the end of the survey, or at natural breaks, go back to the questions that caused hesitation and to the key questions for the decision, and ask what each meant to the participant. Useful probes are plain: "In your own words, what was this question asking?", "What did you have in mind when you chose that answer?", "Was there an answer you wanted to give that wasn't there?", "How did you decide between these two options?". Listen for terms that carry a different meaning for customers, time periods people cannot recall, options that force a choice, and questions people find intrusive. Write down the words they use; they are often the better wording. For a translated questionnaire, run these probes in the target language with a native-speaking moderator, as the guide to researching customers in a new market explains alongside back-translationA translation of the target-language version back into the source language, made to check what risky lines say; it shows meaning, not how the text reads..
Test every route and skip
Draw the routing as a list of paths: every answer that sends someone to a different question, and every group that should see a different block. Walk each path through the test link and check that the right questions appear, the wrong ones do not, randomisation works, required questions are required and optional ones are not, and validation accepts ordinary answers. Pay particular attention to the routes nobody on the team would take, such as the person who answers "none" or "don't know" to an early filter question, because broken routes usually hide there. Check that the data export records each answer in the right column with the right labels.
Soft launch to a small part of the sample
Send the survey to a small, random part of the people it will go to, in the same way, with the same invitation, and then pause. The soft launch tests what the sessions cannot: how the real audience behaves without anyone watching, on the real invitation and the real devices. If the sample was drawn from the business's own records, take the soft-launch group at random from the draw, as the tutorial on drawing a survey sample from a customer list describes, so the full fieldwork can carry on from the same draw. Decide in advance whether soft-launch answers will be kept if nothing changes, and record that decision.
Read the soft-launch data for drop-off, speeding and 'other' answers
Read the completion funnel question by question, the way the forms guide reads field-level drop-off and the Form Abandonment RateThe share of people who start filling in a form, by entering a first field, but leave without submitting it.: a steep fall at one question points at that question. Look at how long people took and find the ones who finished implausibly fast or answered every grid in the same column, which are the survey guide's cleaning rules applied early. Read every "other, please specify" answer: several people writing the same thing means a missing option. Look for questions where most people choose "don't know", and for answers that contradict each other. Check that the data arrives in the shape the analysis plan needs.
Fix, log each change and retest
Fix each problem the sessions and the soft launch found, and record it in the change log with the evidence and the date. Retest anything that changed: a reworded question with further people from the audience, a changed route by walking it again, a moved question by reading the funnel again if the change was large. Keep the fixes proportionate. A pretest that turns into a redesign has found a problem with the analysis plan, and that goes back to the brief.
Freeze the questionnaire before full fieldwork
Sending the full survey is a one-way door in the sense of the one-way and two-way door framework: answers already collected cannot be re-asked with better wording, so the care belongs before it. Freeze the wording, the order, the routing and the invitation, save a copy, and note the frozen version in the change log. On a tracked survey, the brand tracker tutorial's rule applies: after the baseline, change only with a logged reason and the wave it took effect in. Then launch, and keep watching the funnel in the first hours of fieldwork for anything the pretest missed.
What each check finds
Each check is good at a different kind of problem, which is why a pretest uses several rather than one. The table summarises what each tends to catch and what it cannot.
| Check | What it tends to find | What it cannot find |
|---|---|---|
| Review against the analysis plan | Questions that feed no output; missing grouping questions; wording that breaks the survey guide's rules | Words the audience understands differently |
| Reading on respondents' devices | Grids and lists that break on phones; dishonest time estimates; hard-to-use inputs | How real respondents interpret the questions |
| Think-aloud sessions | Hesitation, rereading, skipped questions, intrusive topics | Problems that appear only at volume or without a moderator present |
| Meaning probes | Terms with a different meaning for customers; missing options; recall periods people cannot manage | Whether a problem is common or limited to one participant |
| Route-by-route test | Questions shown to the wrong people; broken skips; validation that rejects ordinary answers | Whether the questions are understood |
| Soft launch | Drop-off points, speeding, straight-lining, repeated "other" answers, export problems | Why people left; that needs the sessions |
| Change log and freeze | Unrecorded edits; versions that drift during fieldwork | Nothing on its own; it keeps the other checks honest |
The same discipline applies to a short survey intended for publication. The guide to campaign ideation and data stories notes that the Market Research Society and ESOMAR publish codes and guidance on sampling, question wording and reporting, and that journalists check survey methods; a pretest is the cheapest way to make sure the published wording means what the headline claims. Polls of a single question on one page are a lighter case, covered by the tutorial on on-page polls and post-purchase surveys.
A kitchen fitter pretests its lost-quote survey (illustrative)
Hexomby Kitchens pretests a survey for people who bought elsewhere
Hexomby Kitchens, an invented kitchen fitter, wants to know why people who received a quote went on to buy from someone else. The draft survey has 16 questions, routes people who bought a kitchen elsewhere to an installation block and people who did nothing yet to a timing block, and promises to take 6 minutes.
The review against the analysis plan cuts 3 questions that fed no output. On a phone, a grid of 7 kitchen features needs sideways scrolling, so it becomes a list of single questions, and the survey takes 9 minutes at a respondent's pace; the invitation is changed to say so.
Five think-aloud sessions follow, with people who took a quote in the past year. Three of them read "Did you book a design consultation?" as asking whether they paid for a visit, and answer "no" although they had a free home visit; the probes produce the customers' own phrase, "a home visit to measure up", which replaces it. The route test finds that people who chose "bought from a builder as part of wider work" skip the installation block meant for them.
The soft launch goes to 200 people from the draw. 64 start and 41 finish; 15 of the 23 who leave do so at the question asking for their exact kitchen budget. The budget question is moved near the end and changed to 5 bands with a "prefer not to say" option. 9 "other" answers name a supermarket's kitchen range, which becomes an option. Each change goes into the change log, the reworded question is checked with 2 more people, and the questionnaire is frozen. In full fieldwork, drop-off at the budget question falls to 2% of the people who reach it.
Verify it worked
- Open the change log. Each entry should show the question, the problem found, the evidence, the change made and the date, and the log should note the frozen version.
- Hold the questionnaire against the analysis plan. Every remaining question should feed an output, and no output should lack its question.
- Open the survey in the build, on the most common device first. Nothing should need sideways scrolling, and the time the invitation promises should match a respondent's pace.
- Walk every path through the test link. The right questions should appear, the wrong ones should not, and the export should put each answer in the right column with the right labels.
- Read the funnel again if a fix was large. No single question should show a steep fall, and repeated "other" answers should now be options.
If it goes wrong
- Participants read a question differently from your intent. The likely cause is a term that means something else to customers. Use the words they gave in the probes, then retest with further people from the audience.
- Many soft-launch respondents leave at one question. The likely cause is that it is intrusive, hard to answer or badly placed. Reword it, move it or add a "prefer not to say" option, then read the funnel again.
- Several "other, please specify" answers say the same thing. The likely cause is a missing option. Add it to the list and check the change with further people.
- Respondents see the wrong questions, or the export columns are wrong. The likely cause is a broken route, usually on a path nobody on the team took. Redraw the paths, walk each one again and check the export.
- The fixes keep growing into a redesign. The likely cause is a problem with the analysis plan, not the questionnaire. Go back to the brief before changing more.
Next steps
- Read the survey and questionnaire design guide for the wording and order rules that your fixes should follow.
- Read the tutorial on drawing a survey sample from a customer list to see how a random draw from the business's own records is made before you launch.
- Read the usability testing tutorial for the moderating rules that carry over to further think-aloud sessions.
Common mistakes
- Pretesting with colleagues only. They know what every term means and what each question is for, so the audience's misreadings never surface.
- Helping in a think-aloud session. Explaining a confusing question removes the finding and leaves the question confusing for everyone in the field.
- Testing a document, not the build. Routing, validation and phone layout live in the survey tool, and so do most of the faults.
- Walking only the main route. Broken skips hide on the paths nobody on the team would take.
- Launching to everyone at once. Without a soft launch, a question that drives people away does so across the whole sample.
- Changing questions mid-fieldwork. Answers before and after a change answer different questions, and a tracked survey loses comparison with earlier waves.
- Fixing without a log. Nobody can later say which version a respondent saw or why a question changed.
- Treating pretest participants as testers rather than participants. They are taking part in research and are owed the same information, consent and confidentiality as respondents.
