Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Marketing Research › Customer Research

Pretesting a Questionnaire Before It Goes Out

How to pretest a survey before launch: think-aloud sessions with the target audience, routing and device checks, a soft launch, and fixing what each one finds.
Tutorialintermediate13 min readUpdated

What a pretest catches that a review does not

The survey design guide says to test every survey before launch with a few people from the target audience, ideally watching them answer and asking them to think aloud, because pretesting finds confusing questions, missing options and broken routing faster than any amount of internal review. The reason is simple. The people who wrote the survey know what each question is for, so they read it as intended. Respondents read it as written, on a small screen, after a day's work, with their own meaning for every word.

An internal review is still worth doing, and it comes first. It catches what the writers can see: a question that feeds no output, a typing error, a scale that runs the wrong way. What it cannot catch is the gap between what the business means and what the audience understands. Nielsen Norman Group's articles on survey design cover many of the wording effects involved, in a user research context, and the survey design guide sets out the wording and order rules the pretest checks against.

A pretest also protects something less obvious. The guide asks for a soft check on completion as data arrives, reading the completion rate as you would a CVR. A pretest makes that check deliberate: it runs the survey on a small slice of the sample first, while there is still time to act on what it shows. If a survey is already in the field with far fewer answers than planned, the diagnosis for a low survey response rate works back from that funnel to the questionnaire, the invitation, the frame, fatigue, a single broken question or delivery.

Pretesting has a close relative in conversion research. The usability testing tutorial runs a pilot session before a round of usability tests (the usability test entry defines the method), and its moderating rules carry over to the think-aloud sessions below. The brand tracker tutorial pilots its questionnaire before the baseline wave, because after that every fix is a break in the series.

Before you start

Have the analysis plan beside the questionnaire: the decision, the figures that will inform it, the comparisons the analysis will make and the tables the report will contain. The research brief template is where those are written down. Without them, the first step has nothing to check against. The survey analysis plan template holds the question-to-output map this first step checks against. The analysis plan comes before the questionnaire, which is what lets it act as the standard every question is checked against.

Build the survey in the tool it will run in, with all routing, randomisation and validation switched on, and set up a test link whose answers are kept apart from real data or can be removed cleanly. A pretest on a document or a slide misses the faults that live in the build.

Recruit a few people from the audience for think-aloud sessions, screened as the recruitment tutorial describes, and include people from each group whose routes differ. Colleagues are not a substitute, for the reason the FAQ gives.

Pretest participants are research participants. The principles the site's research guides describe from the MRS and ESOMAR codes apply: they take part voluntarily, are told who is asking and why, and their answers stay confidential unless they agree otherwise. Sessions are often recorded, and recordings are Personal Data; agree what is recorded, who sees it and when it is deleted with whoever owns data protection, as the customer interviews tutorial does for its sessions.

Open a change log: a simple table with the question, the problem found, the evidence, the change made and the date.

The steps

  1. Check every question against the analysis plan

    Go through the questionnaire with the analysis plan open and mark, for each question, the output it feeds. Cut any question that feeds nothing, because every extra question costs completion and attention. For the rest, check that the answer format can produce the output: a comparison between groups needs the grouping question; a trend needs the wording of the earlier wave, unchanged; a ranking needs options people can genuinely compare. Then check the survey design guide's wording rules question by question: no leading or double-barrelled wording, no internal product names, scales labelled and running in the same direction, a "don't know" or "not applicable" where some people cannot answer, and option lists that are complete and do not overlap.

  2. Read it as respondents will see it, on the devices they use

    Take the survey yourself through the test link on each kind of device the audience uses, with the most common one first. Most survey problems on phones are layout problems: grids that need sideways scrolling, long option lists that push the "next" button off the screen, sliders that are hard to set, text boxes too small to show what was typed. The guide to forms covers the field-level friction a survey shares with any form: labels, input types, error messages and accessibility. Time the survey at a respondent's pace, not a writer's, and make sure the time the invitation promises is honest.

  3. Run think-aloud sessions with people from the audience

    Ask each participant to complete the survey while saying aloud what they are thinking: what they read, what they think it means, why they choose an answer. Borrow the usability testing tutorial's rules for moderating: say that the survey is being tested, not the person; then stay quiet. Do not explain a question that confuses them, because respondents in the field will have no one to ask; the moment they hesitate is the finding. Note what each participant did and said, where they paused, what they reread and what they skipped, keeping observations apart from your interpretation of them.

  4. Probe what each question meant to them

    At the end of the survey, or at natural breaks, go back to the questions that caused hesitation and to the key questions for the decision, and ask what each meant to the participant. Useful probes are plain: "In your own words, what was this question asking?", "What did you have in mind when you chose that answer?", "Was there an answer you wanted to give that wasn't there?", "How did you decide between these two options?". Listen for terms that carry a different meaning for customers, time periods people cannot recall, options that force a choice, and questions people find intrusive. Write down the words they use; they are often the better wording. For a translated questionnaire, run these probes in the target language with a native-speaking moderator, as the guide to researching customers in a new market explains alongside back-translation.

  5. Test every route and skip

    Draw the routing as a list of paths: every answer that sends someone to a different question, and every group that should see a different block. Walk each path through the test link and check that the right questions appear, the wrong ones do not, randomisation works, required questions are required and optional ones are not, and validation accepts ordinary answers. Pay particular attention to the routes nobody on the team would take, such as the person who answers "none" or "don't know" to an early filter question, because broken routes usually hide there. Check that the data export records each answer in the right column with the right labels.

  6. Soft launch to a small part of the sample

    Send the survey to a small, random part of the people it will go to, in the same way, with the same invitation, and then pause. The soft launch tests what the sessions cannot: how the real audience behaves without anyone watching, on the real invitation and the real devices. If the sample was drawn from the business's own records, take the soft-launch group at random from the draw, as the tutorial on drawing a survey sample from a customer list describes, so the full fieldwork can carry on from the same draw. Decide in advance whether soft-launch answers will be kept if nothing changes, and record that decision.

  7. Read the soft-launch data for drop-off, speeding and 'other' answers

    Read the completion funnel question by question, the way the forms guide reads field-level drop-off and the Form Abandonment Rate: a steep fall at one question points at that question. Look at how long people took and find the ones who finished implausibly fast or answered every grid in the same column, which are the survey guide's cleaning rules applied early. Read every "other, please specify" answer: several people writing the same thing means a missing option. Look for questions where most people choose "don't know", and for answers that contradict each other. Check that the data arrives in the shape the analysis plan needs.

  8. Fix, log each change and retest

    Fix each problem the sessions and the soft launch found, and record it in the change log with the evidence and the date. Retest anything that changed: a reworded question with further people from the audience, a changed route by walking it again, a moved question by reading the funnel again if the change was large. Keep the fixes proportionate. A pretest that turns into a redesign has found a problem with the analysis plan, and that goes back to the brief.

  9. Freeze the questionnaire before full fieldwork

    Sending the full survey is a one-way door in the sense of the one-way and two-way door framework: answers already collected cannot be re-asked with better wording, so the care belongs before it. Freeze the wording, the order, the routing and the invitation, save a copy, and note the frozen version in the change log. On a tracked survey, the brand tracker tutorial's rule applies: after the baseline, change only with a logged reason and the wave it took effect in. Then launch, and keep watching the funnel in the first hours of fieldwork for anything the pretest missed.

What each check finds

Each check is good at a different kind of problem, which is why a pretest uses several rather than one. The table summarises what each tends to catch and what it cannot.

CheckWhat it tends to findWhat it cannot find
Review against the analysis planQuestions that feed no output; missing grouping questions; wording that breaks the survey guide's rulesWords the audience understands differently
Reading on respondents' devicesGrids and lists that break on phones; dishonest time estimates; hard-to-use inputsHow real respondents interpret the questions
Think-aloud sessionsHesitation, rereading, skipped questions, intrusive topicsProblems that appear only at volume or without a moderator present
Meaning probesTerms with a different meaning for customers; missing options; recall periods people cannot manageWhether a problem is common or limited to one participant
Route-by-route testQuestions shown to the wrong people; broken skips; validation that rejects ordinary answersWhether the questions are understood
Soft launchDrop-off points, speeding, straight-lining, repeated "other" answers, export problemsWhy people left; that needs the sessions
Change log and freezeUnrecorded edits; versions that drift during fieldworkNothing on its own; it keeps the other checks honest

The same discipline applies to a short survey intended for publication. The guide to campaign ideation and data stories notes that the Market Research Society and ESOMAR publish codes and guidance on sampling, question wording and reporting, and that journalists check survey methods; a pretest is the cheapest way to make sure the published wording means what the headline claims. Polls of a single question on one page are a lighter case, covered by the tutorial on on-page polls and post-purchase surveys.

A kitchen fitter pretests its lost-quote survey (illustrative)

Illustrative example — not a real result

Hexomby Kitchens pretests a survey for people who bought elsewhere

Hexomby Kitchens, an invented kitchen fitter, wants to know why people who received a quote went on to buy from someone else. The draft survey has 16 questions, routes people who bought a kitchen elsewhere to an installation block and people who did nothing yet to a timing block, and promises to take 6 minutes.

The review against the analysis plan cuts 3 questions that fed no output. On a phone, a grid of 7 kitchen features needs sideways scrolling, so it becomes a list of single questions, and the survey takes 9 minutes at a respondent's pace; the invitation is changed to say so.

Five think-aloud sessions follow, with people who took a quote in the past year. Three of them read "Did you book a design consultation?" as asking whether they paid for a visit, and answer "no" although they had a free home visit; the probes produce the customers' own phrase, "a home visit to measure up", which replaces it. The route test finds that people who chose "bought from a builder as part of wider work" skip the installation block meant for them.

The soft launch goes to 200 people from the draw. 64 start and 41 finish; 15 of the 23 who leave do so at the question asking for their exact kitchen budget. The budget question is moved near the end and changed to 5 bands with a "prefer not to say" option. 9 "other" answers name a supermarket's kitchen range, which becomes an option. Each change goes into the change log, the reworded question is checked with 2 more people, and the questionnaire is frozen. In full fieldwork, drop-off at the budget question falls to 2% of the people who reach it.

Verify it worked

  • Open the change log. Each entry should show the question, the problem found, the evidence, the change made and the date, and the log should note the frozen version.
  • Hold the questionnaire against the analysis plan. Every remaining question should feed an output, and no output should lack its question.
  • Open the survey in the build, on the most common device first. Nothing should need sideways scrolling, and the time the invitation promises should match a respondent's pace.
  • Walk every path through the test link. The right questions should appear, the wrong ones should not, and the export should put each answer in the right column with the right labels.
  • Read the funnel again if a fix was large. No single question should show a steep fall, and repeated "other" answers should now be options.

If it goes wrong

  • Participants read a question differently from your intent. The likely cause is a term that means something else to customers. Use the words they gave in the probes, then retest with further people from the audience.
  • Many soft-launch respondents leave at one question. The likely cause is that it is intrusive, hard to answer or badly placed. Reword it, move it or add a "prefer not to say" option, then read the funnel again.
  • Several "other, please specify" answers say the same thing. The likely cause is a missing option. Add it to the list and check the change with further people.
  • Respondents see the wrong questions, or the export columns are wrong. The likely cause is a broken route, usually on a path nobody on the team took. Redraw the paths, walk each one again and check the export.
  • The fixes keep growing into a redesign. The likely cause is a problem with the analysis plan, not the questionnaire. Go back to the brief before changing more.

Next steps

Common mistakes

  • Pretesting with colleagues only. They know what every term means and what each question is for, so the audience's misreadings never surface.
  • Helping in a think-aloud session. Explaining a confusing question removes the finding and leaves the question confusing for everyone in the field.
  • Testing a document, not the build. Routing, validation and phone layout live in the survey tool, and so do most of the faults.
  • Walking only the main route. Broken skips hide on the paths nobody on the team would take.
  • Launching to everyone at once. Without a soft launch, a question that drives people away does so across the whole sample.
  • Changing questions mid-fieldwork. Answers before and after a change answer different questions, and a tracked survey loses comparison with earlier waves.
  • Fixing without a log. Nobody can later say which version a respondent saw or why a question changed.
  • Treating pretest participants as testers rather than participants. They are taking part in research and are owed the same information, consent and confidentiality as respondents.

Frequently asked questions

Is asking colleagues to try the survey a pretest?

It is a useful first check, not a pretest. Colleagues catch typing errors, broken links and obvious routing faults, but they know what the business means by its own terms and what each question is for, so they rarely misread anything. The problems that spoil a survey are the ones only the audience runs into: a word that means something else to customers, a list that lacks their answer, a question they would rather not answer. Those need people from the audience, watched while they answer.

Can we change a question after the survey has launched?

Avoid it. Answers collected before and after a change are answers to different questions and cannot simply be added together, and on a survey that is repeated over time a change breaks comparison with every earlier wave. If a fault is serious enough to make the answers useless, stop, fix it, record the change and the point it took effect, and analyse the two parts separately. The pretest and the soft launch exist so that this decision almost never arises.

Do pretest participants need the same consent as respondents?

Yes. They are taking part in research, often while being watched and recorded, so the principles the site's research guides describe from the MRS and ESOMAR codes apply in full: they take part voluntarily, know who is asking and why, and their answers stay confidential unless they agree otherwise. Tell them the session tests the questionnaire, not them, and agree any recording and its deletion with whoever owns data protection.

Terms used on this page

Frameworks

Where this page is used

Also related

Sources

  1. Nielsen Norman Group articles — Nielsen Norman Group
  2. Market Research Society — MRS
  3. ESOMAR — ESOMAR

Related