Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Marketing Research › Troubleshooting

Survey Results Not Representative

Why survey findings do not match the customer base: a frame treated as the population, respondents unlike those invited, wording that led and no pretest.
Troubleshootingintermediate15 min readUpdated

First, write the population down and compare who answered with who was invited

The tutorial on drawing a survey sample from a customer list makes the point that decides most disputes: the draw decides what the results can claim, since the only people in them are customers who were both invited and willing to reply. So start with one sentence that says who the decision concerned, in terms the records can apply, such as everyone who bought in the last year or subscribers active at the start of the quarter. If that sentence was written in the brief, copy it. If it was not, write it now and notice how far it sits from "whoever was on the email list".

Then build the comparison the tutorial puts at the end of every draw: the share of the frame and the share of respondents, side by side, for each group the records can identify, such as recency, order count, spend, tenure, product or region. The survey and questionnaire design guide expects engaged and satisfied customers to respond more, so a skew is normal; what matters is its size, which groups it falls on, and whether anyone reported it. If the comparison was never built, build it from the draw record and the fields the records hold, within the data handling the study agreed, before anyone argues about a figure.

The comparison answers one question: are the respondents unlike the people invited? Two further readings narrow the rest of the table. Are the people invited unlike the population the decision concerned, because the frame never held them? And would a perfect sample have answered the question well anyway? The stated and revealed preference framework explains why a question about future behaviour misleads whoever answers it: nothing is at stake in the answer, people predict themselves poorly, and an easier question gets answered in place of the hard one. A representative sample asked whether it would enjoy a longer season is still a sample of hopes. Read the table with those three readings in hand, because a frame gap, a response gap and a wording fault need different fixes and support different claims.

Diagnosis table

Diagnosis

SymptomCheckLikely causeFix
Findings quoted as what customers or the market think, when the frame was the business's own listThe population sentence against the frame: who the list never held and who the cleaning and contact rules removedThe customer list treated as the market, so the results describe a frame with gapsRestate what the results describe, name who the frame misses near the top, and use a different frame for any claim about the market
Respondents longer-tenured, higher-spending or more satisfied than the people invitedA respondent-versus-frame table on the fields the records already holdRespondents more engaged and satisfied than the people invited, with no comparison against the frameBuild the comparison, weight on a few variables with the frame's shares as targets and report both figures
Answers leaning one way on every question, or agreement with every statement put to peopleEach question for leading, double-barrelled or loaded wording, and the order of questions and optionsLeading, double-barrelled or loaded wording and order effects that pushed answers one wayReport the affected questions with their wording and a caveat, and reword or reorder them for the next wave
A comparison the decision needs with a handful of answers on one side, or groups that appeared only after the data arrivedThe brief for the groups the decision compares, and the draw record for the strata chosen before the drawGroups the decision compares that were never represented on purpose, or strata chosen after the data arrivedSay which comparisons the sample cannot make, mark any new cut as exploratory, and stratify on purpose next time
Weighted and unweighted figures far apart, or a group's result swinging on a few respondentsThe weighting variables, the largest weights and the counts behind each groupWeighting on many variables or heavy weights that make small groups drive the resultWeight on a few variables, show counts and ranges beside every figure, and hold back groups too thin to read
A result that contradicts what the records show, or "other" answers that restate the questionWhat the question meant to a few people from the audience, asked to think aloudRespondents who misread a question because it was never pretested with the audienceDrop or caveat the misread question and pretest its replacement before the next wave

Fix, verify and prevent each cause

The customer list was treated as the market

The sampling tutorial is explicit about what a list can and cannot represent. Results from the business's own records hold for the customers on that list, and only if the draw was planned and the respondents checked against the frame; they say nothing about the wider market, about buyers who went to a competitor or about anyone the list never contained. Every frame leaves people out: non-customers, customers whose details have gone stale and, depending on the rules applied, customers who opted out or were surveyed recently. The design guide puts the same fault in its list of common mistakes, results from engaged subscribers reported as if they described all buyers.

Fix: rewrite the finding to say what it describes, such as recent buyers on the list rather than customers or the market, and move the frame's gaps to the first page beside the answer they qualify. Where the decision needs a claim about the category, use a frame that reaches it: the tutorial on setting up a brand tracker samples category buyers from a panel through a screener rather than surveying the brand's own customers.

Verify: a reader who sees only the summary can say whom the result describes and whom it leaves out, and no sentence in the report extends the finding beyond the frame.

Prevent: write the population sentence in the brief before the draw, as the tutorial's first step requires, and keep the counts each cleaning and contact rule removed, because a large removal is a finding about the list that belongs in the report.

Respondents were more engaged than the people invited

The survey design guide calls this non-response bias: the people who answer differ from those who do not, and engaged and satisfied customers tend to respond more. Its remedy is to check respondents against the frame on what the records already show, such as tenure, spend, region or product, and to weight or caveat where they differ. The tutorial on testing whether survey scores predict churn shows the sharpest form of the problem: if the customers who did not answer leave more often than any score band, the survey is missing the customers most at risk, and a figure that looks sound among respondents covers few of the people who matter.

Fix: build the respondent-versus-frame table on the fields the draw exported, then weight on a small number of variables that bear on the answers, with the frame's shares as the targets, and report the weighted and unweighted figures together wherever they differ, as both guides ask. Where a group is too thin to read, report it as such instead of giving it a heavy weight.

Verify: the two figures sit in the report side by side with the comparison table, and the direction of the finding holds under both, or the report says that it does not.

Prevent: plan the comparison before fieldwork by exporting the comparison fields with the draw, and release reserve invitations in the strata that answer less, which the sampling tutorial notes adds answers without removing the bias, so the comparison is still needed afterwards.

The wording or the order pushed answers one way

Small wording changes move results, and the design guide says most survey bias is introduced by the people writing the questions rather than the people answering them. It names the faults: a leading question that contains the expected answer, a double-barrelled question that asks two things at once, loaded or unfamiliar terms, acquiescence towards agree-or-disagree statements, social desirability on sensitive topics, and order effects where earlier questions frame later ones and items near the top of a list are chosen more. The professional codes the site's research guides describe from ESOMAR and the Market Research Society expect findings to be reported with their limits, so a question known to lean one way is reported with its wording and a caveat rather than quietly, and the leading question entry summarises the first fault.

Fix: go through the questionnaire question by question with the guide's list, mark every question that fails it, and report those questions with their exact wording and a caveat that the figure is likely to lean the way the wording pointed. Reword them neutrally for the next wave, split the double-barrelled ones, balance or replace agreement statements, and randomise option order except where the order is natural.

Verify: a colleague who did not write the survey reads each rewritten question and cannot tell which answer the team hoped for, and the next wave's figures on the reworded questions are reported as a new series rather than compared with the old.

Prevent: write questions in the words customers use, which interviews supply, and read every draft against the guide's wording and order rules before it reaches the pretest.

The groups the decision compares were never represented on purpose

The research design guide asks the brief's audience and sample section to say who the study must hear from, the sample frame and its known gaps, and warns against convenience samples presented as representative, where the researcher knows the caveat and the reader never sees it. Its own example of a legitimate design is a sample drawn from existing customers that under-represents people who have never bought, with findings about non-customers stated as directional only. The sampling tutorial adds the other half: it builds the draw around the groups the analysis will compare, chosen before the draw, because strata chosen after the data came in are groups drawn to fit a story rather than the decision, and the survey analysis plan template marks any comparison added once the data is in as exploratory, since a pattern spotted while browsing the data counts for less than one the plan was built to test.

Fix: list the comparisons the decision needs, check each against the draw record for a stratum chosen before the draw and the count that answered, and state plainly which comparisons the sample can make and which it cannot; label any group invented after fieldwork as exploratory.

Verify: every comparison in the report names its two bases and traces to the brief or is marked exploratory.

Prevent: write the groups into the brief, stratify on fields the records already hold, as the tutorial does with the business's existing Segmentation, and plan the invitations per stratum from the smallest group the report must read.

Weighting let a handful of respondents drive the result

The design guide limits weighting to a few variables that matter to the answers, because weighting corrects imbalances only on the variables weighted and heavy weights make results less stable. The guide to reading small numbers explains the mechanism: the number that controls how much a figure can move by chance is the smaller count behind it, the most extreme figures in any list come from the smallest groups, and a few respondents carrying large weights become exactly such a group. A regional split weighted up from a handful of answers is a league table of noise.

Fix: cut the weighting back to a few variables with targets from the frame, show the counts behind every rate and a range around the figures that drive the decision, and hold back any group the team's own minimum count rule would not let it read alone, offering a pooled view instead.

Verify: weighted and unweighted figures move together rather than apart, and no reported difference between groups rests on a count the report would flag.

Prevent: fix the weighting variables and the small-group rules in the analysis plan before fieldwork, as the template's sample and weighting field asks, so the choice is never made after seeing which version looks better.

A question was misread because nobody pretested it

The tutorial on pretesting a questionnaire explains why internal review cannot catch this fault: the authors know what each question is meant to ask and read it that way, while respondents read it as written, with their own meaning for every word. Its example of a question about a design consultation that customers read as a paid visit is the shape of the problem; the answers were honest and wrong.

Fix: put the suspect question in front of a few people from the audience, ask them to think aloud as they answer and then ask, in their own words, what the question was asking; if their reading differs from the team's, drop the figure or report it with a plain caveat, and read the open answers around it, coded with a codeframe, for what people were actually telling you.

Verify: the rewritten question, phrased in the words the sessions produced, is checked with further people from the audience and no longer draws answers that contradict the records.

Prevent: run the tutorial's full pretest before every launch, from the think-aloud sessions and meaning probes to a soft launch read for "other" answers, and freeze the questionnaire so the next wave measures the same thing.

Prevention: a recorded draw, a pretest and a respondent-versus-frame table in every report

Most of this page is prevented by three habits, each of which already belongs to a tutorial. The draw is written down: the population, when the list was exported, each cleaning step and what it removed, the strata, the invitations and reserves sent to each, and how the random order was made, all filed with the brief rather than with the data export, since the export is eventually deleted and the record is what later waves rely on. The questionnaire is pretested with the audience and frozen. And the report carries a table comparing respondents with the frame, with weighted and unweighted figures beside it. A reader who can see those three things can judge the result for themselves, which is the point.

The survey analysis plan ties the habits to the decision before anything is fielded: the decision and population copied from the brief, the comparisons that matter, the strata, the weighting variables, and the rules for small groups and uncertainty, written down so that none of them is chosen after the data is in. The sample size follows from the smallest group the report must compare, not from the total, which is where most plans that produce an unreadable comparison went wrong.

Two further checks belong in any programme that repeats its surveys. A tracked survey needs the same frame and the same draw every wave, because a wave drawn differently is a different measurement; the brand tracker tutorial keeps a change log for exactly that reason, and the sampling tutorial's draw record does the same job for a list. And a score that is meant to warn of behaviour should be checked against behaviour in the business's own data, as the churn tutorial does, which also shows whether the people who never answer are the ones who leave.

What the report may still say is the subject of the guide to reporting research findings: the answer first, each finding with its base and the exact question wording, the limits near the top rather than in a footnote, and words such as "too few to tell" where they apply. A representativeness problem found by a sceptical reader after the decision costs the next study its trust as well, so the limits go in before the reader finds them; if that has already happened and the study now changes nothing, the diagnosis for research findings that are not acted on is the next page. When the shortfall is also in the number of answers, the diagnosis for a low survey response rate reads the completion funnel first, and the rest of the customer research section holds the methods this page leans on. Other research symptoms are collected in the marketing research troubleshooting section.

A boat hire company finds its results described its regulars (illustrative)

Illustrative example — not a real result

Tavrenqo Boat Hire

Tavrenqo Boat Hire, an invented canal boat hire company, surveys customers who hired in the past 2 years about extending its booking season by 6 weeks and reports that 72% support it. The finance director, who has watched off-season bookings for years, does not believe it.

The population sentence is written down: everyone who hired at least once in the past 2 years, 3,100 people. The draw record shows 1,800 invited and 410 answers. The respondent-versus-frame table explains the finance director's doubt. Customers with 3 or more hires are 18% of the frame and 46% of respondents; first-time hirers are 55% of the frame and 24% of respondents, so the regulars answered at several times the rate of the people the decision most concerns. The question itself asked, "Would you enjoy a longer boating season?", a question about a hope with nothing at stake. And the regional split rested on a weight of 4.3 applied to one region's 9 respondents, which is why that region appeared to want the extension most.

The report is rewritten. Weighted to the frame's shares by hire count, support is 58% against 72% unweighted, and both figures appear with the table beside them. The regional comparison is withdrawn as too thin to read, with the 9-person base shown. The first page states that the survey describes customers on the list and says nothing about people who hire elsewhere. For the next wave the question becomes "When did you last book outside the main season, and what decided it?", the draw is repeated from the same record with 500 reserve invitations held for first-time hirers, and the analysis plan fixes the weighting variable and a minimum of 40 answers before any group is reported alone.

Common mistakes

  • Defending the chart instead of checking the frame. The argument is settled by the comparison table, not by the confidence of the presenter.
  • Treating the email list as the market. The list is a frame with gaps, and the report has to say who it misses.
  • Reporting only the weighted figure. Readers need both versions and the comparison to judge how far respondents differ.
  • Weighting a handful of people up to a region. Heavy weights on a small group make that group the result.
  • Cutting the data by groups invented afterwards. A comparison the brief never asked for is exploratory and should be labelled as such.
  • Asking people to predict themselves. A representative sample of hopes is still hopes; ask about the last real episode.
  • Finding the limits from the finance director. State the frame's gaps on the first page before a reader does it for you.

Frequently asked questions

Can we still use the results if the respondents do not match the people we invited?

Yes, as a description of the people who answered, which is what every survey is. Compare respondents with the frame on the fields the records hold, weight on a few variables that matter to the answers using the frame's shares as targets, and show the weighted and unweighted figures together. Say near the top of the report who the frame missed and which groups answered too thinly to read. Whether that is enough depends on the decision: a reversible, low-cost decision can rest on a caveated result, while one that commits a large budget or a target for the year deserves a better draw first.

Would a bigger sample have fixed this?

No. More invitations add answers from the same kinds of people, so a frame with gaps stays a frame with gaps and engaged customers still answer first. The number of answers governs how precisely a group can be read, which is why it is planned from the smallest group the report must compare, but it says nothing about whether those answers resemble the people who did not reply. Representativeness comes from the frame, the draw and the comparison afterwards, not from the count.

Should we weight the results or not?

Weight when respondents are clearly skewed on a variable that matters to the answers and the frame gives a target to weight to, and keep it to a few variables. Weighting corrects imbalances only on the variables weighted, heavy weights make results less stable, and a group answered by a handful of people should be reported as too thin to read rather than weighted up. Decide the weighting in the analysis plan before fieldwork, so the choice is not made after seeing which version looks better, and always report both figures when they differ.

Terms used on this page

Frameworks

Where this page is used

Sources

  1. Market Research Society — MRS
  2. ESOMAR — ESOMAR

Related