Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Glossary term

Non-Response Bias

The gap between the people a survey or study invited and those who answered; engaged, satisfied customers tend to reply more, so results lean their way.
AnalyticsconceptUpdated

Definition

Non-response bias is the difference between the people a survey or study invited and the people who answered. The research methods hub names it as one of three biases that do the most harm in marketing research, each a different gap:

  • Selection bias lies between the people a decision is about and those the study reached.
  • Non-response bias lies between those the study reached and those who chose to reply.
  • Measurement bias lies between people's answers and the truth.

The middle gap is the one a team can least see from inside the data: replies arrive and the report looks complete. Yet answering is a choice, and the people who make it are rarely a random slice of those asked. The guide to survey and questionnaire design gives the usual direction: the customers who are engaged and content are the likeliest to reply, so an unchecked result leans their way.

Why it grows as response falls, and what weighting can fix

The methods hub adds that the bias grows as response falls. The survey design guide spells out what any survey result describes: the customers who chose to reply, who tend to be keener and more favourable than the customer base overall.

More invitations do not cure it. The tutorial on drawing a survey sample from a customer list makes the point about its reserve sample: extra people invited in the same way produce more answers from the same kinds of people. Inviting the whole list has the same limit, because sending to everyone does not make everyone answer.

Weighting helps, within a boundary. It adjusts results so that each group counts in proportion to its known share of the population, but it corrects only the variables it weights on. If newer customers answered too little and are weighted up, the result is corrected for tenure. It is not corrected for whatever else made the newer customers who did answer different from the newer customers who did not.

In practice: comparing respondents with the frame

The check the guides use costs almost nothing, because the business already holds the evidence. Once fieldwork ends, set the respondents beside the list they came from, using fields the records already hold, such as tenure, spend, recency, order count, region or product. For each group, put its share of the list next to its share of replies and see where the two part company.

What follows from the comparison is a set of reporting decisions rather than a fix:

  • Where respondents are skewed, weight on a small number of variables likely to change the answers, and publish the weighted and raw figures side by side whenever they disagree.
  • Where a group returned too few replies to support a reading, report that plainly rather than inflating it with a large weight.
  • For the next wave, invite proportionally more of the groups that answer less, which the sample tutorial plans stratum by stratum.

The same gap opens in qualitative work by another route. The guide to recruiting research participants warns that recruitment which takes whoever answers first fills sessions with the customers who answer everything, rarely the people whose behaviour the decision turns on, and asks for a written line on the people each source leaves out, so the findings are clear about whom they describe.

Illustrative example — not a real result

A dental group finds its newer patients barely answered

Hontaeil Dental, a group of 3 dental practices, sent a satisfaction survey to all 8,200 patients on its list, and 1,150 replied. Booking satisfaction came out at 87%. Before reporting, the practice manager compared respondents with the list by how long each patient had been registered. Patients of more than 5 years made up 44% of the list but 72% of respondents; patients registered within the last 2 years were 30% of the list and only 9% of respondents, 104 people in all. Among those 104, booking satisfaction was 61%. The report showed weighted and unweighted figures, stated that newer patients could be read only roughly, and the next wave invited newer patients at twice the rate of the rest.

A larger Sample Size narrows the margin around a result but leaves its lean in place: more replies from the same kind of people are still replies from the same kind of people. A Screener confirms that each research candidate fits the study; it cannot show who never applied, which is why recruitment sources are written down. A Leading Question produces measurement bias, a separate gap, so a survey can suffer from both at once and fixing the wording does nothing for the response pattern. A Soft Launch sends the survey to a small random slice first, and an early look at who is answering there can show a group falling silent before the full send.

Common confusions

  • A big response means no bias. The count says how precisely the respondents were measured, not whether they resemble the people asked.
  • Inviting everyone removes it. Everyone was asked; not everyone answered, and the people who did still chose themselves.
  • Weighting fixes it. Weighting corrects the variables weighted and nothing else, so unweighted figures and the comparison with the list still belong in the report.
  • It is the same as selection bias. Selection bias concerns who was reached at all; non-response bias concerns who, among those reached, replied.
  • A low response rate proves the result wrong. It makes the bias more likely and the check more important. The comparison with the list, not the rate alone, shows how far respondents differ.

More terms to look up

Where this term is used

Related