What a sample from a list can and cannot represent
The survey design guide sets out the ideas this tutorial puts into practice. The population is the group the decision concerns. The frame is the list that reaches them, and every frame misses someone: a customer list misses everyone who never bought, customers whose records are out of date and, depending on the rules applied, customers who cannot be contacted for research. Non-response biasThe gap between the people a survey or study invited and those who answered; engaged, satisfied customers tend to reply more, so results lean their way. is the gap between the people invited and the people who answered, and engaged, satisfied customers tend to answer more. Weighting adjusts results towards known population figures on the variables weighted, and on nothing else.
A sample drawn from a customer list can therefore speak for the customers on that list, provided the draw is planned and the respondents are checked against the frame. It cannot speak for the market, for people who chose a competitor, or for people the list never held. The methods hub describes the selection, non-response and measurement biases that every sample carries; this tutorial is about keeping the first two small and visible.
| Claim the report wants to make | Can a sample from the customer list support it? | What it needs |
|---|---|---|
| How recent buyers rate the delivery experience | Yes | A frame of recent buyers and a check of who answered |
| Whether lapsed customers left over price | Partly | Lapsed customers in the frame, and an honest note on how few answer |
| How satisfied customers are overall | Yes, with care | Strata for the groups that differ, and weighting where respondents are skewed |
| Why people in the category buy from rivals | No | A different frame, such as a panel, as the brand tracker tutorial uses |
| What share of the market knows the brand | No | A sample of category buyers, not of the business's own customers |
Other frames have their own pages. The tutorial on setting up a brand tracker samples category buyers from a panel, and the tutorial on on-page polls and post-purchase surveys intercepts visitors on the site. A brand trackerA survey that asks the same questions of a fresh sample of category buyers in repeated waves, so changes in awareness and associations show over time. needs the same frame in every wave, because its waves are compared with one another. Interviews and group sessions are recruited rather than sampled, as the recruitment tutorial shows.
Before you start
Have the brief and the analysis plan in front of you: the decision, the figures that will inform it and the groups the analysis will compare. The groups matter most, because the draw is built around the smallest group the report must read.
Find out who owns data protection and settle the following with them before exporting anything: whether the customer records may be used to invite people to research at all, since they are Personal DataAny information relating to an identified or identifiable living person, such as a name, email address, location data or an online identifier. and their use is that person's decision, not the survey owner's; whether people who opted out of marketing are to be invited, with exclusion as the default; and when the extract will be deleted.
Find the business's contact rule. The voice of the customer guide recommends a rule that a customer surveyed recently is not surveyed again for a set period, whichever team wants to ask; if no rule exists yet, agree one with the teams that send surveys before drawing this sample.
Gather what the business already knows about its past surveys: how many people were invited in each group and how many answered. That history, not a figure from elsewhere, is the starting point for how many to invite.
The steps
Write down the population the decision concerns
Write one sentence that says who the results must describe, in terms the customer records can apply: "everyone who bought at least once in the last year", "subscribers active at the start of this quarter", "customers who ordered the new range". Add who is deliberately left out and why. The sentence stops a common drift, in which the population quietly becomes "whoever is on the email list", and it gives the report its first caveat. If the decision concerns people the records do not hold, such as prospects or a competitor's customers, a customer list is the wrong frame and the survey design guide's other frames apply.
Export the frame with only the fields the draw needs
Export the records that match the population, and only the fields the draw and the later comparison need: a customer ID, the contact field the invitation will use, the fields for the strata, the fields you will compare respondents on, and the flags that say who must not be contacted. Leave out everything else. This is the GDPR's data minimisation and purpose limitation as the GDPR glossary entry summarises them: data limited to what the purpose needs and not reused for an incompatible one. Store the extract where only the people running the survey can reach it, note its deletion date beside it, and delete it when the survey closes.
Remove duplicates, staff, test records and anyone who must not be contacted
A frame inflated by records that are not customers distorts every share computed from it. Remove duplicate accounts for the same person, which is DeduplicationRemoving double counts when one conversion is reported by more than one source. in the sense of making one real customer count once (the deduplication entry describes the same idea for conversions), matching on email address and, where the records allow, on name and delivery address. Remove staff, test and demonstration accounts, and records that belong to a different population, such as trade, wholesale or reseller accounts in a consumer survey. Then remove everyone the business must not contact: people on the suppression list, anyone who asked not to be contacted by any channel, and records with no usable contact details. Count what each rule removes and keep the counts, because a large removal is a finding about the list, and the report should say what the frame lost.
Apply the contact rules and the research footing
Take out everyone surveyed or interviewed within the contact rule's period, across every team that sends surveys. Then apply the decision agreed with the data protection owner on people who opted out of marketing, excluding them unless that owner has said otherwise. The invitation that follows must be research and nothing else. As the marketing research hub notes, an invitation that promotes a product is marketing whatever its subject line says, and the comparison of promotional and transactional email shows how purpose, not the label, decides what a message is. The list building and consent guide explains the consent records behind the marketing flags you are reading. Note how many records each rule removed and who they were.
Choose strata from fields the list already holds
Stratifying means dividing the frame into groups and drawing from each one separately, so that every group the decision compares is represented on purpose rather than by chance. Choose strata from fields the records already hold, and only fields that are likely to change the answers. Recency, order count and spend are the usual candidates, and the RFM segmentation framework shows how to band the order history into recency, frequency and value groups. Product line, plan, channel or region can serve too, and the SegmentationDividing a market or customer list into groups that need different offers or messages. the business already uses is a sound source of dividing lines. Keep strata few: every extra split shrinks the groups and makes the draw harder to repeat.
Decide how many to invite in each stratum
For each stratum, decide how many answers the report needs, then how many invitations that implies. The number of answers comes from how precisely the group must be read: the sample size and significance guide explains how precision depends on the count, and the sample size entry summarises the idea for tests. The share who answer comes from the business's own past surveys of similar customers, read as the guide to reading small numbers advises: as a range rather than a single figure, pooled across past waves where one survey is too small. Groups that answer less, often lapsed and infrequent customers, need proportionally more invitations. Where a stratum is too small to supply the answers it needs, invite all of it and say in the report that the group can be read only roughly.
Draw at random within each stratum and set aside a reserve
Within each stratum, select at random: give every record a random number, sort by it and take the records from the top. Never take the first records in the export, the most recent sign-ups or the most active accounts, because each of those orders carries a bias of its own. Take a further set from the same sort as a reserve and hold it back. If a stratum answers far less than expected, release reserve invitations in that stratum only, and record when and how many. A reserve adds answers; it does not fix non-response, because the extra people are invited in the same way and the same kinds of people still answer first. Before any reserve is released, the diagnosis for a low survey response rate reads the completion funnel to check whether the shortfall is the frame at all, rather than the questionnaire, the invitation or delivery.
Record the draw so the next wave can repeat it
Write the draw down while it is fresh: the population sentence, the export date, the fields exported, every cleaning and contact rule with the count it removed, the strata and their bands, the invitations and reserves per stratum, how the random order was produced and any reserve released. Keep the record with the brief, not with the extract, because the extract will be deleted and the record should outlive it. A later wave that follows the same record can be compared with this one; a later wave drawn differently is a different measurement, a point the brand tracker tutorial makes about its own change log. For a repeated customer survey, the customer survey wave SOP starts each later wave from this record.
After fieldwork, compare respondents with the frame
When the survey closes, compare the people who answered with the frame on the fields exported for that purpose: recency, order count, spend, tenure, product, region. Show the share of the frame and the share of respondents for each group side by side. Where respondents are skewed, the survey design guide's rule applies: weight on a few variables that matter to the answers, using the frame's shares as targets, and report weighted and unweighted figures together when they differ. Where a group answered too thinly to read, say so rather than weighting it heavily. Then delete the extract, keep the record and add the respondents' survey date to the contact history so the contact rule protects them next time. If that comparison shows respondents unlike the people invited, or a reader disputes the findings, the diagnosis for survey results that are not representative separates a frame gap from non-response, wording, weighting and a missed pretest.
Frame problems and what each does to the result
Most of the damage in a list sample is done before anyone is invited, by records that should not be there or people who are missing. The table names the common problems, what each does to the figures and the step that deals with it.
| Frame problem | What it does to the result | Where it is handled |
|---|---|---|
| Duplicate accounts for one person | That person has more than one chance of selection, and frequent buyers with several accounts look like separate customers | Remove duplicates |
| Trade, wholesale or reseller accounts in a consumer list | Their buying pattern inflates order counts and spend, and their answers describe a business, not a household | Remove records outside the population |
| Staff, test and demonstration accounts | Insiders answer, and test records distort the frame's shares | Remove staff and test records |
| Out-of-date contact details | Invitations never arrive, and the loss falls mostly on long-standing or lapsed customers | Remove unusable records and report the loss |
| People excluded for marketing opt-outs | A group that may feel differently about the brand is absent from the frame | Apply the research footing; note the exclusion in the report |
| People surveyed recently by another team | Fatigue lowers response, and the same voices recur across studies | Apply the contact rules |
| Strata chosen after the data came in | Groups are drawn to fit a story rather than the decision | Choose strata before the draw |
| An undocumented draw | The next wave cannot be compared, and nobody can say what the first one measured | Record the draw |
Before the invitations go out, the survey launch checklist gathers these frame checks with the pretest, invitation and consent lines into one gate to tick for each survey or wave.
A tea retailer samples its customers (illustrative)
Pelzorin Teas draws a sample from last year's buyers
Pelzorin Teas, an invented online loose-leaf tea retailer, wants to know whether to change its delivery options and how buyers who ordered once differ from regular buyers in what they value. The population is everyone who placed at least one order in the past 12 months.
The export holds 18,400 records with a customer ID, email address, last order date, order count, total spend and the marketing flags. Cleaning removes 610 duplicate accounts, matched on email address and delivery address; 140 wholesale cafe accounts that had been set up on the consumer site and account for a large share of orders; 25 staff and test accounts; and 1,020 people on the suppression list. The data protection owner excludes a further 2,300 who opted out of marketing, and the contact rule removes 900 people surveyed by the service team in the last 6 months. The frame is 13,405 customers.
The team stratifies by recency (ordered in the last 3 months or earlier) and order count (one order, two to four, five or more), giving 6 strata. Past surveys suggest roughly 15% of regular buyers and 6% of one-time buyers answer, so one-time strata get more invitations. The draw invites 3,200 customers and holds back 600 as a reserve; 300 reserve invitations are released in the two one-time strata after the first week.
527 people answer. Customers with five or more orders are 9% of the frame but 24% of respondents, and one-time buyers are 58% of the frame but 37% of respondents. The report weights to the frame's shares by order count and shows both versions: interest in a slower, cheaper delivery option is 41% unweighted and 52% weighted, because one-time buyers want it more. It states that wholesale cafes and customers who opted out of marketing are not covered, the extract is deleted when the survey closes, and the draw record goes into the brief for next year.
Verify it worked
Check the draw before any invitation goes out, then check the answers against the frame.
- Read the population sentence to a colleague. They should say who is in the survey and who is left out.
- Subtract every removal count from the export. The result should equal the frame, as it does in the illustrative example above.
- Hand the draw record to someone uninvolved. They should follow every rule and count without asking you.
- After fieldwork, set each group's share of the frame beside its share of respondents. In the illustrative example above, regular buyers are a small share of the frame but a much larger share of respondents.
If it goes wrong
- The frame is far smaller than the export. A rule removed more than you meant, or the export matched the wrong population. Re-read each removal count against the population sentence.
- A stratum answers far less than expected. Release reserve invitations in that stratum only and record how many. A reserve adds answers but does not cure non-response.
- Respondents look unlike the frame. Weight on a few variables that matter, using the frame's shares, and report weighted and unweighted figures together.
- The next wave cannot be compared with this one. The record has a gap. Record what is still known and tell readers the waves were not drawn the same way, so they are not comparable.
Next steps
- Read the survey design guide to write the questionnaire and review the weighting rule used after fieldwork.
- Read reading small numbers to judge whether a thin stratum can be read at all.
- Read the voice of the customer guide to agree one contact rule across every team that surveys customers.
Common mistakes
- Treating the email list as the population. The list is a frame with gaps, and the report has to say who it misses.
- Exporting everything. A full customer export for a survey that needs a handful of fields breaks data minimisation and leaves a copy nobody remembers to delete.
- Leaving trade and duplicate accounts in. A few heavy buyers and doubled customers can move every share the survey reports.
- Ignoring other teams' surveys. Without a shared contact rule, the same engaged customers are asked again and again.
- Inviting in export order. The first records in an export are rarely random, and the bias travels into the results.
- Treating the reserve as a cure. More invitations add answers from the same kinds of people; they do not remove non-response bias.
- Reporting only weighted figures. Readers need both versions, and the comparison with the frame, to judge how far the respondents differ.
- No record of the draw. A second wave drawn differently cannot be compared with the first.
