First, read two transcripts for how many questions asked about a specific past event
Before deciding what went wrong, take two transcripts, one from a session the team thought went well and one it thought went badly, and sort every question the interviewer asked into three piles: a specific past event ("tell me about the last time you looked for a tutor"), opinions and preferences ("what do you think of the service?") and the future ("would you use something like this?"). The customer interviews tutorial is built on the first pile, because what someone did on a particular occasion is evidence and what they predict is optimism, and the jobs-to-be-done framework gives those questions their shape: the moment the status quo stopped being acceptable, what the person tried first, what nearly stopped them, who else was involved and what tipped the decision. If the past-event pile is the smallest, the guide is the cause, whoever was in the room.
Then compare the participants with the screener. Open the recruitment record and describe the participants by behaviour: when they last did the thing the decision concerns, whether any of them left, lapsed or chose a rival, and whether any quota went unfilled. The tutorial on recruiting research participants is explicit that recruitment decides what research finds, and that loyal customers who answer emails are rarely the people the decision turns on. If every participant is an engaged customer who replied to a newsletter, the round could only ever have produced praise.
Two further readings narrow the diagnosis. Where the participants were business buyers, list the roles interviewed against the roles the decision needs; a list holding only the champion at each account is its own cause. And look for any synthesis document at all, a coded table or a theme list with excerpts: if the only artefact is a slide of quotes, the sessions may have been fine and the work after them never happened.
Diagnosis table
Diagnosis
| Symptom | Check | Likely cause | Fix |
|---|---|---|---|
| Every participant is positive, and all of them are current customers who answered an invitation | The recruitment record against the brief: how each person was described, which frame they came from, which quotas were filled | Participants recruited from whoever answered, so the room is full of loyal fans | Recruit lapsed, lost and prospective people through a neutral screener from a frame that matches the question |
| Transcripts hold opinions and predictions but no reconstructed episode | The share of the interviewer's questions that asked about a specific past event | Questions about opinions and the future instead of specific past behaviour | Rebuild the guide around the last time, the trigger, the alternatives and the switch, and pilot it |
| Business-buyer sessions all sound like a satisfied account review | The roles interviewed at each account, and who chose the participants | Business buyers reached only through account owners, so only the champion is heard | The researcher draws the list by role; account owners introduce and never nominate or attend |
| The readout is a slide of quotes and the best stories, with no theme traced to evidence | Whether any coded table or evidence table exists | No synthesis step, so themes never emerge from the transcripts | Code the transcripts against a codeframe and build an evidence table before any report |
| The team argues about how many participants said what and whether that is enough | How findings are phrased: as reasons, or as tallies and shares | Interviews read as counts, so the team argues about prevalence a dozen conversations cannot give | Report themes as reasons with their breadth described in words, and send prevalence questions to a survey |
| One session produced a single view that everyone in the room repeated | Where in the transcript the discussion turned on one person's comment, and whether private reactions were recorded | A group session where one voice led, or a group used where an interview was needed | Read the group by person and by moment, and move individual-journey questions to one-to-one interviews |
Fix, verify and prevent each cause
Participants recruited from whoever answered
The recruiting tutorial names the drift precisely: every recruitment choice either keeps the study pointed at the people the decision concerns or lets it slide towards whoever was convenient, and the slide is invisible in the findings unless someone wrote it down. A newsletter invitation reaches only the customers who still read the newsletter, and an incentive paid as a discount on the product attracts people who already like it; the customer research hub makes the same point about loyal customers who answer emails.
Fix: describe the participants by what they did and when, choose a frame that reaches that behaviour and note in the brief who it misses, set a quota for each group the decision compares, and recruit through a screener that asks about behaviour with a full list of options and asks when someone last did something rather than whether they ever do. Pay the incentive as a voucher or a charity donation, as the hub advises.
Verify: read the completed screeners of everyone booked against the profile and the quotas, confirm the qualifying facts at the start of each session, and where a quota went unfilled, say so in the findings rather than filling it from another group.
Prevent: keep the recruitment log apart from the findings so the next study can see who was asked, and treat an unfilled quota or a frame that reached only engaged customers as a reason to pause before fieldwork rather than a footnote after it.
Questions about opinions and the future
The interviews tutorial lists asking about the future as its first mistake: "would you use it" and "how much would you pay" produce optimism, not evidence, and asking what people think of a service produces politeness. Its discussion guide spends most of the session on one specific episode, reconstructed in order, and the jobs-to-be-done framework describes the same conversation as a switch interview walked back from the moment of decision; a guide that asks for stories produces jobs, and one that asks for wants produces wish-lists. The discussion guide glossary entry describes the plan the tutorial's template fills in.
Fix: rewrite the guide so that after a short warm-up every section asks about a particular occasion: what was going on when the need arose, what they tried and compared first, what nearly stopped them and what finally decided it; keep hypothetical questions and the business's own product to the end, if at all; and in the session ask one question at a time, ask for specifics such as dates and the words someone used, and bring every "I always" back to a particular time. Watch for the leading questionA survey or interview question whose wording contains or suggests its expected answer, producing measurement bias rather than evidence. too, the one that contains its own answer.
Verify: pilot the guide with someone who does not work on the subject, as the research session readiness checklist asks, then sort the pilot's questions into the same three piles and confirm the past-event pile is the largest.
Prevent: keep the guide template beside the brief, write down before fieldwork what finding would change the team's mind, and summarise each session straight after it.
Only the champion is heard
The guide to researching business buyers explains why business-buyer rounds so often sound like a satisfied account review. The buyer is a group, so hearing one role means hearing one part of the decision; account owners asked for participants offer their best relationships, which adds a sales filter to the loyal-customer problem; and a participant with their supplier in the room is managing a relationship rather than describing one.
Fix: draw the list of accounts and roles from records against the brief, plan interviews per role, ask account owners to introduce the study to the people on the list rather than to nominate anyone, record every exclusion an account owner asks for and why, keep account owners out of the sessions unless the participant agrees, and reach lost prospects through lost-deal records on a research footing only.
Verify: before the readout, list the roles interviewed at each account against the roles the decision needs, and check that every requested exclusion is named in the findings.
Prevent: tell participants in the invitation that taking part never triggers sales follow-up, keep research notes out of the account record, and keep one record of who at each account has been asked for research, references or advisory input, so that one team's study does not land on another's request.
No synthesis step
Synthesis turns transcripts into a short list of themes with their evidence, and the guide to using AI for research synthesis sets out the sequence whether or not a model helps: become familiar with the material, build a codeframeThe defined set of codes open-text answers are tagged with, built from a sample, applied consistently, checked by a second coder and counted against a base. naming what to look for, code passage by passage, group codes into themes, weigh what supports each and write what it means for the decision. The output is an evidence table before it is a slide, each row holding a theme, its definition, the participants it comes from, the supporting and contradicting excerpts with their source codes, and the implication. Skip the step and the readout defaults to the stories people remember, usually the most articulate participant's.
Fix: have a person read a sample of transcripts in full and draft the codes, apply them to every transcript with each excerpt copied word for word beside its participant code, and build the evidence table from that coded output rather than from anyone's summary of it. A theme resting on one participant is reported as a single voice, never as a pattern.
Verify: every theme traces to excerpts a reader can find by participant code, every quote has been found word for word in its source, and a second person has coded a random selection of passages blind and compared.
Prevent: block synthesis time when the sessions are booked, because the interviews tutorial warns that note-taking and synthesis take longer than the sessions themselves, and have two people synthesise together.
Interviews read as counts
The comparison of qualitative and quantitative research lists counting interviews among its mistakes: a tally from a purposive handful presented as a proportion. Qualitative research cannot say how common anything is; that is the job of a survey with an adequate sample sizeThe number of users or sessions an experiment needs to detect a given effect reliably. and a sample that stands for the population, and the comparison's sequence, explore, then measure, then explain, puts that survey after the interviews that found the options worth asking about. Read as counts, interview findings start an argument nobody can win: one side says most participants mentioned price, the other that a dozen conversations prove nothing. The guide to reporting research findings settles it by describing breadth in words, a theme across most sessions or from a single participant, never as a percentage.
Fix: rewrite each finding as a reason or pattern worth acting on or measuring, with its breadth in words and the excerpts beside it, and where the decision depends on how many customers share the view, propose the survey that will measure it.
Verify: read the readout for any share, percentage or "most customers" that rests on the interviews alone, and replace each with the breadth among the people spoken to.
Prevent: mark in the brief which questions are about why and which about how many, and when the survey follows, give it its own frame and checks, because the diagnosis for survey results that are not representative describes what happens when a measuring instrument inherits the interviews' convenient sample.
A group where one voice led, or a group where an interview was needed
The tutorial on running a focus group is clear about what a group shows and what it hides. It shows how people talk about a category among peers and react to stimulus together; it hides private behaviour and individual decision journeys, because nobody reconstructs the evening they nearly cancelled in front of strangers. Groups also bring their own effects, such as agreeing with the confident voice, which is why the tutorial has everyone write a first reaction in private before any discussion and analyses by group and by person, never by show of hands.
Fix: if the question was about an individual's journey, move it to one-to-one interviews, which the tutorial calls the safer default where the method could go either way. If the group was the right method, re-read each transcript for the moments where the discussion turned on one person's comment, weigh any theme that appeared only after that moment accordingly, and start the analysis from the written first reactions where they exist.
Verify: each theme names the groups and individuals it came from and whether it appeared before or after someone influential spoke, and no finding is stated as a share of the room.
Prevent: write the one or two things the decision needs before choosing the method, keep people who defer to each other in separate groups, bring in quiet participants by name and meet quick agreement with "who saw it differently?", and debrief observers straight after each session so that their impression of the most quotable participant is checked rather than repeated as the finding.
Prevention: a screener, a guide built on past behaviour and a synthesis table before any report
Every cause above is a missing artefact rather than a bad session, and three artefacts, produced in order, prevent most repeat rounds. The first is the screener and the recruitment record that goes with it: the brief describes the participants by what they did and when, names the frame and who it misses and sets a quota for each group the decision compares, and the screener confirms fit without revealing which answers qualify. The invitation says who is asking, what taking part involves, whether it is recorded, what the incentive is and that participation is voluntary, because the professional codes the site's research guides describe from the Market Research Society and ESOMAR rest on voluntary and informed participation. An invitation that promotes something recruits the people who came for the offer, which is the loyal-fan problem by another route. The recruitment log stays apart from the findings, with participants as codes.
The second is a discussion guide built on past behaviour and piloted before the first real session. The readiness checklist is the gate: the sections reconstruct a specific episode in order, the hypothetical questions sit at the end if anywhere, the pilot has been run, and the notes, the observers and the summary template are ready before anyone arrives. Where the episode unfolds over days or weeks, such as the first weeks with a product, the tutorial on running a diary study has people record each moment as it happens, with an exit interview for the reasons behind what they recorded; interviews about a long episode will otherwise hear the tidied version memory keeps.
The third is the synthesis table, built before any report is drafted: a codeframe defined by a person from a sample, every passage coded with its excerpt and participant code, and an evidence table that carries each theme, its definition, its participants, the excerpts for and against it and the implication, with findings written as reasons and their breadth in words. The same coded transcripts feed the tutorial on turning customer language into page copy, which mines customers' phrases from the coded passages, so the round pays twice. If the themes are sound and reported with their limits and the decision still does not move, the fault lies past the research, and the diagnosis for research findings that are not acted on starts where this page stops.
Tie the three to a short routine. Before fieldwork, the readiness checklist is ticked, with the screener and the quotas checked by someone other than the person who wrote them. After the last session, the synthesis table is built and a second person reads a sample of the coded passages blind before the findings are written. At the readout, the team returns to the line written before fieldwork about what finding would change its mind and says whether it did. The marketing research troubleshooting hub routes the neighbouring symptoms to their own diagnoses.
A tutoring agency replaces compliments with a switch story (illustrative)
Jubbersley Tutoring
Jubbersley Tutoring, an invented online tutoring agency, interviews 12 parents to learn why families choose it before it rewrites its homepage. The readout is warm and useless: 11 of the 12 parents praise the tutors, 3 mention the booking system, and nobody can say what the homepage should say differently.
The marketing lead reads 2 transcripts and marks the questions. Of 23 questions in the first, 17 asked what the parent thought of the service or would want from it and 2 asked about a specific past event; the second transcript splits 19 to 3. The recruitment record shows all 12 parents replied to a newsletter invitation, 11 have been with the agency for more than a year, and the incentive was a free session. There is no coded table, only a slide of 9 quotes chosen the day after the last session.
The second round changes the three artefacts. A screener asks which of 7 kinds of help families used in the past year, when they last looked for a tutor and how they use tutoring now, with a quota of 5 lapsed families, 5 who enquired but never booked and 5 current families, and the incentive becomes a shopping voucher. The guide is rebuilt around the last time the family looked for a tutor: what was happening at school that week, what they tried first, which 2 or 3 options they compared, what nearly stopped them and what tipped the choice. Each transcript is coded against 14 codes with excerpts and participant codes, and the evidence table shows a trigger the first round never surfaced: in 9 of the 15 families, the search began with a parents' evening comment, and the families who did not book had compared the agency against a school-recommended tutor whose availability they could see on a page. The report presents the trigger and the comparison as reasons, with excerpts for and against, gives no percentages, and recommends a short survey of 400 enquirers to measure how common the parents' evening trigger is before the homepage is rewritten around it.
Common mistakes
- Re-presenting the same transcripts with better slides. The readout did not fail at the readout; read the questions and the recruitment record first.
- Interviewing whoever replied to the newsletter. The engaged customers who answer first are the people the question is least likely to be about.
- Asking what people think and what they would do. Opinions and predictions produce politeness and optimism; the last time they did it produces evidence.
- Letting an account owner choose the participants. The study hears from the champion at the friendliest accounts and never from the approver who explained the outcome.
- Skipping synthesis because the stories were vivid. The most quotable participant becomes the finding unless a coded table says otherwise.
- Reporting interviews as shares. A tally of a dozen conversations is breadth among the people spoken to, not prevalence among customers.
- Running a group to hear a private decision. People summarise in front of strangers and reconstruct one to one.
- Treating the fix as finished when the next round is booked. Without a screener check, a pilot and a synthesis table on the plan, the next readout repeats this one.
