What a usability test finds that analytics cannot
A funnel report (FunnelModel of the customer journey stages.) says how many people leave a step. Recordings show what they did before leaving. Neither shows a person's reasoning at the moment they stalled, and neither can test a journey that has no visitors yet. The conversion research methods guide describes usability testing as the one method that reliably finds problems nobody knew to ask about, and as the backbone of research for low-traffic sites and new flows. The Nielsen Norman Group, listed in this page's sources, maintains a large body of published usability research that is a sound reference for the practice; this page draws its method from the site's own guides rather than from any single study.
The method has one bias that matters more than the rest. People who know they are being watched try harder, read more carefully and complain less, so, as the methods guide puts it, a problem that appears in a usability session is almost certainly worse in the wild. The reverse also holds: a journey that every participant completes is not proven to be fine, because real visitors are distracted, hurried and on worse connections.
Usability testing is qualitative research, and the qualitative versus quantitative comparison explains what that means: depth on why and how, no claim about how common anything is. It also differs from two neighbouring methods. Customer interviews are conversations about past decisions; a usability session is an observed attempt at a task. And walking the customer journey yourself has the team go through the journey as a customer, which finds practical failures quickly but cannot show how a person who does not know the business reads a page. An expert review has the same limit: reviewers who know the guides find problems that break a rule, and the sessions show which of them real people meet.
| Choice | Moderated sessions | Unmoderated sessions |
|---|---|---|
| Who is present | A facilitator, live or remote, and a note-taker | Nobody; the participant follows a written script and records themselves |
| Best for | Complex flows, new journeys, prototypes, follow-up questions | Simpler tasks and checking fixes, where the script can stand on its own |
| Cost and reach | More time per session; fewer people | Cheaper per session; scales to more people |
| What you lose | Some natural behaviour, because a person is watching | The chance to ask why a participant hesitated or to rescue a confusing task |
Before you start
- The methods guide, read. The conversion research methods guide is the prerequisite; this tutorial runs its usability section.
- A journey and evidence about it. Ideally a funnel analysis or research sprint that located the step; for a new journey, the prototype and the team's list of worries.
- A facilitator and a note-taker. Two people, so the facilitator can concentrate on the participant.
- Devices that match visitors. The phone types and browsers the analytics show real visitors using, not the team's own laptops.
- A consent form and a place to keep recordings. The checklist in the customer interviews tutorial is the model, agreed with whoever handles data protection.
- Test data. Test accounts, test card details and addresses that the payment and delivery systems accept without placing a real order.
The steps
Choose the journey and what you need to learn
Name the journey from its first page to the conversion, such as finding a product and ordering a sample, or starting a trial from a landing pageThe page a visitor arrives on from a campaign, built to turn that visit into one action.. Then write the few things the round must answer: "Can people find out that samples are free?", "Where do they stall in the address form?" A round with no learning goals produces a list of everything participants said. If a funnel analysis located a step, make sure the tasks pass through it.
Write tasks as goals with realistic details
Write each task as a goal a real customer would have, not as instructions: "You are retiling a bathroom and want to see two of these tiles in your own light before you buy. Get them sent to you." rather than "Click Order sample and fill in the form." Give the realistic details a person would know, such as the room, the budget or the deadline, and leave out the words on the buttons, so the test shows whether the page's own labels make sense. Order the tasks the way the journey runs and keep the set short enough to finish without hurrying. The academy exercise on planning a conversion research sprint for a picture framer practises writing a usability task as a goal inside a wider research sprint.
Decide moderated or unmoderated
Choose by the journey and the learning goals, using the table above. A complex checkout, a prototype or a goal that needs follow-up questions calls for moderated sessions. A simple task, or a check that a fix works, can run unmoderated with a written script that says what to do and asks the participant to think aloud. Either way, run sessions on the device the funnel pointed to; a phone problem does not appear on a desktop.
Recruit participants who match real visitors
Recruit people who resemble the visitors the journey serves: the kind of customer, how familiar they are with the category, whether they are new to the business, and the device they would use. Staff, friends and people who know the product are not participants. Use the recruiting steps in the customer interviews tutorial: a source that reaches the right people, a short screener that does not reveal which answers qualify, and an incentive that does not bias the result, such as a voucher rather than a discount on your own product. The tutorial on recruiting research participants and writing a screener covers the exclusions and the reserves that keep a round from being filled with whoever is free. Plan a small round, not a large one; the next round will follow the fixes.
Prepare consent, test data and devices
Before any session, tell participants what will be recorded, why, who will see it, how long it will be kept and how to withdraw, and get their agreement before recording starts, as the customer interviews tutorial's consent checklist sets out. Data protection law, including the GDPR listed in this page's sources, treats recordings and transcripts as personal data, so collect the minimum, restrict access and delete on a stated schedule; the personal data guide explains how to keep names and contact details out of analytics if sessions run on the live site. As general practice, give participants test accounts and test payment details and never ask them to type their own card or address. This page is educational, not legal advice.
Run a pilot session
Run the whole session once with a colleague who does not work on the journey. The pilot finds tasks that give the answer away, tasks that cannot be completed with the test data, a prototype that breaks partway through, a recording set-up that misses the phone screen and a script that runs long. Fix each, then run the pilot again if the changes were large. A broken session with a real participant costs a slot in a small round. The research session readiness checklist puts the pilot beside the recruits, consent, test data and notes as one gate to tick before the first participant arrives.
Moderate without helping
Start by saying that the journey is being tested, not the participant, that there are no wrong answers and that thinking aloud helps. Then read the task and stay quiet. When a participant asks "Should I click this?", reply with a question: "What would you do if I were not here?" When they fall silent, prompt with "What are you looking for now?" rather than a hint. Never rescue a participant from a problem until it is clear they would have left; the moment they stall is the finding. Keep follow-up questions for the end of each task.
Note what people did and said
The note-taker records, for each task, what the participant did, what they said, where they hesitated, what they misread and whether they completed it, with a timestamp so the moment can be found in the recording. Keep behaviour and statements separate from interpretation, as the customer interviews tutorial asks: "tapped Continue repeatedly" is an observation, "thought the button was broken" is an inference. Straight after each session, list the problems seen and the participant's own words for them.
Rate each problem by severity
After the round, merge the problems across sessions and rate each on a scale the team agrees before rating. No published scale is used here; a short scale like this one is enough for most journeys:
Severity Meaning Usual decision Blocks The participant could not complete the task without help Fix now; retest in the next round Delays The participant completed it, but only after a detour, error or long hesitation Fix or write a hypothesis, by cost Irritates The participant noticed and commented, but it did not change the outcome Record; fix when the page is next changed For each problem record where it happened, its severity, how many participants met it and their words for it. Remember that the count describes the round, not all visitors; a blocking problem met by one participant is still a blocking problem.
Fix, retest and pass the rest to the backlog
Fix the blocking problems that are bugs, such as a field that rejects valid input or a button hidden under a phone keyboard, without a test, and check the fixes in a second small round alongside any new wording. The forms guide and the checkout optimisation guide catalogue the usual fixes. Sort the remaining problems with the impact-effort matrix and add them to the research backlog in the conversion research hub, with the funnel's reach for the step where analytics can supply it. Changes whose effect is uncertain become a hypothesisA testable statement of the change to make, the effect expected on a named metric and the evidence-based reason for expecting it. for the prioritisation guide to weigh against the traffic, with the step's conversion rate (CVRShare of visits or clicks that convert.) as the metric.
A tile shop tests its sample ordering (illustrative)
Mortarbloom Tiles tests free sample ordering on phones
Mortarbloom Tiles is an invented online tile retailer whose customers usually order a sample before buying. Its funnel shows that phone visitors who open a tile page rarely reach the sample confirmation, although sample orders lead to most of its tile sales. The marketing lead writes two learning goals, whether people can find out that samples are free and where they stall in the sample form, and one task: "You are retiling a small bathroom and want to see two of these tiles in your own light. Get them sent to you." She runs a small round of moderated sessions on phones with recent browsers of the site recruited through a short screener, with a voucher for each participant, test addresses and recorded consent. A pilot with a colleague from accounts shows that the task's original wording said "free sample" and gave the answer away, so it is rewritten. In the sessions, several participants scroll past the "Order a sample" link because it sits beside the price and reads as a charge, and some stall at the postcode field, which rejects postcodes typed with a space. The postcode problem is rated "blocks" and fixed the same week without a test. The sample wording is rated "delays"; the team writes "Free sample, delivered in two days" beside the button and checks it in a second small round, where participants find the sample without prompting. The remaining problems, including a delivery note that is hard to read on small screens, go to the backlog, and the wording change becomes a hypothesis for a test on sample orders from phones.
Common mistakes
- Tasks written as instructions. "Click Order sample" tests whether people can follow orders, not the page.
- Testing with colleagues. People who know the product cannot see it as a visitor does.
- The wrong device. A journey most visitors use on a phone tested on a laptop misses its problems.
- Helping. A rescued participant is a hidden finding.
- No pilot. The first real session finds the broken task instead of the broken journey.
- Real card details. Test data protects participants and keeps the session about the journey.
- Counts read as shares. How many participants met a problem describes the round, not all visitors.
- One big round. A small round, fixes and another round finds more than many sessions at once.
- Testing bugs. A field that rejects valid input is fixed and retested, not A/B tested.
