Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Conversion Rate Optimisation › Conversion Research

Running a Usability Test on a Conversion Journey

How to run a usability test on a checkout, form or sign-up journey: tasks written as goals, recruiting, moderating without helping, notes, severity and fixes.
Tutorialintermediate10 min readUpdated

What a usability test finds that analytics cannot

A funnel report (Funnel) says how many people leave a step. Recordings show what they did before leaving. Neither shows a person's reasoning at the moment they stalled, and neither can test a journey that has no visitors yet. The conversion research methods guide describes usability testing as the one method that reliably finds problems nobody knew to ask about, and as the backbone of research for low-traffic sites and new flows. The Nielsen Norman Group, listed in this page's sources, maintains a large body of published usability research that is a sound reference for the practice; this page draws its method from the site's own guides rather than from any single study.

The method has one bias that matters more than the rest. People who know they are being watched try harder, read more carefully and complain less, so, as the methods guide puts it, a problem that appears in a usability session is almost certainly worse in the wild. The reverse also holds: a journey that every participant completes is not proven to be fine, because real visitors are distracted, hurried and on worse connections.

Usability testing is qualitative research, and the qualitative versus quantitative comparison explains what that means: depth on why and how, no claim about how common anything is. It also differs from two neighbouring methods. Customer interviews are conversations about past decisions; a usability session is an observed attempt at a task. And walking the customer journey yourself has the team go through the journey as a customer, which finds practical failures quickly but cannot show how a person who does not know the business reads a page. An expert review has the same limit: reviewers who know the guides find problems that break a rule, and the sessions show which of them real people meet.

ChoiceModerated sessionsUnmoderated sessions
Who is presentA facilitator, live or remote, and a note-takerNobody; the participant follows a written script and records themselves
Best forComplex flows, new journeys, prototypes, follow-up questionsSimpler tasks and checking fixes, where the script can stand on its own
Cost and reachMore time per session; fewer peopleCheaper per session; scales to more people
What you loseSome natural behaviour, because a person is watchingThe chance to ask why a participant hesitated or to rescue a confusing task

Before you start

  • The methods guide, read. The conversion research methods guide is the prerequisite; this tutorial runs its usability section.
  • A journey and evidence about it. Ideally a funnel analysis or research sprint that located the step; for a new journey, the prototype and the team's list of worries.
  • A facilitator and a note-taker. Two people, so the facilitator can concentrate on the participant.
  • Devices that match visitors. The phone types and browsers the analytics show real visitors using, not the team's own laptops.
  • A consent form and a place to keep recordings. The checklist in the customer interviews tutorial is the model, agreed with whoever handles data protection.
  • Test data. Test accounts, test card details and addresses that the payment and delivery systems accept without placing a real order.

The steps

  1. Choose the journey and what you need to learn

    Name the journey from its first page to the conversion, such as finding a product and ordering a sample, or starting a trial from a landing page. Then write the few things the round must answer: "Can people find out that samples are free?", "Where do they stall in the address form?" A round with no learning goals produces a list of everything participants said. If a funnel analysis located a step, make sure the tasks pass through it.

  2. Write tasks as goals with realistic details

    Write each task as a goal a real customer would have, not as instructions: "You are retiling a bathroom and want to see two of these tiles in your own light before you buy. Get them sent to you." rather than "Click Order sample and fill in the form." Give the realistic details a person would know, such as the room, the budget or the deadline, and leave out the words on the buttons, so the test shows whether the page's own labels make sense. Order the tasks the way the journey runs and keep the set short enough to finish without hurrying. The academy exercise on planning a conversion research sprint for a picture framer practises writing a usability task as a goal inside a wider research sprint.

  3. Decide moderated or unmoderated

    Choose by the journey and the learning goals, using the table above. A complex checkout, a prototype or a goal that needs follow-up questions calls for moderated sessions. A simple task, or a check that a fix works, can run unmoderated with a written script that says what to do and asks the participant to think aloud. Either way, run sessions on the device the funnel pointed to; a phone problem does not appear on a desktop.

  4. Recruit participants who match real visitors

    Recruit people who resemble the visitors the journey serves: the kind of customer, how familiar they are with the category, whether they are new to the business, and the device they would use. Staff, friends and people who know the product are not participants. Use the recruiting steps in the customer interviews tutorial: a source that reaches the right people, a short screener that does not reveal which answers qualify, and an incentive that does not bias the result, such as a voucher rather than a discount on your own product. The tutorial on recruiting research participants and writing a screener covers the exclusions and the reserves that keep a round from being filled with whoever is free. Plan a small round, not a large one; the next round will follow the fixes.

  5. Prepare consent, test data and devices

    Before any session, tell participants what will be recorded, why, who will see it, how long it will be kept and how to withdraw, and get their agreement before recording starts, as the customer interviews tutorial's consent checklist sets out. Data protection law, including the GDPR listed in this page's sources, treats recordings and transcripts as personal data, so collect the minimum, restrict access and delete on a stated schedule; the personal data guide explains how to keep names and contact details out of analytics if sessions run on the live site. As general practice, give participants test accounts and test payment details and never ask them to type their own card or address. This page is educational, not legal advice.

  6. Run a pilot session

    Run the whole session once with a colleague who does not work on the journey. The pilot finds tasks that give the answer away, tasks that cannot be completed with the test data, a prototype that breaks partway through, a recording set-up that misses the phone screen and a script that runs long. Fix each, then run the pilot again if the changes were large. A broken session with a real participant costs a slot in a small round. The research session readiness checklist puts the pilot beside the recruits, consent, test data and notes as one gate to tick before the first participant arrives.

  7. Moderate without helping

    Start by saying that the journey is being tested, not the participant, that there are no wrong answers and that thinking aloud helps. Then read the task and stay quiet. When a participant asks "Should I click this?", reply with a question: "What would you do if I were not here?" When they fall silent, prompt with "What are you looking for now?" rather than a hint. Never rescue a participant from a problem until it is clear they would have left; the moment they stall is the finding. Keep follow-up questions for the end of each task.

  8. Note what people did and said

    The note-taker records, for each task, what the participant did, what they said, where they hesitated, what they misread and whether they completed it, with a timestamp so the moment can be found in the recording. Keep behaviour and statements separate from interpretation, as the customer interviews tutorial asks: "tapped Continue repeatedly" is an observation, "thought the button was broken" is an inference. Straight after each session, list the problems seen and the participant's own words for them.

  9. Rate each problem by severity

    After the round, merge the problems across sessions and rate each on a scale the team agrees before rating. No published scale is used here; a short scale like this one is enough for most journeys:

    SeverityMeaningUsual decision
    BlocksThe participant could not complete the task without helpFix now; retest in the next round
    DelaysThe participant completed it, but only after a detour, error or long hesitationFix or write a hypothesis, by cost
    IrritatesThe participant noticed and commented, but it did not change the outcomeRecord; fix when the page is next changed

    For each problem record where it happened, its severity, how many participants met it and their words for it. Remember that the count describes the round, not all visitors; a blocking problem met by one participant is still a blocking problem.

  10. Fix, retest and pass the rest to the backlog

    Fix the blocking problems that are bugs, such as a field that rejects valid input or a button hidden under a phone keyboard, without a test, and check the fixes in a second small round alongside any new wording. The forms guide and the checkout optimisation guide catalogue the usual fixes. Sort the remaining problems with the impact-effort matrix and add them to the research backlog in the conversion research hub, with the funnel's reach for the step where analytics can supply it. Changes whose effect is uncertain become a hypothesis for the prioritisation guide to weigh against the traffic, with the step's conversion rate (CVR) as the metric.

A tile shop tests its sample ordering (illustrative)

Illustrative example — not a real result

Mortarbloom Tiles tests free sample ordering on phones

Mortarbloom Tiles is an invented online tile retailer whose customers usually order a sample before buying. Its funnel shows that phone visitors who open a tile page rarely reach the sample confirmation, although sample orders lead to most of its tile sales. The marketing lead writes two learning goals, whether people can find out that samples are free and where they stall in the sample form, and one task: "You are retiling a small bathroom and want to see two of these tiles in your own light. Get them sent to you." She runs a small round of moderated sessions on phones with recent browsers of the site recruited through a short screener, with a voucher for each participant, test addresses and recorded consent. A pilot with a colleague from accounts shows that the task's original wording said "free sample" and gave the answer away, so it is rewritten. In the sessions, several participants scroll past the "Order a sample" link because it sits beside the price and reads as a charge, and some stall at the postcode field, which rejects postcodes typed with a space. The postcode problem is rated "blocks" and fixed the same week without a test. The sample wording is rated "delays"; the team writes "Free sample, delivered in two days" beside the button and checks it in a second small round, where participants find the sample without prompting. The remaining problems, including a delivery note that is hard to read on small screens, go to the backlog, and the wording change becomes a hypothesis for a test on sample orders from phones.

Common mistakes

  • Tasks written as instructions. "Click Order sample" tests whether people can follow orders, not the page.
  • Testing with colleagues. People who know the product cannot see it as a visitor does.
  • The wrong device. A journey most visitors use on a phone tested on a laptop misses its problems.
  • Helping. A rescued participant is a hidden finding.
  • No pilot. The first real session finds the broken task instead of the broken journey.
  • Real card details. Test data protects participants and keeps the session about the journey.
  • Counts read as shares. How many participants met a problem describes the round, not all visitors.
  • One big round. A small round, fixes and another round finds more than many sessions at once.
  • Testing bugs. A field that rejects valid input is fixed and retested, not A/B tested.

Frequently asked questions

How many participants does a usability test need?

This page gives no number, because no source the site cites supports one and the answer depends on the journey. The practical rule in the conversion research methods guide is to run a small round of sessions, fix what they found, and run another small round, rather than running many sessions at once. Plan enough participants to cover the kinds of visitor who behave differently on the journey, such as new and returning customers or phone and desktop users, and stop a round when sessions keep showing the same problems.

Should a usability test be moderated or unmoderated?

Moderated sessions, with a facilitator present, allow follow-up questions and suit complex flows such as a checkout with delivery choices or a multi-step application. Unmoderated sessions, where the participant records themselves following a written script, are cheaper and scale to more people, but nobody can ask why a participant hesitated and a confusing task cannot be rescued mid-session. Many teams use moderated sessions to find problems in a new or complicated journey and unmoderated sessions to check simpler fixes.

Can a usability test replace an A/B test?

No; they answer different questions. A usability test shows whether people can complete the journey and where they struggle, with a handful of people under observation. It cannot say how much a change would move the conversion rate across all visitors. Problems that stop people, such as a field rejecting valid input, are fixed without a test and checked in the next round. Larger changes whose effect is uncertain become hypotheses and, where the traffic allows, A/B tests.

Terms used on this page

Frameworks

Where this page is used

Part of these learning paths

Also related

Sources

  1. Nielsen Norman Group articles — Nielsen Norman Group
  2. Regulation (EU) 2016/679 (GDPR) — EUR-Lex

Related