Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Conversion Rate Optimisation › Beginner

Writing Test Hypotheses from Research Notes

A practice exercise in moving from a pile of research notes to hypotheses a test could confirm or reject. You group the findings into problems, separate the bugs that should simply be fixed, and write each remaining problem up with the hypothesis template, including the metric that will judge it.
ExerciseBeginner levelModule 3: Research and first hypotheses5 minUpdated

Why this matters

Research produces notes; tests need hypotheses. Programmes go wrong in between, by testing the first idea someone had or by writing a preference with a metric attached. A good hypothesis states evidence, problem, change and reason, so that even a losing test teaches something.

This exercise applies the tutorial on writing a test hypothesis and the fields of the experiment hypothesis template. The notes come from the methods in the guide to conversion research methods, and part of the task is judging how far each can be trusted alone.

The research notes

Pinfold Guesthouse is a fictional bed and breakfast. Its booking page has a calendar, a room list, an extras step and payment. The owner gathered a month of notes.

Illustrative example — not a real result

Pinfold Guesthouse: a month of research notes

SourceNote
AnalyticsBooking page visits: 1,300. Bookings: 26, a booking rate of 2.0%. On phones: 860 visits, 9 bookings (1.0%). On desktop: 440 visits, 17 bookings (3.9%)
AnalyticsOf 310 visitors who reached the extras step, 140 left there. The extras step is where the total including the cleaning fee first appears
Recordings30 phone sessions on the calendar reviewed. In 17, the visitor tapped a date, the calendar jumped back to the current month, and they tapped again or left
RecordingsIn 8 of the 30, the visitor opened the room list, scrolled, then left the page for the "Contact" page
Survey (after booking, 22 replies)9 said they nearly booked elsewhere because they "couldn't see the full price until the end"
Survey6 asked whether dogs are allowed; the website does not say
Inbox11 emails in the month asking about dogs; 5 asking about the cancellation policy, which sits in the terms page footer
Inbox3 emails from guests who said the "Pay now" button did nothing on an older tablet; the owner reproduced it on one browser

Traffic note: at around 1,300 visits a month, only large effects could be detected by a test in reasonable time.

From finding to hypothesis

Write each note as an observation with its source, then group observations describing the same visitor difficulty into one problem, stated without a solution. Anything simply broken is a defect to fix, not a hypothesis. For the rest, fill the template's core fields: evidence, problem, change, the hypothesis sentence ("If we…, then… will… for…, because…"), primary metric and guardrail. A problem seen in one source is a lead, not yet evidence. And be honest about traffic: an A/B Test needs enough visitors to detect the predicted effect, so some changes are better shipped and watched.

Task

  1. Group the notes into problems, each stated without a solution.
  2. Identify any bug and say what should happen to it.
  3. For each remaining problem, write the evidence, change, hypothesis sentence, primary metric and a guardrail.
  4. Say which hypotheses rest on thin evidence and what research would strengthen them.
  5. Decide which to test first and whether the traffic allows a test at all.

Worked answer

Problems. The notes group into four.

  • Phone visitors cannot pick dates reliably: the phone booking gap plus the calendar recordings.
  • The total price appears late: the extras-step drop plus survey answers about the full price.
  • Visitors cannot find the dog or cancellation policy: survey, inbox and recordings of exits to Contact.
  • The "Pay now" button fails on one browser: inbox reports the owner reproduced.

The bugs. The failing button is broken: fix it, check it on that browser and log it. The calendar jump is probably a bug too; if a developer can reproduce it, fix it rather than test it.

Illustrative example — not a real result

Pinfold Guesthouse: hypotheses written up

FieldHypothesis A: full price earlyHypothesis B: dogs and cancellation answered
Evidence140 of 310 leave at the extras step, where the fee first appears; 9 of 22 survey replies mention the late price6 survey replies and 11 emails ask about dogs; 5 emails ask about cancellation; 8 of 30 recordings leave for Contact
ProblemVisitors choosing a room cannot see what the stay will cost in total, so some leave when the fee appearsVisitors cannot find two facts that decide whether they can book, so they leave to ask or go elsewhere
ChangeShow the total for the chosen dates, fee included, on each room in the room listAdd a short "good to know" panel beside the room list: the dog policy and the cancellation terms in one line each
HypothesisIf we show the full total on the room list, then the share of booking page visitors who book will rise for all visitors, because the fee will no longer surprise them at the extras stepIf we state the dog and cancellation policies beside the rooms, then bookings per booking page visit will rise and enquiry emails will fall, because visitors get the answer without leaving
Primary metricBookings ÷ booking page visitors, per visitBookings ÷ booking page visitors, per visit
GuardrailAverage booking value must not fallCancellation rate must not rise

Gaps. Hypothesis A's survey reached only people who booked; a poll on the extras step would hear from those who leave. Hypothesis B has three sources but nothing from non-bookers; a booking page poll would add it.

What to test first. Fix the bugs first: the phone gap is the largest loss and may close once dates work. This traffic can only detect a large change in CVR. Hypothesis A addresses the biggest measured drop and could produce one, so it is the first test, run for whole weeks to its planned sample. Hypothesis B is cheap and low risk; ship it as a considered change and watch bookings and emails.

Common mistakes

  • Writing the solution as the problem. "We need a new booking system" is a fix; the problem is what visitors cannot do.
  • Testing bugs. A button that does nothing needs repair, not an experiment.
  • Bundling changes. Moving the price and adding the policy panel together cannot teach you which mattered.
  • No mechanism. "Because it looks clearer" cannot be wrong, so it cannot teach.
  • Ignoring traffic. A hypothesis predicting a small lift at a small guesthouse may never reach a result.

Bring this into the landing page audit capstone, or review the module on the beginner conversion optimisation level page.

Pages this lesson applies

Terms used on this page

Also related

Related