Why this matters
Research produces notes; tests need hypotheses. Programmes go wrong in between, by testing the first idea someone had or by writing a preference with a metric attached. A good hypothesis states evidence, problem, change and reason, so that even a losing test teaches something.
This exercise applies the tutorial on writing a test hypothesis and the fields of the experiment hypothesis template. The notes come from the methods in the guide to conversion research methods, and part of the task is judging how far each can be trusted alone.
The research notes
Pinfold Guesthouse is a fictional bed and breakfast. Its booking page has a calendar, a room list, an extras step and payment. The owner gathered a month of notes.
Pinfold Guesthouse: a month of research notes
| Source | Note |
|---|---|
| Analytics | Booking page visits: 1,300. Bookings: 26, a booking rate of 2.0%. On phones: 860 visits, 9 bookings (1.0%). On desktop: 440 visits, 17 bookings (3.9%) |
| Analytics | Of 310 visitors who reached the extras step, 140 left there. The extras step is where the total including the cleaning fee first appears |
| Recordings | 30 phone sessions on the calendar reviewed. In 17, the visitor tapped a date, the calendar jumped back to the current month, and they tapped again or left |
| Recordings | In 8 of the 30, the visitor opened the room list, scrolled, then left the page for the "Contact" page |
| Survey (after booking, 22 replies) | 9 said they nearly booked elsewhere because they "couldn't see the full price until the end" |
| Survey | 6 asked whether dogs are allowed; the website does not say |
| Inbox | 11 emails in the month asking about dogs; 5 asking about the cancellation policy, which sits in the terms page footer |
| Inbox | 3 emails from guests who said the "Pay now" button did nothing on an older tablet; the owner reproduced it on one browser |
Traffic note: at around 1,300 visits a month, only large effects could be detected by a test in reasonable time.
From finding to hypothesis
Write each note as an observation with its source, then group observations describing the same visitor difficulty into one problem, stated without a solution. Anything simply broken is a defect to fix, not a hypothesis. For the rest, fill the template's core fields: evidence, problem, change, the hypothesis sentence ("If we…, then… will… for…, because…"), primary metric and guardrail. A problem seen in one source is a lead, not yet evidence. And be honest about traffic: an A/B TestControlled comparison of two variants. needs enough visitors to detect the predicted effect, so some changes are better shipped and watched.
Task
- Group the notes into problems, each stated without a solution.
- Identify any bug and say what should happen to it.
- For each remaining problem, write the evidence, change, hypothesis sentence, primary metric and a guardrail.
- Say which hypotheses rest on thin evidence and what research would strengthen them.
- Decide which to test first and whether the traffic allows a test at all.
Worked answer
Problems. The notes group into four.
- Phone visitors cannot pick dates reliably: the phone booking gap plus the calendar recordings.
- The total price appears late: the extras-step drop plus survey answers about the full price.
- Visitors cannot find the dog or cancellation policy: survey, inbox and recordings of exits to Contact.
- The "Pay now" button fails on one browser: inbox reports the owner reproduced.
The bugs. The failing button is broken: fix it, check it on that browser and log it. The calendar jump is probably a bug too; if a developer can reproduce it, fix it rather than test it.
Pinfold Guesthouse: hypotheses written up
| Field | Hypothesis A: full price early | Hypothesis B: dogs and cancellation answered |
|---|---|---|
| Evidence | 140 of 310 leave at the extras step, where the fee first appears; 9 of 22 survey replies mention the late price | 6 survey replies and 11 emails ask about dogs; 5 emails ask about cancellation; 8 of 30 recordings leave for Contact |
| Problem | Visitors choosing a room cannot see what the stay will cost in total, so some leave when the fee appears | Visitors cannot find two facts that decide whether they can book, so they leave to ask or go elsewhere |
| Change | Show the total for the chosen dates, fee included, on each room in the room list | Add a short "good to know" panel beside the room list: the dog policy and the cancellation terms in one line each |
| Hypothesis | If we show the full total on the room list, then the share of booking page visitors who book will rise for all visitors, because the fee will no longer surprise them at the extras step | If we state the dog and cancellation policies beside the rooms, then bookings per booking page visit will rise and enquiry emails will fall, because visitors get the answer without leaving |
| Primary metric | Bookings ÷ booking page visitors, per visit | Bookings ÷ booking page visitors, per visit |
| Guardrail | Average booking value must not fall | Cancellation rate must not rise |
Gaps. Hypothesis A's survey reached only people who booked; a poll on the extras step would hear from those who leave. Hypothesis B has three sources but nothing from non-bookers; a booking page poll would add it.
What to test first. Fix the bugs first: the phone gap is the largest loss and may close once dates work. This traffic can only detect a large change in CVRShare of visits or clicks that convert.. Hypothesis A addresses the biggest measured drop and could produce one, so it is the first test, run for whole weeks to its planned sample. Hypothesis B is cheap and low risk; ship it as a considered change and watch bookings and emails.
Common mistakes
- Writing the solution as the problem. "We need a new booking system" is a fix; the problem is what visitors cannot do.
- Testing bugs. A button that does nothing needs repair, not an experiment.
- Bundling changes. Moving the price and adding the policy panel together cannot teach you which mattered.
- No mechanism. "Because it looks clearer" cannot be wrong, so it cannot teach.
- Ignoring traffic. A hypothesis predicting a small lift at a small guesthouse may never reach a result.
Bring this into the landing page audit capstone, or review the module on the beginner conversion optimisation level page.
