Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Conversion Rate Optimisation › Advanced

Checkpoint: Shipping, Redesigns and Accessibility Without a Split Test

Scenario questions set at an invented architectural salvage dealer test whether you can tell a change that should ship from one worth testing, make a test possible on thin traffic, read a before-and-after change honestly, shrink a forced relaunch and treat accessibility barriers as bugs. Each answer links to the guide that teaches it.
CheckpointAdvanced levelModule 2: Conversion work beyond the split test4 minUpdated

What this checkpoint covers

The questions are set at Garnoway Reclamation, an invented dealer in reclaimed doors, fireplaces and flooring whose items are mostly one of a kind, so each listing sees few visitors. They draw on the guide to conversion work without an A/B test, the guide to evaluating a website redesign and the accessibility guide for conversion pages, and follow the cookware relaunch exercise. Sample-size arithmetic is assumed from the intermediate CRO level.

Task

  • Write a one-line answer and an action for each question, marked confident or unsure.
  • Compare with the explanations and note the rule behind every miss.
  • Reread the linked guide for any confident miss before the final module.

Questions

  1. Three items sit in Garnoway's backlog: the enquiry form rejects phone numbers typed with spaces; the owner wants new wording on the "About reclamation" page; and a reservation deposit is proposed for fireplaces, which sell a few a week. Which ships, which is not worth testing, and which the traffic cannot answer?
  2. The owner wants an A/B Test of a new photo layout on one rare fireplace listing. The first example below has the arithmetic. What do you propose instead?
  3. A delivery-cost calculator shipped in March. The owner compares April with February and calls it a win. What should have been written down before it shipped?
  4. The shop platform is closing, so a move is forced, and the designer wants a new brand look launched with it. What do you advise, and what must be recorded before the build?
  5. The developer plans to give phones the new site first and desktops a month later, call it a test, and "decide on rollback if it looks bad". What is wrong?
  6. The collection-slot step has always converted a little worse than the steps around it. A keyboard check finds the slot picker cannot be used without a mouse, though an automated scan passed it. The team proposes testing an accessible picker against the current one. What do you say?
Illustrative example — not a real result

Garnoway Reclamation: the listing and the template

PageVisitors a weekEnquiry rateVisitors per arm the team's calculation needs
One fireplace listing1203.0%9,000
All listings on the item template7,5002.4%11,000

One listing: 9,000 × 2 ÷ 120 = 150 weeks, and the fireplace will have sold long before. The template: 11,000 × 2 ÷ 7,500 ≈ 3 weeks.

Illustrative example — not a real result

Garnoway's before-and-after on the delivery calculator

PeriodVisitsOrdersOrders per visit
February26,0001800.69%
April31,0002480.80%

A trade fair ran in April, renovation projects pick up in spring, and nobody wrote a prediction or chose the comparison period before the calculator went live. Any one of those three could explain the change on its own.

Answers explained

  1. The form ships; the wording is not worth a test; the deposit cannot be tested at this traffic. The no-split-test guide fixes broken things without a test, ships cheap, reversible changes with a prediction and a date to check, and takes untestable ideas to a template, a bolder change, an earlier step or a judgement stated as such.
  2. Test the item template across every listing. The guide's template option buys the combined traffic, and the answer is about the template, which is the decision the business has anyway. One listing could never finish.
  3. The prediction, the reading date, the comparison period and its choice, the minimum count, the reversal rule and everything else changing. The guide's routine aligns weeks and holidays and uses last year's weeks where the season matters. Here the season and the fair are rival explanations, so the honest wording is "consistent with the prediction" at best, with both listed.
  4. Make the one-way door smaller. The redesign guide moves the platform with the old design nearly intact, then changes the design in measured steps, or the reverse, never both together. Before the build, record what converts now, the proof elements on each page and the measurement definitions, and find what quietly persuades, such as a provenance note on each listing, so the brief keeps it.
  5. That is a sequence of before-and-after comparisons, not a test, and the rollback is improvised. A staged rollout compares only when assignment is random, concurrent and sticky, with identical tracking. Rollback rules are written before launch: faults fixed at once, a verdict on the reading date against the agreed range, by a named decision-maker, restoring specific elements rather than the whole site.
  6. Fix it and ship it; do not test it. The accessibility guide treats a barrier as a bug: a test keeps it for half the traffic and could not detect the effect on the few people affected. Barriers look like ordinary drop-off, which is why the keyboard and screen reader checks against the Web Content Accessibility Guidelines find what scans miss. Test only between accessible alternatives, and read the step's CVR afterwards as a check, not a verdict.

Where to go next

Missed the reading questions? Rework the cookware relaunch exercise without the order system. Then start the final module with auditing urgency, defaults and price framing for a campervan hire firm.

Pages this lesson applies

Terms used on this page

Also related

Related