Hypothesis
If visitors to product pages are offered chat with staff who know the products, orders per product-page visitor will rise, because questions that would end the visit are answered at the moment of choice, unless the answers raise returns or the staff time costs more than the extra orders earn. The decision behind it: offer staffed chat on product pages, or keep the existing contact routes only.
The guide to answering pre-purchase questions treats those questions as a sales channel. It sets who answers, the briefing agents need, what they must not promise and how each question is recorded, and it says plainly that no general figure tells a business how much answering them is worth; a controlled test of a new channel, such as chat on product pages, is a separate piece of work designed for the business's own traffic. This is that design. Chat has a cost the page elements in the library do not: someone has to be there to answer, for every hour the window is open.
Metrics and guardrails
The primary metric is orders, or revenue, per product-page visitor, with each visitor held in one arm across sessions. Every visitor assigned to an arm counts in its denominator, whether they opened chat or not, so the metric measures what offering chat does to the whole audience, not what happens to the minority who use it. Orders per visitor is a conversion rate (CVRShare of visits or clicks that convert.); a shop whose order values vary widely can use Revenue per VisitorRevenue divided by the number of visitors in the same period. instead, chosen before launch.
Three guardrails, each with a limit written in advance:
- Return rate on orders from each arm. An answer that promises more than the product delivers wins the order and loses it again. Read the Return RateShare of orders or units sold that customers sent back. of each arm's orders once the return period has passed.
- Staff time per extra order. Divide the staff hours spent on chat by the extra orders the treatment arm produced over the control. It is the cost that decides whether a reliable lift is worth keeping, and the limit should come from the margin on an order.
- Phone and email contacts per visitor. Chat may replace other contacts rather than add sales; a fall here is a service gain, but it is not the sales effect the test is meant to find. A rise suggests chat answers are sending people elsewhere for detail.
Read the Cart Abandonment RateShare of shopping carts created that did not become an order. for the same products alongside, as the pre-purchase guide suggests: chat that removes a last doubt should show up there too.
Design
This is an A/B testControlled comparison of two variants. at visitor level. Assign each visitor at random at their first product-page view and hold them in their arm across sessions, using a first-party identifier, because buyers of considered products often return several times before ordering.
| Design element | Choice | Reason |
|---|---|---|
| Unit of randomisation | Visitor, assigned at the first product-page view, held across sessions | A visitor who sees chat on one visit and not the next blurs both arms |
| Arms | Existing contact routes only; the same routes plus staffed chat on product pages | The decision is whether to add the channel |
| Staffing and hours | Fixed in advance and the same across the run | Changing either mid-test changes the treatment |
| Held constant | Phone, email, help centre, page content, prices, offers and delivery terms | Only chat differs, so the result belongs to it |
| Agent briefing | The pre-purchase guide's briefing and approved wording | Answers must not promise more than the page |
| Primary metric | Orders or revenue per product-page visitor | Counts everyone offered chat, not only those who used it |
Choosing and staffing a channel is its own work. The guide to support channels and self-service describes live chat as fast and able to rescue a sale, but needing real-time staffing; set the hours to those chat would keep after the test, not a showcase rota. Outside the hours, the chat option should be absent in the treatment arm rather than leaving a form nobody answers.
What agents say is part of the treatment. The pre-purchase guide records that in the UK the Advertising Standards Authority's codes are the reference for marketing claims, and that an agent's answer carries the same risk as the claim on the page: if it promises a result the product cannot deliver, the customer has been misled. Its rules apply in full here: approved wording, no promise the page does not make, say when you do not know, and no pressure. This is general direction, not legal advice.
Count chats properly. The guide to measuring calls and chats separates a chat started from a chat that became an enquiry and explains how to tie a conversation to its source; use the same definitions here, and record the arm on each conversation so returns can be traced back.
Sample size reasoning
Most visitors never open chat, so its effect on orders is diluted across everyone in the arm. The inputs are the current conversion rate on product pages, their traffic and the smallest rise worth paying the staffing for. The sample size and significance guide shows how those combine and how dilution raises the number needed: a real effect on the few who chat becomes a small effect on the arm as a whole. The conversion rate calculator turns each arm's orders and visitors into the rates the guide works with. A team that cannot reach the number can test chat on the range that raises the most questions, where the share of visitors who need an answer is highest.
Duration
Run whole weekly cycles covering the opening hours chat would keep after the test, with staffing fixed in advance and the same across the run. Avoid sales and launches, when the questions people ask change. Then keep the reading open for a further window, until orders from both arms have passed the return period, because the return guardrail is the one that catches answers that overpromise.
Reading the results and the decision rule
Read once, after the return window. Compare orders per visitor between arms with the interval around the difference. Then read the guardrails: the return rate of each arm's orders, staff hours divided by the extra orders, and other contacts per visitor. Look at the chat record last, for what people asked. Write the rule before launch:
| Result | Decision |
|---|---|
| Orders per visitor rise reliably and returns and staff time per extra order stay within their limits | Keep chat on product pages, at the hours tested |
| Orders per visitor rise but returns pass their limit | Review the answers given before deciding; keep the current routes meanwhile |
| Orders rise but staff time per extra order passes its limit | Keep chat only on the range that raised most questions, or not at all |
| Orders do not move reliably | Keep the current routes and put the recurring answers on the pages |
Remember what the test cannot say. It measures chat at the hours and staffing tested; a cheaper rota or a different team is a different treatment.
Pitfalls
Reading only visitors who chatted is the first trap. They were closer to buying before they opened the window, and their conversion rate says nothing about what chat added. Compare the arms whole.
Staffing that changes mid-test is the second. A busy week with slow answers and a quiet week with instant ones are different treatments; fix the rota and log any gaps.
Answers that promise more than the product are the third. A delivery date the business cannot meet or a capability the page does not claim wins an order and loses it to a return or a complaint. Review a sample of transcripts against the briefing every week.
Returns read too early are the fourth. The orders chat wins on the strength of an answer are the ones most likely to come back, so wait for the return window to close.
Two simpler page tests sit beside this one and need no staff: the social proof placement test and the exit-intent popup test. If a recurring question can be answered on the page, the product page optimisation guide sets out where.
Example: an invented sewing machine shop tests chat on its product pages (illustrative)
Sauvrekt Sewing Machines tests staffed chat on its product pages
Sauvrekt Sewing Machines, an invented online sewing machine shop, offers chat with two trained staff to half of its product-page visitors from 9am to 6pm, Monday to Saturday, for six weeks. Machines carry a 30-day trial period, so the reading closes 30 days after the last order. Before launch the team sets two limits: the return rate in the chat arm may not exceed the control's, and staff time may not exceed four hours per extra order.
| Arm | Visitors | Orders | Orders per visitor | Returned in trial | Phone and email contacts |
|---|---|---|---|---|---|
| Existing routes | 20,000 | 380 | 1.90% | 23 (6.1%) | 180 |
| With chat | 20,000 | 444 | 2.22% | 24 (5.4%) | 130 |
About 800 visitors in the chat arm open a conversation. Orders per visitor rise by 0.32 points, with an interval of roughly 0.05 to 0.59 points. The 240 staff hours give 64 extra orders, 3.75 hours each, inside the limit, and returns from the chat arm do not rise. Sauvrekt keeps chat at the hours tested and adds the three most-asked answers to its product pages.
Common mistakes
Chat tests usually fail on whom they count. The experiment library lists the other page and offer designs that sit beside this one.
- Comparing chatters with non-chatters. They differ before chat starts; compare the arms whole.
- Judging on chats started. A busy chat window is a cost, not a result.
- Changing hours or staffing mid-test. The treatment is the chat as it will actually run.
- Letting agents improvise claims. Use the approved wording and never promise beyond the page.
- Reading before the return period ends. Overpromised orders come back later.
- Ignoring other contacts. Chat that only replaces phone calls is a service change, not a sales gain.
