Hypothesis
If visitors who open chat on product pages are split at random between an assistant that answers from approved product content and hands over to staff on request, and staff answering from the start, contribution per chat visitor after the cost of answering will rise, because routine questions are answered at once and staff time goes to the questions that need it, unless wrong or incomplete answers cost orders or raise returns. The decision behind it: let an automated assistant answer product questions first in product-page chat, with a visible route to staff, or keep chat answered by staff only.
The product page chat test decides whether staffed chat belongs on product pages at all, and says that an automated assistant is a different treatment that would need its own test with its own accuracy checks. This is that test. The guide to AI in customer service and chat scopes contacts into automate, assist or keep human and compares automated and human-handled contacts by reason, which is the right way to run an assistant but cannot say what it changed, because the contacts each side handles differ. Randomising visitors before the first answer removes that difference.
Metrics and guardrails
The primary metric is Contribution MarginRevenue left after every cost that varies with each sale. per visitor who opened chat. For each arm, take the contribution margin on every order placed by its chat visitors during the test and its return window, from the business's own order records, after returns. Subtract the cost of answering: staff time spent on the arm's chats, including handovers, and the assistant's fees. Divide by the number of chat visitors in the arm. Every visitor counts, whether the assistant answered them, handed them over or lost them, so the figure is what the arrangement earns from everyone who asked a question.
Pre-purchase questions answered wrongly cost twice: the order that was lost, or the order that came back. The Return RateShare of orders or units sold that customers sent back. sits inside the primary metric for that reason, and the expectation-disconfirmation model explains why an answer that oversells is worse than no answer: the buyer judges what arrives against the expectation the answer set.
Three guardrails, each a Guardrail MetricA metric an experiment must not worsen beyond an agreed bound, watched beside the primary metric that decides the test. with a limit written before launch:
- Share of sampled assistant answers that match current approved content. A person reads a random sample of the assistant's answers each week, as part of the Human ReviewThe step where a named person checks AI-assisted work for accuracy, sources, originality, privacy and disclosure before it is published or sent. the AI customer service guide sets out, and checks each against the content it cited. Sample across every question type, not only the easy ones.
- Time from asking for a person to reaching one. The handover is where an assistant is judged. Measure from the visitor's request, or the assistant's offer being accepted, to a member of staff joining.
- Repeat contacts on the same subject within a set window per chat visitor. A Repeat ContactA customer contacting the business again about the same problem, the clearest sign that the first answer did not end the customer's work. on the same question shows it was not resolved, whichever route the visitor took next.
Report the share of chats closed without staff beside the result, never as the result. The Goodhart's law framework describes what happens when a measure such as deflectionThe share of customer contacts handled without ever reaching a person, usually by an assistant or help content; read beside resolution, never as the goal. becomes the target: an assistant can raise it by making people give up.
Design
This is an A/B test at visitor level, with the unit fixed at the moment of the first chat. Assign each visitor at random when they first open chat from a product page, and hold them in that arm across later chats and sessions using a first-party identifier, because buyers of considered products often return before ordering.
| Design element | Choice | Reason |
|---|---|---|
| Unit of randomisation | Visitor, assigned at the first product-page chat, held across later chats | A visitor who meets the assistant once and staff the next time blurs both arms |
| Arms | Assistant answers first, with a visible route to staff; staff answer from the start | The decision is who answers first, not whether chat exists |
| Held constant | Chat button and its hours, staffing, phone, email, help centre, prices and offers | Only the first responder differs, so the result belongs to it |
| Assistant content and scope | Approved product content and handover rules, frozen for the test | Changing either mid-test changes the treatment |
| Kept human in both arms | Refunds, complaints and exceptions | The guide keeps these with people; the test does not change that |
| Primary metric | Contribution margin per chat visitor, after returns and the cost of answering | Counts every chat visitor and both costs of answering |
Before the test starts, the assistant passes the AI customer service launch checklist: scope, approved content, handover, disclosure, data and a test set. The test set is built and scored as the guide to testing prompts and models describes, from real past product questions, so the assistant reaches customers having already shown it can answer them. The assistant answers only from approved content, which is what GroundingMaking a language model answer from specified material, such as documents retrieved for the question, rather than from what it absorbed in training. means here, cites what it used, and hands over when the content does not cover the question. The academy exercise on planning a coffee machine retailer's assistant-first chat test practises writing this plan from entry condition to decision rule, with every chat visitor counted, a limit set for each guardrail and the reading held until orders have passed the returns window.
Keep the chat's hours the same in both arms. An assistant answering overnight while the staff arm shows a closed window measures opening hours, not who answers first; that is a separate decision. The support channels and self-service guide sets the handover rule both arms follow: a person reachable at any point without defeating the bot, receiving the whole conversation.
The assistant is part of the treatment, and so is what it says. The AI customer service guide, drawing on the European Commission's overview of its AI regulatory framework and the FTC's business guidance, says customers should be told when they are talking to an automated assistant, and that an assistant presented as a person is treated like any other deceptive practice; the guide to answering customer questions about AI sets the same rule for the moment a visitor asks. The guide to answering pre-purchase questions records that the Advertising Standards Authority's codes are the reference for marketing claims, and that an answer which promises more than the product page and the terms carries the same risk as the claim on the page. Both apply to the assistant in full. This is general direction, not legal advice.
Tag every chat in both arms with a contact reason at close, using the list built as the tutorial on tagging support contacts by reason describes, and name before launch the question types the result will be read by.
Sample size reasoning
Only visitors who open chat enter the test, so the sample is far smaller than product-page traffic, and margin per visitor is mostly zeros with a few orders, which makes it noisy. The inputs are chats per week, the share of chat visitors who order, how much margin per visitor varies and the smallest difference worth changing who answers first. Because the question is whether the assistant is no worse rather than better, the margin the team will accept is part of the plan. The sample size and significance guide shows how those combine and why intervals matter more than a single figure, and the conversion rate calculator turns each arm's orders and chat visitors into the rates it works with.
If chat volume is low, read orders per chat visitor as the primary metric instead, and treat the saving in staff time as known from the rota rather than estimated from the test.
Duration
Run whole weekly cycles that cover every pattern of staffed hours the chat runs, weekday and weekend, busy and quiet, with the assistant's content, scope and handover rules frozen. Avoid launches and sales, when the questions change. Then keep the reading open until orders placed during the test have passed the return period, because the returns are where an overconfident answer shows.
Reading the results and the decision rule
Read once, after the return window. Compare contribution per chat visitor between arms with the interval around the difference, against the margin set before launch. Then read the guardrails, overall and by the question types named in advance. Read the transcripts last, for what went wrong. Write the rule before launch:
| Result | Decision |
|---|---|
| Contribution per chat visitor is reliably no lower, within the margin set before launch, and accuracy, handover and repeat-contact guardrails stay within their limits | Let the assistant answer first, at the scope tested |
| Margin falls reliably, or any guardrail breaks across question types | Keep staff-only chat |
| Results differ by question types named before launch, with some passing every guardrail | Narrow the assistant's scope to the question types that passed |
The test measures the assistant at the content, scope and handover tested. A new model, a rewritten prompt or a restructured help centre is a different treatment: re-run the test set first, and a new test if the change is large.
Pitfalls
Deflection read as success is the first trap. A rising share of chats closed without staff looks like a saving and may be visitors giving up. The primary metric counts them; the deflection figure does not.
Content changed mid-test is the second. A product sheet corrected partway through changes the assistant's answers from that day. Freeze the content, log any correction that cannot wait, and read the weeks either side separately.
A handover that is hard to find is the third. If the route to staff sits behind several replies, the assistant arm tests patience, not the assistant. The handover guardrail catches it only if it is measured from the visitor's first request.
Staff answering assistant chats from the queue is the fourth. When staff can see the assistant's conversations and step in before a handover, the arms converge. Staff join an assistant chat only when it is handed over.
Accuracy checked only on easy questions is the fifth. A sample drawn from conversations that ended well flatters the assistant; sample at random across every question type, and add every handover and every complaint. The guide to common testing pitfalls covers the general traps, from a broken split to reading early. For enquiries where the question is how fast a person responds, the lead response time test is the design.
Example: an invented online telescope shop tests an assistant in its product-page chat (illustrative)
Plovitzer Telescopes tests an assistant that answers first in chat
Plovitzer Telescopes, an invented online telescope shop, runs staffed chat on its product pages and wants to know whether an assistant answering from its compatibility guides can take the first reply. Over eight weeks, 6,400 visitors open chat and are split at random, 3,200 per arm. Before launch the team sets three limits: contribution per chat visitor no more than £1.00 below staff-only chat, at least 95% of sampled answers matching approved content in every question type, and a person reached within three minutes of asking. The reading closes after the 30-day return period.
| Per arm | Staff only | Assistant first |
|---|---|---|
| Chat visitors | 3,200 | 3,200 |
| Orders | 544 (17.0%) | 528 (16.5%) |
| Contribution after returns | £63,360 | £61,760 |
| Staff time | 420 hours, £7,560 | 190 hours, £3,420 |
| Assistant fees | — | £1,600 |
| Contribution per chat visitor after answering costs | £17.44 | £17.73 |
The assistant arm earns £0.29 more per chat visitor, with an interval of roughly −£0.70 to £1.28, inside the margin. Visitors who ask for a person reach one in about one and a half minutes, and repeat contacts are close between arms. Accuracy splits by question type: 117 of 120 sampled compatibility answers and 59 of 60 delivery answers match the approved content, but only 49 of 60 answers about choosing a first telescope do. Plovitzer narrows the assistant to compatibility and delivery questions and keeps first-telescope advice with staff.
Common mistakes
Assistant tests usually fail on whom they count and what they call success. The experiment library lists the page, offer and lead-handling designs that sit beside this one.
- Comparing assistant chats with staff chats. The assistant hands over its hardest questions; compare the arms whole.
- Reporting deflection. A chat nobody answered is not a saving if the visitor left.
- Testing an assistant that has not passed its checklist. The test is not where accuracy is discovered.
- Changing content or prompts mid-test. The treatment is the assistant as frozen at launch.
- Opening chat for more hours in one arm. That tests availability, not who answers first.
- Sampling only easy questions. Accuracy is read across every question type, handovers and complaints included.
