Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Experiment Library

AI Chat Assistant Test

An experiment design for letting an AI assistant answer product questions first in chat, tested against staff-only chat and read on margin per chat visitor.
Experimentintermediate10 min readUpdated

Hypothesis

If visitors who open chat on product pages are split at random between an assistant that answers from approved product content and hands over to staff on request, and staff answering from the start, contribution per chat visitor after the cost of answering will rise, because routine questions are answered at once and staff time goes to the questions that need it, unless wrong or incomplete answers cost orders or raise returns. The decision behind it: let an automated assistant answer product questions first in product-page chat, with a visible route to staff, or keep chat answered by staff only.

The product page chat test decides whether staffed chat belongs on product pages at all, and says that an automated assistant is a different treatment that would need its own test with its own accuracy checks. This is that test. The guide to AI in customer service and chat scopes contacts into automate, assist or keep human and compares automated and human-handled contacts by reason, which is the right way to run an assistant but cannot say what it changed, because the contacts each side handles differ. Randomising visitors before the first answer removes that difference.

Metrics and guardrails

The primary metric is Contribution Margin per visitor who opened chat. For each arm, take the contribution margin on every order placed by its chat visitors during the test and its return window, from the business's own order records, after returns. Subtract the cost of answering: staff time spent on the arm's chats, including handovers, and the assistant's fees. Divide by the number of chat visitors in the arm. Every visitor counts, whether the assistant answered them, handed them over or lost them, so the figure is what the arrangement earns from everyone who asked a question.

Pre-purchase questions answered wrongly cost twice: the order that was lost, or the order that came back. The Return Rate sits inside the primary metric for that reason, and the expectation-disconfirmation model explains why an answer that oversells is worse than no answer: the buyer judges what arrives against the expectation the answer set.

Three guardrails, each a Guardrail Metric with a limit written before launch:

  • Share of sampled assistant answers that match current approved content. A person reads a random sample of the assistant's answers each week, as part of the Human Review the AI customer service guide sets out, and checks each against the content it cited. Sample across every question type, not only the easy ones.
  • Time from asking for a person to reaching one. The handover is where an assistant is judged. Measure from the visitor's request, or the assistant's offer being accepted, to a member of staff joining.
  • Repeat contacts on the same subject within a set window per chat visitor. A Repeat Contact on the same question shows it was not resolved, whichever route the visitor took next.

Report the share of chats closed without staff beside the result, never as the result. The Goodhart's law framework describes what happens when a measure such as deflection becomes the target: an assistant can raise it by making people give up.

Design

This is an A/B test at visitor level, with the unit fixed at the moment of the first chat. Assign each visitor at random when they first open chat from a product page, and hold them in that arm across later chats and sessions using a first-party identifier, because buyers of considered products often return before ordering.

Design elementChoiceReason
Unit of randomisationVisitor, assigned at the first product-page chat, held across later chatsA visitor who meets the assistant once and staff the next time blurs both arms
ArmsAssistant answers first, with a visible route to staff; staff answer from the startThe decision is who answers first, not whether chat exists
Held constantChat button and its hours, staffing, phone, email, help centre, prices and offersOnly the first responder differs, so the result belongs to it
Assistant content and scopeApproved product content and handover rules, frozen for the testChanging either mid-test changes the treatment
Kept human in both armsRefunds, complaints and exceptionsThe guide keeps these with people; the test does not change that
Primary metricContribution margin per chat visitor, after returns and the cost of answeringCounts every chat visitor and both costs of answering

Before the test starts, the assistant passes the AI customer service launch checklist: scope, approved content, handover, disclosure, data and a test set. The test set is built and scored as the guide to testing prompts and models describes, from real past product questions, so the assistant reaches customers having already shown it can answer them. The assistant answers only from approved content, which is what Grounding means here, cites what it used, and hands over when the content does not cover the question. The academy exercise on planning a coffee machine retailer's assistant-first chat test practises writing this plan from entry condition to decision rule, with every chat visitor counted, a limit set for each guardrail and the reading held until orders have passed the returns window.

Keep the chat's hours the same in both arms. An assistant answering overnight while the staff arm shows a closed window measures opening hours, not who answers first; that is a separate decision. The support channels and self-service guide sets the handover rule both arms follow: a person reachable at any point without defeating the bot, receiving the whole conversation.

The assistant is part of the treatment, and so is what it says. The AI customer service guide, drawing on the European Commission's overview of its AI regulatory framework and the FTC's business guidance, says customers should be told when they are talking to an automated assistant, and that an assistant presented as a person is treated like any other deceptive practice; the guide to answering customer questions about AI sets the same rule for the moment a visitor asks. The guide to answering pre-purchase questions records that the Advertising Standards Authority's codes are the reference for marketing claims, and that an answer which promises more than the product page and the terms carries the same risk as the claim on the page. Both apply to the assistant in full. This is general direction, not legal advice.

Tag every chat in both arms with a contact reason at close, using the list built as the tutorial on tagging support contacts by reason describes, and name before launch the question types the result will be read by.

Sample size reasoning

Only visitors who open chat enter the test, so the sample is far smaller than product-page traffic, and margin per visitor is mostly zeros with a few orders, which makes it noisy. The inputs are chats per week, the share of chat visitors who order, how much margin per visitor varies and the smallest difference worth changing who answers first. Because the question is whether the assistant is no worse rather than better, the margin the team will accept is part of the plan. The sample size and significance guide shows how those combine and why intervals matter more than a single figure, and the conversion rate calculator turns each arm's orders and chat visitors into the rates it works with.

If chat volume is low, read orders per chat visitor as the primary metric instead, and treat the saving in staff time as known from the rota rather than estimated from the test.

Duration

Run whole weekly cycles that cover every pattern of staffed hours the chat runs, weekday and weekend, busy and quiet, with the assistant's content, scope and handover rules frozen. Avoid launches and sales, when the questions change. Then keep the reading open until orders placed during the test have passed the return period, because the returns are where an overconfident answer shows.

Reading the results and the decision rule

Read once, after the return window. Compare contribution per chat visitor between arms with the interval around the difference, against the margin set before launch. Then read the guardrails, overall and by the question types named in advance. Read the transcripts last, for what went wrong. Write the rule before launch:

ResultDecision
Contribution per chat visitor is reliably no lower, within the margin set before launch, and accuracy, handover and repeat-contact guardrails stay within their limitsLet the assistant answer first, at the scope tested
Margin falls reliably, or any guardrail breaks across question typesKeep staff-only chat
Results differ by question types named before launch, with some passing every guardrailNarrow the assistant's scope to the question types that passed

The test measures the assistant at the content, scope and handover tested. A new model, a rewritten prompt or a restructured help centre is a different treatment: re-run the test set first, and a new test if the change is large.

Pitfalls

Deflection read as success is the first trap. A rising share of chats closed without staff looks like a saving and may be visitors giving up. The primary metric counts them; the deflection figure does not.

Content changed mid-test is the second. A product sheet corrected partway through changes the assistant's answers from that day. Freeze the content, log any correction that cannot wait, and read the weeks either side separately.

A handover that is hard to find is the third. If the route to staff sits behind several replies, the assistant arm tests patience, not the assistant. The handover guardrail catches it only if it is measured from the visitor's first request.

Staff answering assistant chats from the queue is the fourth. When staff can see the assistant's conversations and step in before a handover, the arms converge. Staff join an assistant chat only when it is handed over.

Accuracy checked only on easy questions is the fifth. A sample drawn from conversations that ended well flatters the assistant; sample at random across every question type, and add every handover and every complaint. The guide to common testing pitfalls covers the general traps, from a broken split to reading early. For enquiries where the question is how fast a person responds, the lead response time test is the design.

Example: an invented online telescope shop tests an assistant in its product-page chat (illustrative)

Illustrative example — not a real result

Plovitzer Telescopes tests an assistant that answers first in chat

Plovitzer Telescopes, an invented online telescope shop, runs staffed chat on its product pages and wants to know whether an assistant answering from its compatibility guides can take the first reply. Over eight weeks, 6,400 visitors open chat and are split at random, 3,200 per arm. Before launch the team sets three limits: contribution per chat visitor no more than £1.00 below staff-only chat, at least 95% of sampled answers matching approved content in every question type, and a person reached within three minutes of asking. The reading closes after the 30-day return period.

Per armStaff onlyAssistant first
Chat visitors3,2003,200
Orders544 (17.0%)528 (16.5%)
Contribution after returns£63,360£61,760
Staff time420 hours, £7,560190 hours, £3,420
Assistant fees—£1,600
Contribution per chat visitor after answering costs£17.44£17.73

The assistant arm earns £0.29 more per chat visitor, with an interval of roughly −£0.70 to £1.28, inside the margin. Visitors who ask for a person reach one in about one and a half minutes, and repeat contacts are close between arms. Accuracy splits by question type: 117 of 120 sampled compatibility answers and 59 of 60 delivery answers match the approved content, but only 49 of 60 answers about choosing a first telescope do. Plovitzer narrows the assistant to compatibility and delivery questions and keeps first-telescope advice with staff.

Common mistakes

Assistant tests usually fail on whom they count and what they call success. The experiment library lists the page, offer and lead-handling designs that sit beside this one.

  • Comparing assistant chats with staff chats. The assistant hands over its hardest questions; compare the arms whole.
  • Reporting deflection. A chat nobody answered is not a saving if the visitor left.
  • Testing an assistant that has not passed its checklist. The test is not where accuracy is discovered.
  • Changing content or prompts mid-test. The treatment is the assistant as frozen at launch.
  • Opening chat for more hours in one arm. That tests availability, not who answers first.
  • Sampling only easy questions. Accuracy is read across every question type, handovers and complaints included.

Frequently asked questions

Why not compare the chats the assistant handled with the chats staff handled?

Because the assistant hands its hardest conversations to staff, so the chats it keeps are the easy ones and the chats staff receive are the difficult ones. Any comparison between them measures that sorting, not the effect of letting the assistant answer first. Randomising visitors before the first answer, and reading everyone in each arm including those who were handed over, is the only reading that isolates what the change does.

Why is deflection not the measure of success?

A chat that never reached a person can mean the question was answered or that the visitor gave up. An assistant that makes people work to reach staff scores well on deflection while orders fall and complaints rise. The test reads what the business keeps, margin per chat visitor after returns and the cost of answering, and treats the staff time saved as one of the costs inside that figure rather than as the result.

Does the assistant have to say it is automated?

The site's guides set the practical rule: say it is an automated assistant at the start of the conversation and whenever asked, and offer a person at any point. They describe this as the direction of regulation and consumer protection guidance in several regions, where presenting an assistant as a person is treated as deceptive. It applies in the test arm exactly as it would after launch. This is general direction, not legal advice; take advice on the rules where the business operates.

Terms used on this page

Tools to try

Frameworks

Where this page is used

Part of these learning paths

Also related

Sources

  1. Regulatory framework proposal on artificial intelligence — European Commission
  2. FTC business guidance — Federal Trade Commission
  3. Advertising Standards Authority — ASA

Related