Why this matters
A help assistant grounded at the advanced AI in marketing level can answer; whether it should, and whether the business gains, are expert decisions. The guide to AI in customer service places each contact by harm, coverage and judgement, the AI customer service launch checklist gates the launch, and the AI chat assistant test decides on margin whether it answers first.
The retailer and its chat queue
Skyzeilt Coffee Machines is an invented online retailer of espresso machines, grinders and spare parts. Staff answer product-page chat in opening hours, email otherwise. A supplier proposes an assistant that replies first in chat and drafts replies for agents, with success reported as the share of chats closed without staff.
Skyzeilt Coffee Machines: the team and the contact reasons support tags
Team: a service lead, 4 agents, a technical lead, an operations manager and a head of ecommerce.
| Contact reason | A month | Mainly by | Content behind the answer |
|---|---|---|---|
| Will this fit my machine | 480 | Chat | Owned compatibility tables |
| Which machine should I buy | 340 | Chat | Buying guides for 4 of the 9 machine families |
| Delivery times before ordering | 290 | Chat | Delivery policy |
| Where is my order | 560 | Read-only order lookup | |
| Spare part for my model | 260 | Chat | Parts diagrams; some parts changed between production years |
| Descaling and cleaning | 310 | Owned, current how-to articles | |
| Returns and refunds | 220 | Returns policy | |
| Price match requests | 130 | Chat | Price match policy; exceptions case by case |
| Faulty machine | 270 | Email and phone | Warranty terms; safety notice |
| Complaints | 80 | All | None |
- Product-page chat: about 1,500 visitors open it a month. Returns window: 30 days.
- Agents are measured on chats closed per hour.
- Saved replies: 46; 11 have no owner, and 2 still state last year's returns window.
- The autumn sale runs for the last 2 weeks of November.
Scope by risk, judge by margin
Scoping by volume treats the biggest queue as the biggest saving, whatever a wrong answer there would cost; the guide's three questions place each Contact ReasonThe main reason a customer contacted support, tagged by the agent from a fixed, defined list so volume and other service metrics can be read by cause. on its own facts. Judging the assistant on the chats it closed leaves it the simple ones and counts buyers who left, so the test splits visitors before anyone replies and reads Contribution MarginRevenue left after every cost that varies with each sale. per chat visitor in each arm, not DeflectionThe share of customer contacts handled without ever reaching a person, usually by an assistant or help content; read beside resolution, never as the goal..
Task
- Place each contact reason in automate, assist or keep human, and say what would move it.
- Split answers between owned saved replies, AI drafts and the agent's own words, naming each owner.
- Say what must change before drafts reach agents.
- Design the test of assistant-first chat, from entry condition to decision rule.
- Say what happens to the supplier's measure.
Worked answer
The sort. Each reason is placed by harm, coverage and judgement, not volume.
| Contact reason | Decision | Why |
|---|---|---|
| Delivery; order status; descaling | Automate | Low harm; a policy, a lookup and owned articles cover them |
| Will this fit my machine | Automate, handing over when the model is unclear | Owned tables cover it; sampled, since a wrong answer costs a return |
| Returns: how and when | Automate once the replies giving last year's window are corrected; by email, agents send the owned saved reply | Low harm; the returns policy covers it fully |
| Spare part for my model | Assist | Parts changed between production years, so coverage is in doubt |
| Which machine should I buy | Assist where a buying guide exists; agents write the rest | Guides cover only some families, so coverage is in doubt |
| Refund and price match decisions | Keep human | They commit money and set precedent |
| Faulty machines; complaints | Keep human, faults routed at once | Safety, warranty and recovery need a person |
The assistant runs in chat only, so agents answer automated reasons that arrive by email. Spare parts move to automate once each diagram shows its production-year changes and the test questions pass.
Replies and drafts. The comparison between owned saved replies and AI drafts gives recurring policy answers to owned replies and varied product questions to drafts. Each Saved ReplyA pre-written answer a support agent inserts and then edits for the customer in front of them, keeping facts and tone consistent across the team. names its owner: the operations manager for delivery and returns, the technical lead for warranty, the head of ecommerce for price match. Drafts take compatibility, parts, guide-backed buying advice and descaling, drawing on the library. Agents write every refund, promised date, exception and complaint reply. First, unowned replies get an owner or go, and the returns window is corrected.
Before drafts arrive. A target of chats closed per hour rewards sending drafts unread, so it changes first, to repeat contacts and quality scores by contact reason.
Skyzeilt Coffee Machines: the test plan, written before launch
| Element | Choice |
|---|---|
| Entry condition | The launch checklist passed, including 150 real past product-page questions scored against a rubric, and a named weekly reviewer |
| Unit and arms | Visitors randomised at their first product-page chat and held in that arm: assistant first with staff on offer, against staff from the start |
| Held constant | Chat hours, staffing, other routes, prices and offers; the same replies and drafts in both arms; the assistant's content, scope and instructions frozen |
| Primary metric | Contribution after returns on the arm's chat visitors' orders, less staff time and assistant fees, divided by everyone in the arm who opened chat |
| Guardrails | 96% or more of sampled answers match approved content in each in-scope question type; a person joins within 2 minutes on 9 requests in 10; repeat contacts within 14 days at most 2 points above staff-only |
| Margin | No more than £0.80 per chat visitor below staff-only |
| Duration | 8 full weeks from early September, ending before the autumn sale; read once the last orders pass the returns window |
| Volume check | About 2,800 chat visitors, 1,400 per arm; if a sample size reading says that cannot separate the arms at the margin, orders per chat visitor becomes primary, the staff saving taken from the rota |
| Decision rule | The test page's rule, read against the margin and guardrail limits above, written before launch |
The supplier's measure. Chats closed without staff are reported beside the result, not as it. A new model, rewritten instructions or restructured help centre is a new treatment, so the test set runs again first.
Common mistakes
- Automating the biggest queue. Volume says nothing about harm.
- Sending refunds or exceptions in a draft. Commitments are the agent's own words.
- Adding drafts while speed is the target. The measure changes first.
- Comparing the assistant's chats with the staff's. Handovers leave staff the hard questions.
