Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

AI in Marketing › Expert

Launching and Testing a Chat Assistant for a Coffee Machine Retailer

You take an invented coffee machine retailer's plan to let an AI assistant answer product-page chat first and draft replies for its agents, and turn it into decisions the business can defend. You finish with every contact reason placed in automate, assist or keep human, a split between owned saved replies and AI drafts, and a test of assistant-first chat that counts every chat visitor, carries guardrails and a margin set in advance, and states its decision rule before launch.
ExerciseExpert levelModule 2: AI that answers customers5 minUpdated

Why this matters

A help assistant grounded at the advanced AI in marketing level can answer; whether it should, and whether the business gains, are expert decisions. The guide to AI in customer service places each contact by harm, coverage and judgement, the AI customer service launch checklist gates the launch, and the AI chat assistant test decides on margin whether it answers first.

The retailer and its chat queue

Skyzeilt Coffee Machines is an invented online retailer of espresso machines, grinders and spare parts. Staff answer product-page chat in opening hours, email otherwise. A supplier proposes an assistant that replies first in chat and drafts replies for agents, with success reported as the share of chats closed without staff.

Illustrative example — not a real result

Skyzeilt Coffee Machines: the team and the contact reasons support tags

Team: a service lead, 4 agents, a technical lead, an operations manager and a head of ecommerce.

Contact reasonA monthMainly byContent behind the answer
Will this fit my machine480ChatOwned compatibility tables
Which machine should I buy340ChatBuying guides for 4 of the 9 machine families
Delivery times before ordering290ChatDelivery policy
Where is my order560EmailRead-only order lookup
Spare part for my model260ChatParts diagrams; some parts changed between production years
Descaling and cleaning310EmailOwned, current how-to articles
Returns and refunds220EmailReturns policy
Price match requests130ChatPrice match policy; exceptions case by case
Faulty machine270Email and phoneWarranty terms; safety notice
Complaints80AllNone
  • Product-page chat: about 1,500 visitors open it a month. Returns window: 30 days.
  • Agents are measured on chats closed per hour.
  • Saved replies: 46; 11 have no owner, and 2 still state last year's returns window.
  • The autumn sale runs for the last 2 weeks of November.

Scope by risk, judge by margin

Scoping by volume treats the biggest queue as the biggest saving, whatever a wrong answer there would cost; the guide's three questions place each Contact Reason on its own facts. Judging the assistant on the chats it closed leaves it the simple ones and counts buyers who left, so the test splits visitors before anyone replies and reads Contribution Margin per chat visitor in each arm, not Deflection.

Task

  1. Place each contact reason in automate, assist or keep human, and say what would move it.
  2. Split answers between owned saved replies, AI drafts and the agent's own words, naming each owner.
  3. Say what must change before drafts reach agents.
  4. Design the test of assistant-first chat, from entry condition to decision rule.
  5. Say what happens to the supplier's measure.

Worked answer

The sort. Each reason is placed by harm, coverage and judgement, not volume.

Contact reasonDecisionWhy
Delivery; order status; descalingAutomateLow harm; a policy, a lookup and owned articles cover them
Will this fit my machineAutomate, handing over when the model is unclearOwned tables cover it; sampled, since a wrong answer costs a return
Returns: how and whenAutomate once the replies giving last year's window are corrected; by email, agents send the owned saved replyLow harm; the returns policy covers it fully
Spare part for my modelAssistParts changed between production years, so coverage is in doubt
Which machine should I buyAssist where a buying guide exists; agents write the restGuides cover only some families, so coverage is in doubt
Refund and price match decisionsKeep humanThey commit money and set precedent
Faulty machines; complaintsKeep human, faults routed at onceSafety, warranty and recovery need a person

The assistant runs in chat only, so agents answer automated reasons that arrive by email. Spare parts move to automate once each diagram shows its production-year changes and the test questions pass.

Replies and drafts. The comparison between owned saved replies and AI drafts gives recurring policy answers to owned replies and varied product questions to drafts. Each Saved Reply names its owner: the operations manager for delivery and returns, the technical lead for warranty, the head of ecommerce for price match. Drafts take compatibility, parts, guide-backed buying advice and descaling, drawing on the library. Agents write every refund, promised date, exception and complaint reply. First, unowned replies get an owner or go, and the returns window is corrected.

Before drafts arrive. A target of chats closed per hour rewards sending drafts unread, so it changes first, to repeat contacts and quality scores by contact reason.

Illustrative example — not a real result

Skyzeilt Coffee Machines: the test plan, written before launch

ElementChoice
Entry conditionThe launch checklist passed, including 150 real past product-page questions scored against a rubric, and a named weekly reviewer
Unit and armsVisitors randomised at their first product-page chat and held in that arm: assistant first with staff on offer, against staff from the start
Held constantChat hours, staffing, other routes, prices and offers; the same replies and drafts in both arms; the assistant's content, scope and instructions frozen
Primary metricContribution after returns on the arm's chat visitors' orders, less staff time and assistant fees, divided by everyone in the arm who opened chat
Guardrails96% or more of sampled answers match approved content in each in-scope question type; a person joins within 2 minutes on 9 requests in 10; repeat contacts within 14 days at most 2 points above staff-only
MarginNo more than £0.80 per chat visitor below staff-only
Duration8 full weeks from early September, ending before the autumn sale; read once the last orders pass the returns window
Volume checkAbout 2,800 chat visitors, 1,400 per arm; if a sample size reading says that cannot separate the arms at the margin, orders per chat visitor becomes primary, the staff saving taken from the rota
Decision ruleThe test page's rule, read against the margin and guardrail limits above, written before launch

The supplier's measure. Chats closed without staff are reported beside the result, not as it. A new model, rewritten instructions or restructured help centre is a new treatment, so the test set runs again first.

Common mistakes

  • Automating the biggest queue. Volume says nothing about harm.
  • Sending refunds or exceptions in a draft. Commitments are the agent's own words.
  • Adding drafts while speed is the target. The measure changes first.
  • Comparing the assistant's chats with the staff's. Handovers leave staff the hard questions.

Pages this lesson applies

Terms used on this page

Also related

Related