Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

AI in Marketing › Expert

Checkpoint: AI Assistants, Reply Drafts and Chat Tests

Scenario questions set at an invented ski and snowboard hire shop test the second expert module: placing contact reasons by risk, keeping policy wording in owned replies while drafts take the varied questions, keeping commitments with agents, judging both methods on what customers received, and reading an assistant-first chat test on every chat visitor with its guardrails. Answer each question in writing before opening the explanations, then reread the page behind any miss.
CheckpointExpert levelModule 2: AI that answers customers4 minUpdated

What this checkpoint covers

Griskeisk Ski Hire is an invented online ski and snowboard hire shop. Its support team answers chat and email, and is adding an AI assistant to the chat and AI drafts to the inbox.

Illustrative example — not a real result

Griskeisk Ski Hire: the queue, the library and the supplier's readout

Contact reason, a month in seasonContacts
Change my booking dates (a fee may apply; depends on stock at the resort)900
Boot, ski and board sizing640
Cancellations and refunds380
What the damage waiver covers260
  • 5 agents in season, measured on replies sent per hour.
  • Two versions of the damage waiver article are live: last season's and this season's.
  • The supplier's 6-week test, 2,400 chat visitors per arm: 68% of chats closed without staff; chats the assistant kept booked at 31%, staff-only chats at 24%; accuracy checked on 40 chats, all rated 5 stars by customers.

Task

  • Write each answer with its reasoning first.
  • Score against the explanations; a sound action defended for an unsound reason counts as a miss.
  • Seven or eight: carry on. Fewer: revisit the page each miss links.

Questions

  1. The supplier wants the assistant on date changes, the largest queue, and the damage waiver left to staff. What do you decide for each, and what must happen to the waiver first?
  2. Agents answer sizing and cancellation policy questions every day. Which suits a saved reply, which suits a draft, and who owns the answer in each case?
  3. With drafting on, one agent's replies per hour rise sharply; she sends drafts unchanged. What is wrong, and what should have changed first?
  4. The drafting tool reads the help centre but not the saved reply library, "to keep things clean". A draft for a customer whose flight was cancelled offers a full refund and new dates. What changes?
  5. A month in, the team lead reports that drafts beat saved replies because handling time fell. How should the two methods be judged?
  6. Read the supplier's booking comparison. What should the test have reported as its result?
  7. Given the supplier's headline and accuracy check, what should the guardrails have been?
  8. Mid-season the supplier upgrades the model, and next season the help centre is restructured. Can the team keep relying on this test?

Answers explained

  1. By risk, not by queue length, with the library fixed first. A date change can carry a fee and depends on stock, a commitment that stays with agents. The waiver question is low harm and fully covered once one current article exists, so last season's comes out first. The AI customer service launch checklist will not pass while any Contact Reason is unplaced or the library holds conflicting versions.
  2. Cancellation policy to a saved reply, sizing to drafts. Every customer must hear the same cancellation terms, so a Saved Reply owned by whoever sets the policy holds them; sizing questions vary, and complete sizing guides can answer them. The comparison of saved replies and AI drafts sets that split.
  3. Drafts are being sent unread, which quietly turns assist into automate. The agent checks each fact, edits and sends the reply as her own: the Human Review drafting depends on. A replies-per-hour target rewards skipping it, so the measure changes first. The guide to AI in customer service sets the agent's checks.
  4. The library becomes a source, and the commitment returns to the agent. Drafts that draw on the owned replies carry the agreed wording, and no draft offers a refund or new dates: the agent writes those. The saved reply library tutorial explains why outdated replies matter more once a team drafts with AI.
  5. On what customers received, question type by question type. Read a Repeat Contact within a set window and sampled accuracy, by contact reason. The quality review tutorial for support conversations gives the scorecard that reads both methods alike.
  6. Contribution per chat visitor after refunds and the cost of answering, counting everyone in each arm. The assistant kept the easy chats, so its booking rate says nothing about the arrangement. The AI chat assistant test fixes that metric before launch.
  7. Accuracy, time to a person and repeat contacts, with limits set in advance. Accuracy is sampled at random across every question type, not drawn from happy chats. The share closed without staff sits beside the result, not as it: the Goodhart's law framework warns that a measure made into a target stops tracking what it was meant to.
  8. Each change is a new treatment. The test measured one model, one set of instructions and one help centre. After the upgrade the stored test set runs again before the assistant keeps answering, and the restructured help centre may need a new test. The prompt and model testing guide says which changes call for that.

Where to go next

Missed the first, third or fourth? Redo the scope and reply split in the coffee machine retailer exercise. For the grounding these answers assume, see the advanced AI in marketing level; the last module tests what ad platform automation adds, and its exercise is the model railway retailer's broad match and audience tests.

Pages this lesson applies

Terms used on this page

Also related

Related