What this checkpoint covers
Griskeisk Ski Hire is an invented online ski and snowboard hire shop. Its support team answers chat and email, and is adding an AI assistant to the chat and AI drafts to the inbox.
Illustrative example — not a real result
Griskeisk Ski Hire: the queue, the library and the supplier's readout
| Contact reason, a month in season | Contacts |
|---|---|
| Change my booking dates (a fee may apply; depends on stock at the resort) | 900 |
| Boot, ski and board sizing | 640 |
| Cancellations and refunds | 380 |
| What the damage waiver covers | 260 |
- 5 agents in season, measured on replies sent per hour.
- Two versions of the damage waiver article are live: last season's and this season's.
- The supplier's 6-week test, 2,400 chat visitors per arm: 68% of chats closed without staff; chats the assistant kept booked at 31%, staff-only chats at 24%; accuracy checked on 40 chats, all rated 5 stars by customers.
Task
- Write each answer with its reasoning first.
- Score against the explanations; a sound action defended for an unsound reason counts as a miss.
- Seven or eight: carry on. Fewer: revisit the page each miss links.
Questions
- The supplier wants the assistant on date changes, the largest queue, and the damage waiver left to staff. What do you decide for each, and what must happen to the waiver first?
- Agents answer sizing and cancellation policy questions every day. Which suits a saved reply, which suits a draft, and who owns the answer in each case?
- With drafting on, one agent's replies per hour rise sharply; she sends drafts unchanged. What is wrong, and what should have changed first?
- The drafting tool reads the help centre but not the saved reply library, "to keep things clean". A draft for a customer whose flight was cancelled offers a full refund and new dates. What changes?
- A month in, the team lead reports that drafts beat saved replies because handling time fell. How should the two methods be judged?
- Read the supplier's booking comparison. What should the test have reported as its result?
- Given the supplier's headline and accuracy check, what should the guardrails have been?
- Mid-season the supplier upgrades the model, and next season the help centre is restructured. Can the team keep relying on this test?
Answers explained
- By risk, not by queue length, with the library fixed first. A date change can carry a fee and depends on stock, a commitment that stays with agents. The waiver question is low harm and fully covered once one current article exists, so last season's comes out first. The AI customer service launch checklist will not pass while any Contact ReasonThe main reason a customer contacted support, tagged by the agent from a fixed, defined list so volume and other service metrics can be read by cause. is unplaced or the library holds conflicting versions.
- Cancellation policy to a saved reply, sizing to drafts. Every customer must hear the same cancellation terms, so a Saved ReplyA pre-written answer a support agent inserts and then edits for the customer in front of them, keeping facts and tone consistent across the team. owned by whoever sets the policy holds them; sizing questions vary, and complete sizing guides can answer them. The comparison of saved replies and AI drafts sets that split.
- Drafts are being sent unread, which quietly turns assist into automate. The agent checks each fact, edits and sends the reply as her own: the Human ReviewThe step where a named person checks AI-assisted work for accuracy, sources, originality, privacy and disclosure before it is published or sent. drafting depends on. A replies-per-hour target rewards skipping it, so the measure changes first. The guide to AI in customer service sets the agent's checks.
- The library becomes a source, and the commitment returns to the agent. Drafts that draw on the owned replies carry the agreed wording, and no draft offers a refund or new dates: the agent writes those. The saved reply library tutorial explains why outdated replies matter more once a team drafts with AI.
- On what customers received, question type by question type. Read a Repeat ContactA customer contacting the business again about the same problem, the clearest sign that the first answer did not end the customer's work. within a set window and sampled accuracy, by contact reason. The quality review tutorial for support conversations gives the scorecard that reads both methods alike.
- Contribution per chat visitor after refunds and the cost of answering, counting everyone in each arm. The assistant kept the easy chats, so its booking rate says nothing about the arrangement. The AI chat assistant test fixes that metric before launch.
- Accuracy, time to a person and repeat contacts, with limits set in advance. Accuracy is sampled at random across every question type, not drawn from happy chats. The share closed without staff sits beside the result, not as it: the Goodhart's law framework warns that a measure made into a target stops tracking what it was meant to.
- Each change is a new treatment. The test measured one model, one set of instructions and one help centre. After the upgrade the stored test set runs again before the assistant keeps answering, and the restructured help centre may need a new test. The prompt and model testing guide says which changes call for that.
Where to go next
Missed the first, third or fourth? Redo the scope and reply split in the coffee machine retailer exercise. For the grounding these answers assume, see the advanced AI in marketing level; the last module tests what ad platform automation adds, and its exercise is the model railway retailer's broad match and audience tests.
