When to run this checklist
Run it before an AI assistant starts answering customers in chat, messaging or email, and before an AI tool starts drafting replies that agents edit and send. Run it again before the assistant's scope changes (a new contact reason moves from assist to automate), before its content source changes (a help centre is restructured or a new policy library is connected), and before its model or prompt changes, because a test passed on one combination of model, prompt and content says little about the next. Passing this list is also the entry condition for the AI chat assistant test, which compares assistant-first and staff-only product-page chat once the assistant is ready to answer. The academy exercise on scoping, gating and testing a coffee machine retailer's chat assistant practises sorting the retailer's contact reasons by risk, splitting owned saved replies from drafts and writing the test this list opens, judged on contribution per chat visitor rather than chats closed without staff.
The customer service lead runs the list, with the owner of the help content for the content lines and whoever owns the business's AI use policyA team's written rules for AI: approved tools, what each use may do, what data may go in, what is reviewed, when customers are told and who owns problems. for the scope and data lines. A vendor or implementation partner can do much of the work, but the ticks belong to the people who will answer for the assistant's replies.
Two neighbouring lists sit on either side. The governance checklist in the disclosure and data privacy guide settles disclosure, the data that may enter a tool, vendor terms and retention for any AI use; run it first, and this list points to it rather than repeating its data and vendor lines. The AI-assisted content review checklist gates copy that is published, not conversations. Where the assistant sits inside an automated journey, such as an order confirmation that invites a chat, the email automation flow launch checklist covers the flow itself. How to build a test setA collection of real inputs for an AI task, each noting what a good output must contain, run to compare prompts or models and rerun when they change. and rubric is taught in the guide to testing prompts and models; this list only asks whether that test was run.
Before you start
- The list of contact reasons support already tags, with a note of which ones are complaints, refunds, exceptions, legal matters or likely to involve a customer in a vulnerable situation.
- The help content the assistant will answer from, with each article's owner and last review.
- The written service tone, including approved and rejected example replies, built on the brand voiceThe consistent personality a brand expresses in its words across channels, with tone adjusting to the situation and the audience..
- The business's AI use policy and its register of approved uses and owners.
- The handover design: how a customer reaches a person, what the person receives, and what happens out of hours.
- The integrations the assistant will use to read account, order or booking data.
- The test set, rubric, pass bar and results from testing the assistant on real past questions.
- The quality scorecard the team uses to review human conversations.
The checklist
The groups follow the order in which the decisions are made: what the assistant handles, what it answers from, how customers leave it and know what it is, what data it sees, and how its answers are tested and watched. Scope comes first because every later line depends on which contact reasons are in play.
Scope
Content and answers
Handover and disclosure
Data
Testing and review
Where each check is explained
The widget keeps the lines short so they can be ticked; the reasoning lives in the guides. Each check links to the page that explains it. The tutorial on writing the standing instructions for an AI assistant turns the scope, content, handover, disclosure and tone lines into the written instructions the assistant follows, with a test conversation for each.
| Check | Explained in |
|---|---|
| Each contact reason sorted into automate, assist or keep human | Scoping service work in the AI customer service guide |
| Complaints, vulnerable customers and legal matters routed to people | Keep-human tasks in the AI customer service guide |
| Scope started from the contact reasons support already tags | Tagging support contacts by reason |
| Use and owner recorded under the AI use policy | Writing a marketing AI use policy |
| Answers only from approved help content, citing the article | Answering from approved content |
| An owner for every help article, updated before policy changes | Support channels and self-service |
| Source library current, with no conflicting versions | How grounding and retrieval work |
| Service tone written with approved and rejected examples | Writing customer service tone guidelines |
| Agents write any reply that commits a refund, date or exception | Drafting replies for agents |
| A person reachable at any point | Chatbots and escalation in the support channels guide |
| Full conversation passed at handover | Handover to a person |
| Automated assistant disclosed at the start and when asked | Disclosure and data privacy for AI in marketing |
| Honest message when no person is available | Handover to a person |
| Account access only through read-only, defined integrations | Answering from approved content |
| Personal data limited to what the task needs | Minimising and redacting data before prompting |
| Personal data masked in review logs | Retention in the disclosure and data privacy guide |
| Tested on real past questions with a rubric and pass bar | Testing prompts and models before you rely on them |
| Re-test planned for model or prompt changes | Re-testing when the model changes |
| Weekly sample review booked | Disclosure and sampling review |
| Same scorecard for automated and human conversations | Reviewing support conversations for quality |
| Service quality and repeat contact reported, not deflection | Measuring service quality, not deflection |
If an item fails
The scope and handover groups are the hard stops. If any contact reason has not been placed, the assistant answers it by default, so it does not go live until every reason has a decision. If complaints, legal matters or signs of a customer in a vulnerable situation do not route to people, stop: an assistant that tries to handle a complaint usually makes it worse, and the recovery sequence in the service recovery and complaints guide depends on a person who can take responsibility. A customer who cannot reach a person, or who has to repeat everything after reaching one, is the failure customers remember, so a broken handover also stops the launch.
Disclosure is not a line to note for later. Both the AI customer service guide and the disclosure guide read the European Commission's overview of its approach to AI as expecting people to be told when they are interacting with an AI system, and the FTC's business guidance as holding that deception standards do not change with the technology. An assistant that does not say it is automated does not go live. That is general direction from those pages, not legal advice.
In the content group, an article without an owner or a library with two versions of the same policy is the line most often found failing, and the one that turns into wrong answers delivered confidently. Grounding narrows invented answers to what the library says; it cannot fix a library that says the wrong thing. Fix the library, then re-run the affected test questions. A tone guide still in draft can go ahead with a note if agents review every draft, but not for automated answers. The academy checkpoint on scoping assistants, owning replies and reading chat tests tests these lines at a ski and snowboard hire shop, starting with a supplier who wants its largest queue automated while two versions of one policy article are still live.
The data group stops the launch when the assistant can change account data, or can see more personal dataAny information relating to an identified or identifiable living person, such as a name, email address, location data or an online identifier. than the task needs. Account and order data is first-party dataData a business collects directly from its own customers, visitors and users. collected for service; the minimising data tutorial shows how to strip it back.
A failed test means the assistant does not go live for the failing contact reasons. Narrow the scope to the reasons that passed, move the rest to assist, and re-test after the fix. The review lines can be finished on the day, but the first weekly sample must be booked before switch-on, with a named reviewer. The NIST AI Risk Management Framework, as the AI use policy tutorial describes it, pairs measuring how a system behaves in practice with managing what is found; a review nobody has booked does neither. If the report still leads with deflectionThe share of customer contacts handled without ever reaching a person, usually by an assistant or help content; read beside resolution, never as the goal., rewrite it before launch: Goodhart's law explains why a target the assistant can hit by being hard to get past stops measuring service, and a repeat-contact guardrail metricA metric an experiment must not worsen beyond an agreed bound, watched beside the primary metric that decides the test. keeps the report honest.
A furniture retailer switches on an assistant (illustrative)
Delivery and assembly questions, with complaints kept human
Thexilo Furniture, an invented online furniture retailer, plans a chat assistant for its two busiest contact reasons: delivery status and assembly questions. The customer service lead runs the list with the help centre owner and the operations director, who owns the AI use policy. Scope passes after one change: damaged-delivery reports had been tagged under delivery status, so the team splits them into their own reason and keeps them human, because each one is a complaint that may need a replacement or refund. The test set of 120 real past questions fails the content group. Eleven answers about returning flat-pack items quote a returns article that was replaced eight months earlier but never unpublished, so the assistant has two versions to choose from. The help centre owner removes the old article, names an owner for each of the 34 delivery and assembly articles, and the affected questions are re-run and pass. The handover passes: customers can type for a person at any point, and agents see the full chat. The data lines pass because the order lookup is read-only. Before switch-on, the lead books the first weekly sample review of 40 automated conversations plus every handover, scored on the same scorecard as human replies, and the launch report tracks repeat contact on delivery questions within a week rather than the number of chats closed.
Common mistakes
- Scoping by volume instead of risk. The commonest contact reason is not automatically safe to automate.
- Treating the help centre as finished. The assistant repeats whatever the content says, including the article that should have been retired.
- Designing the handover last. A route to a person that customers have to fight for turns the assistant into the reason for the complaint.
- Leaving disclosure to the vendor's default greeting. Check what the customer actually sees at the start and when they ask.
- Letting drafts commit the business. Refunds, dates and exceptions written by a draft and sent unread become promises nobody decided to make.
- Testing once. A model or prompt change after launch quietly replaces the assistant that passed.
- Reviewing only automated conversations. Without the same scorecard for human replies there is nothing to compare the assistant against.
- Reporting deflection. Customers who give up look like savings until they leave.
The checklists library holds the lists that sit beside this one, from AI-assisted content review to the launch of an automated email flow.
