Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

AI in Marketing › Troubleshooting

AI Assistant Not Resolving Customer Questions

Why customers give up on an AI assistant and ask for a person: scope set by volume, gaps in the help, missed passages, refusals, a hidden handover, deflection.
Troubleshootingintermediate19 min readUpdated

First, read the conversations that ended with a request for a person

Start with the conversations, not the dashboard. Pull every conversation from a recent period in which the customer asked for a person, was handed over, stopped replying part-way through or got in touch again about the same thing through another channel, then add a random handful from the same weeks so the sample also holds conversations that looked fine. For each one, note the customer's question word for word, the contact reason it belongs to on the support team's list, what the assistant replied, whether any document in its library holds the answer, and, where the customer reached a person, how long that took and whether they had to tell the story again. If the team has no such list, the tutorial that builds a reason list for support contacts creates one; without it, the sample can be sorted only by impression.

Then sort the sample. Each conversation that did not end as it should lands in one of a small set of groups, and the group points at the cause.

  • The question belongs to a reason the assistant should not handle: a disputed charge, a complaint, a request for an exception, a customer who may be in difficulty. No document fixes this; it is a scope decision.
  • The question is in scope and no document answers it. The assistant said so and offered a person, which is the assistant working as designed on a library with a gap.
  • A document answers it and the assistant did not find it. The customer's wording and the document's wording never met, or the answer sits in a passage that does not say what it is about.
  • The instructions shaped a reply that did not help: a hand-over rule that caught a routine question, a decline on a question the documents nearly answer, or a request to rephrase once more in a conversation that was already going in circles.
  • The customer wanted a person and the route failed: the option appeared only after several failed answers, nobody was available and nothing else was offered, or the person who took over started from scratch.
  • The assistant answered, and the answer was wrong. That is a different problem with its own trace: the diagnosis for assistants that give customers wrong answers follows each wrong answer back to the passage it cited.

Count the groups. A run of unresolved conversations can have more than one cause, and the counts say which to fix first. The last cause in the table below, success read as deflection, does not show in the conversations at all but in the report that should have surfaced them: if the weekly figures showed the assistant improving while these conversations piled up, the measure is part of the problem.

The causes have different owners, and naming them comes before any change. The support lead owns the scope and the route to a person; whoever owns each policy, product or process owns the article about it; the person responsible for the assistant holds the instructions and the library as a whole; whoever writes the service report owns the measures. The site's page on the NIST AI Risk Management Framework, written for marketing teams, puts measuring how a system really behaves, live output sampled included, beside mapping where it is used and governing who answers for it. Reading the failed conversations is that measurement, done before anyone touches a setting.

Diagnosis table

Diagnosis

SymptomCheckLikely causeFix
Customers ask for a person on the same few subjects, such as disputed charges, cancellations or anything needing an exceptionThe contact reasons in the assistant's scope against the help content and the harm a wrong answer could doContact reasons handed to the assistant because they were frequent, not because the content answers them or because they are safeMove reasons that need judgement, money or discretion to a person
The assistant says it has no information on a question it was meant to answer, and the library holds nothing on itThe unanswered questions against the help centre and the list of documents the assistant can reachQuestions no help article or document answersWrite the missing articles in the customers' words, each with an owner, before widening the assistant's scope
The assistant declines or offers an unrelated article, yet a current document answers the questionThe document's heading and opening lines against the customer's wording, and whether the passage names its subjectAn answer that exists but is never foundRewrite the section so it stands alone, names its subject and carries the customers' words beside the internal ones
The assistant hands over routine questions, refuses near matches or keeps asking the customer to rephraseThe standing instructions' scope, silence rule and hand-over list, and whether they carry the trigger for a conversation going in circlesInstructions that make the assistant decline, hand over too readily or go in circlesNarrow the hand-over list to named topics and signals, and add the trigger that hands a conversation going in circles to a person
Customers ask for a person several times, wait without being told anything or have to repeat the storyThe route to a person, the hours it is staffed, what happens outside them and what the person taking over receivesA route to a person that is hidden, unstaffed or loses the conversationOffer a person from the start, carry the full conversation across and give an honest alternative when nobody is free
The report shows the assistant improving while complaints, repeat contacts or phone volume riseWhat the report counts as success, and who reads the handovers and abandoned conversationsSuccess read as deflection, with failed conversations never readReport resolution, repeat contacts and time to reach a person by contact reason, and read the failed conversations every week

Fix, verify and prevent each cause

Contact reasons handed to the assistant by volume

The AI in customer service guide scopes service work by putting three questions to each contact reason: how much harm a wrong answer could do, whether approved content or a written rule answers it fully, and whether it needs judgement, discretion or empathy that no policy captures. Low harm with full coverage and no need for judgement points to automation; any doubt about harm or coverage points to assisting staff instead; a need for discretion keeps the reason with people, as do complaints, refunds and exceptions, legal and safety matters and customers in vulnerable situations. The failure behind this cause is scoping by size. The biggest reasons went to the assistant because they were the biggest, and some of them turn out to need a decision, an exception or access that the assistant does not have. The guide lists scoping by volume instead of risk among its common mistakes.

Fix: take the groups from the sample that ended with a request for a person and put each reason through the three questions again, with the support lead. Move reasons that need discretion or money to a person outright. Signals that a customer may be in difficulty go to people under a cautious rule, using the signs that the guidance on customers in vulnerable circumstances asks a support team to look out for.

Verify: the reasons moved to a person now reach one at the first message, and the next sample shows no requests for a person on reasons the assistant no longer owns.

Prevent: keep the scope as a written list of contact reasons with the decision recorded beside each, read every service measure by reason, and set the contacts the assistant handled beside those staff handled with its help and those staff handled alone, reason by reason, as the customer service guide recommends, so a reason that is failing customers shows up as that reason rather than disappearing into an overall figure. Revisit the list whenever a new product, policy or price changes what customers ask.

Questions no help article or document answers

Some unresolved conversations are the assistant behaving as it should. It was asked something in scope, found nothing in its library and said so, which is the silence rule doing its job; the gap is in the content. The support channels and self-service guide runs the help centre as a product with users, owners and a backlog, writes first the articles for the questions customers bring to support, in their own words, and reads the questions a bot cannot answer as gaps in the help centre. A library assembled from whatever articles the business happened to have, rather than from the questions customers bring, can leave holes in the places the assistant is asked about.

Fix: list the unanswered in-scope questions from the sample and the assistant's logs, group them by contact reason, and write the missing articles with the owner of the policy, product or process each describes: one question per article, the answer first, the customers' words in the heading. Add them to the library through the same routine as any other document, which the tutorial on getting an assistant's documents into shape lays down: an owner, a last-reviewed date and one current source for each fact.

Verify: put the questions that went unanswered to the assistant again; each should now be answered from the new article, the article should be cited, and the article should match the live policy.

Prevent: feed the assistant's unanswered questions into the help centre's backlog every week, beside searches that return nothing and articles followed by a contact, and tie new articles to launches and policy changes so the answer exists before customers start asking. Hold back any widening of the assistant's scope until the articles for the new reasons are written and tested.

An answer that exists but is never found

Here the library holds the right article and the assistant still declines, or offers something that answers a neighbouring question. How grounding and retrieval work explains the mechanism: a retrieval tool splits documents into passages and fetches those that best match the question, and the right material can be missed when the customer's words and the document's words do not meet, when a question hangs on a product code, which search by meaning can match badly, or when the answer sits in a passage cut away from its heading. The silence rule is meant to turn that miss into an honest reply that the documents do not cover the question, with an offer of a person, rather than an invention; grounding narrows invention without removing it, and even when the rule works, the customer gets no answer from the assistant. The wrong-answers diagnosis traces the same misses where the assistant answered anyway; this section covers the ones that end in a decline or an unrelated article.

Fix: for each conversation in this group, open the document that should have answered and read the relevant section as a passage on its own. Rewrite each such section by the preparing tutorial's steps for sections that stand alone and for rules held in tables. Where customers use a different word from the business, put both in the heading or the first line, as the preparing tutorial's step on customers' words describes.

Verify: rerun the questions from the sample; each should now return the intended passage, and the answer should cite it.

Prevent: add every phrasing that failed here, word for word, to the stored test questions, and run them again after any change to the library, the instructions or the tool, scored against a pass bar as the testing guide for prompts and models sets out for any stored test.

Instructions that make the assistant decline, hand over too readily or go in circles

The assistant's standing instructions decide what happens at the edges of its scope, and the tutorial on an assistant's standing instructions writes them rule by rule: the contact reasons it handles, what it answers from and what it says when the sources are silent, which topics and signals pass the conversation on, and a test conversation for every rule, including a question the documents nearly answer. Where a conversation in this group ended in a refusal or a handover the documents did not call for, find the rule that produced it. A hand-over list written as whole subjects, such as anything about tickets, sends routine questions to the queue. Where the assistant declined a question the documents nearly answered, check whether that near match was ever among the tutorial's test conversations. Where the assistant kept asking customers to try again, check whether the instructions carry the trigger the customer service guide names for a conversation going in circles.

Fix: read the instructions beside the conversations in this group. Rewrite the hand-over list as specific topics and signals rather than whole subjects, keeping the cautious rules for complaints, refunds, safety and customers who may be vulnerable. Where the trigger for a conversation going in circles is missing, add it, so that such a conversation is meant to pass to a person rather than ask the customer to try again. Give the change a new version.

Verify: write a test conversation for each rewritten rule and one that probes its edge, run them in the test environment and check that routine questions are answered, sensitive ones are handed over and a conversation going in circles ends in a handover.

Prevent: keep the instructions under the version, owner and rerun routine the tutorial sets out. If answers that used to pass start failing after a change nobody tested, the diagnosis for output that has got worse starts from the date of the change.

A route to a person that is hidden, unstaffed or loses the conversation

The support channels guide, linked above for gaps in the help centre, sets the standard for any bot: design the escalation path before the bot, let the customer reach a person whenever they choose, without first having to beat the bot, pass the full conversation and anything already collected to whoever takes over, and when nobody is available, say so honestly and offer something real instead, a call back or a promised written reply by a stated time. Each part of that can fail in a live assistant. The option to reach a person appears only after a run of failed answers. The handover works during staffed hours and sends the customer nowhere at night. The person who picks up sees a ticket number with no history and asks the customer to start again, which the customer service guide counts against handover quality, together with the wait before a customer who asked for someone got through.

Fix: make the route to a person visible from the first message and honour a request for a person at once, whatever the topic. Check that the full conversation, and anything the customer has already entered, arrives with the person who takes over. Write what the assistant says when nobody is available: that no one is free now, when someone will be, and a way to leave the question that will be answered by a stated time.

Verify: run test conversations through the handover during staffed hours and outside them, on a desktop and on a phone; each should reach a person or an honest alternative without the customer typing the request twice, and the person taking over should see the whole conversation.

Prevent: treat the route to a person as part of the assistant, with an owner and a check whenever the help desk or chat tool changes, and pass every change to the assistant's scope, content source, model or instructions back through the AI customer service launch checklist, whose handover and disclosure lines cover the route. When the support queue itself is growing, the diagnosis for a rising support queue ranks the contact reasons behind the rise; it also asks that a chat assistant be kept to what it answers well.

Success read as deflection, with failed conversations never read

The last cause is how the others can go unnoticed. Deflection, the proportion of contacts no person ever handled, rises when the assistant answers a question and also when the customer gives up, and a report built on it cannot tell the two apart. The customer service guide refuses to judge an assistant by it. The guide to customer service metrics explains the wider trap with the Goodhart's law framework: once a figure is a target, effort goes to moving the figure, and the counterweight is a second figure that is not allowed to slip while the first is chased. Here the counterweights are the ones customers feel: whether the problem ended, and how hard it was to reach someone when it did not.

Fix: stop reporting deflection as the assistant's result. Report by contact reason the measures the customer service guide puts in its place: resolution, read as the absence of a repeat contact inside a window the team fixes in advance, accuracy of the sampled answers and handover quality, with complaints about getting help beside them. Put the conversations behind those figures in front of someone every week: a random sample plus every conversation that ended with a request for a person, an abandoned chat, a complaint or a low rating, read against the content each should have drawn on, as the quality reviews of support conversations do for conversations handled by agents.

Verify: the next report shows the reasons the sample flagged, and the people who own the scope, the library and the instructions each receive a list of conversations to act on.

Prevent: make the weekly sampling review a booked task, give it a reader by name and send every finding where it can be fixed: scope questions to the support lead, missing or unfound answers to the document owners, instruction problems to the assistant's owner and handover failures to the support lead, who owns the route to a person. Where the team is still deciding whether the assistant should take the first reply in chat, the AI chat assistant test compares assistant-first chat with staff-only chat and watches how long people wait for a person, and whether they come back, as guardrails.

Prevention: a scope set by harm and coverage, a library that answers it, a route to a person that works and a weekly read of the failures

The four parts below are meant to stop the six causes returning: a written scope, a library that covers it, a route to a person that works, held in tested instructions and handover lines, and a weekly reading of what failed, each with an owner and a date.

The scope comes first. A written list of the contact reasons the assistant handles, decided by harm and coverage rather than by volume, is kept by the support lead. Nothing joins the list until its articles exist and its test questions pass, and the list is reviewed whenever a launch, a price change or a new policy changes what customers ask.

The library comes second. Every reason on the list has a current article, written in the customers' words, owned by the person who changes the policy or product it describes and linked to the process that ships releases and changes prices. The assistant's unanswered questions are read every week as the library's backlog, beside the help centre's own searches that return nothing.

The instructions and the route to a person come third: a short hand-over list of topics and signals, the trigger that hands a conversation going in circles to a person, a tested handover line and a tested line for when nobody is free. Each rule has a test conversation, and the set is versioned, owned and rerun whenever the model, the documents or the instructions change.

The reading comes fourth. A named person reads a weekly sample that includes every request for a person and every abandoned conversation, logs what they find by cause and routes each finding to its owner. The service report opens with resolution, repeat contacts and the time it takes to reach a person, by contact reason, and drops deflection, for the reason given under the last cause.

A planetarium finds why its visitors kept asking for a person (illustrative)

Illustrative example — not a real result

Celqurvin Planetarium

Celqurvin Planetarium, an invented planetarium and science centre, runs an assistant on its booking pages covering 6 contact reasons: show times, tickets and seating, getting there and parking, accessibility, school and group visits, and membership. Its monthly report shows deflection at 81% and climbing, while calls to the box office about bookings have risen by about a fifth over the last 3 months.

The visitor services lead pulls 60 conversations from one month: the 45 that ended with a request for a person, a handover or silence from the customer, or were followed by a call to the box office about the same booking, and 15 at random. Of the 45, 12 are school and group bookings, which need a quote and a check of dome capacity on the day, so the reason should never have been automated. 8 ask whether children under 5 can attend the dome shows, a question no article answers. 7 ask about wheelchair spaces in the dome, which the access page covers only as a row in a table headed "Seating options". 6 are routine ticket questions handed over because the hand-over list said "anything about tickets". 5 came outside box office hours and ended with "please wait" and nothing else. 3 got a wrong answer about parking charges, which go to the wrong-answers diagnosis, and 4 were complaints rightly handed over, 2 of them to a person who asked the visitor to explain everything again. The 15 random conversations show no further cause.

The fixes follow the groups. School and group visits move to a person. The education manager writes an article on ages for each show. The access page gains a section headed "Wheelchair spaces in the dome", in sentences. The hand-over list becomes "refunds, exchanges after a cancelled show, complaints, safety, signs a visitor may be in difficulty". Out of hours, the assistant now says when the box office opens and takes a message answered by 10am. The handover passes the full conversation to the box office screen. The report swaps deflection for repeat contacts within 7 days and time to reach a person, by reason, and every Monday the lead reads each conversation from the week that ended with a request for a person, silence, a complaint or a low rating, plus a random handful. A second pull from the first full month after the fixes, drawn like the original, finds 9 such conversations, against 45 before; 6 of them are complaints, refunds or group visits that the new scope and the hand-over list pass straight to staff, as designed, and the Monday read traces the other 3 to their causes.

The AI in marketing troubleshooting section lists the neighbouring symptoms, from invented facts in drafts to automated workflows that sent the wrong message.

Common mistakes

  • Reading the dashboard instead of the conversations. A falling handover count says nothing about whether customers were helped; the transcripts do.
  • Counting every handover as a failure. A complaint, a refund or a customer in difficulty passed to a person at once is the scope doing its job; sort the handovers by cause before counting them.
  • Widening the scope before the articles exist. A reason the library cannot answer gives the assistant nothing to answer from, and the customer can end up asking for a person or phoning.
  • Treating every refusal as a model fault. Trace it first: a missing article, an unfound passage and an over-tight rule each have a different fix.
  • Writing routine subjects into the hand-over list. Confine it to the topics and signals that need a person, keep the cautious rules for complaints, refunds, safety and customers who may be vulnerable, and let the assistant answer the rest.
  • Reading only the handovers. Customers who gave up without asking for anyone can carry the same causes; sample the abandoned conversations too.
  • Fixing the route and keeping the report. While the report leads with deflection, the next run of failures can stay hidden the same way.

Frequently asked questions

Should we make it harder to reach a person so the assistant gets a fair chance?

No. The customer service guide counts hiding the route to a person among its common mistakes. This page also counts the conversations people abandon on the way among those the team needs to read. Keep the route to a person visible from the start, carry the conversation across when someone takes over, and judge the assistant on whether customers' problems ended, not on how many conversations it kept to itself.

How do we tell a scope problem from a content problem?

Ask what a good answer would need. If it needs judgement, discretion, an exception or a decision about money, the contact reason belongs with a person whatever the help centre says, and the fix is to take it out of the assistant's scope. If it needs only a fact or a procedure the business has already decided, and a wrong answer would do little harm, the assistant should be able to give it, and the fix is in the documents or in how they are found.

Can an assistant hand over too much?

Yes. The tutorial on standing instructions accepts that a cautious hand-over rule will pass on some conversations that did not need a person, and that is the right trade for complaints, refunds, safety and customers who may be vulnerable. It is the wrong trade when a hand-over list written as whole subjects sends routine questions to the queue, so read the handovers by contact reason and narrow any rule that catches questions the documents answer in full.

Terms used on this page

Frameworks

Sources

  1. AI Risk Management Framework — NIST

Related