Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Customer Experience, Retention & Loyalty › Service and Support

Reviewing Support Conversations for Quality

How to review a sample of support replies and calls each week: a short scorecard from your standards, fair sampling, calibration, coaching and fixing causes.
Tutorialintermediate9 min readUpdated

What a quality review finds that metrics miss

The support channels and self-service guide sets three standards for every channel: response, resolution and tone. It warns against measuring response rather than resolution, because a fast reply that does not solve the problem only starts the clock again. The operational measures that track response and resolution belong to the guide to customer service metrics and how to read them. This tutorial covers what those numbers cannot see.

A reviewer reading a conversation sees things no dashboard records. The answer was confident and wrong. The agent asked for an order number the customer had already given. The reply was polite and answered a different question. The policy was explained correctly, but the policy itself makes no sense to customers. The tone was right for a simple question and wrong for a bereavement. Each of these can sit inside a conversation that met its response time and was marked resolved.

The review also finds causes. A run of wrong answers on one topic is usually not several agents having a bad week; it is an outdated saved reply, a help article that contradicts the website, or a policy nobody explained to the team. The point of reviewing is to find those causes and send them to whoever can fix them, as well as to help individual agents improve. Teams using AI to draft replies or answer customers run a similar weekly sampling review of automated conversations, which the AI in customer service guide sets out; this tutorial covers conversations handled by people.

Before you start

  • Have written standards. The scorecard is built from the response, resolution and tone standards the support channels guide describes. Without them, reviewers judge against their own preferences.
  • Have tone guidelines. The tutorial on writing customer service tone guidelines records the team's tone decisions with examples. Reviewers judge tone against that document, not against what they would have written.
  • Agree who reviews and when. Name the reviewers and give them protected time each week. A review squeezed between tickets is rushed and unfair.
  • Tell the team first. Explain what is reviewed, how it is used and what it is not used for, before the first review. Agents who discover a review by surprise do not trust it.
  • Check what you may review. Conversations contain customers' details, and calls are only available if the business records them under its own policy. The rules on recording and monitoring are outside this page; whoever owns data protection and employment matters in the business decides what is reviewed and how.

The steps

  1. Write the scorecard from your standards

    Turn each standard into a question a reviewer can answer from the conversation. "Resolution" becomes "Did the reply solve the customer's problem, or say clearly what happens next and when?" "Tone" becomes "Does the reply follow the tone guidelines for this situation?" Add the questions your standards imply but do not state: was the information correct and current, did the agent use what the customer had already said, was anything promised that the business cannot deliver. Each line on the scorecard should trace back to a written standard, so agents can see why it is there.

  2. Keep it short and answerable

    A scorecard with a long list of lines is filled in mechanically, and the lines that matter disappear among the ones that do not. Keep a handful of lines, each answerable as met, partly met or not met, with a short note of why. Avoid lines that reward a phrase rather than an outcome, such as "used the customer's name twice": they are easy to score and teach agents to sound compliant rather than helpful. Tone is judged against the written guidelines. Nielsen Norman Group's writing on tone, as the service and support hub describes it, treats tone as a set of choices along dimensions such as formal or casual and serious or funny, made once for the team; the reviewer checks whether the reply made the team's choice, not whether it matches the reviewer's own taste.

  3. Decide how to sample

    Choose conversations at random within groups, so every agent, every channel and the common contact reasons appear over a few weeks. A sample chosen by hand drifts towards the conversations reviewers find interesting or the agents they already worry about. Include short conversations as well as long ones. The review is for finding problems, not for estimating a precise rate, so there is no Sample Size to calculate as there would be for an experiment; pick a number reviewers can read properly every week and keep it steady. Where an outside team answers some of the business's customers, draw its conversations into the same sample and score them on the same card, as the guide to working with an outsourced customer service team recommends, so the review stays with the business rather than with the partner.

  4. Add every complaint, escalation and low rating

    On top of the random sample, review every conversation that ended in a complaint, an escalation, a low satisfaction rating or a repeat contact on the same subject. These are where things went wrong, and reading them shows whether the cause was the reply, the policy or something earlier in the journey. Keep them separate from the random sample when reporting, because a set chosen for being difficult will always score worse and should not be read as typical.

  5. Review the whole conversation, not the last reply

    Read the conversation from the customer's first message, including any handover from a bot or another agent and any earlier contact on the same subject. A good final reply can follow three replies that ignored the question. For calls, use the recording or the notes the business keeps. Check the facts against the help article, the policy and the order record, not against memory. Note the moment the conversation went right or wrong, because that is what feedback should point to.

  6. Calibrate reviewers on the same conversations

    Regularly, all reviewers score the same few conversations separately, then compare. Where they disagree, they talk it through and write down the shared reading, adding an example to the scorecard notes. Disagreements usually come from a line that can be read two ways, such as what counts as resolved. Without calibration, an agent's score depends on who reviewed them, and agents notice quickly.

  7. Give feedback to the agent with the example

    Share each review with the agent, with the conversation in front of them, pointing to the specific moment and what would have worked better. Start with what went well. Ask what made the reply hard to write, because the answer often reveals a missing saved reply, an unclear policy or a tool problem. Coaching is general practice here: short, frequent, specific and based on real conversations. Keep a note of what was agreed so the next review can look for it.

  8. Send patterns to the owner of the cause

    When the same failure appears across several agents, the cause is rarely the agents. Trace it: an outdated reply in the saved reply library goes to the library owner; a wrong help article to its owner; a policy customers cannot follow to the policy owner; a failure the customer complained about to the people running the service recovery and complaints loop. Record what was sent, to whom and what they agreed to do.

  9. Report what changed, not only scores

    A monthly summary should say what the review found, what was fixed and what is still open, with examples in the customers' words. Scores can sit alongside, but read any rate built on a small number of conversations with the care the guide to reading small numbers sets out: small counts jump around by chance, so show the count behind each figure and avoid ranking agents on them. Pair any scorecard figure with a Guardrail Metric such as repeat contacts, so an improving score that comes with more customers writing back is caught.

  10. Revise the scorecard each quarter

    Once a quarter, look at which lines are always met and teach nothing, which are often disputed in calibration, and which problems the review found that no line captures. Change the scorecard and tell agents what changed and why. Goodhart's law is the reason to keep doing this: once a measure becomes a target, people find ways to move it that do not move the outcome, and a scorecard left unchanged for long enough is satisfied rather than met. Treat the review as a KPI source only with that risk in mind.

A scorecard layout: what each line asks

LineThe question the reviewer answersComes fromWhat a "not met" usually points to
UnderstoodDid the reply answer what the customer actually asked, using what they had already said?Resolution standardAgent reading too fast; a saved reply used for the wrong question
CorrectWas the information accurate and current against the help article, policy and records?Resolution standardOutdated saved reply or article; a policy change support was not told about
Resolved or next stepWas the problem solved, or did the customer know exactly what happens next and when?Resolution standardMissing authority to act; an unclear process
ToneDid the reply follow the tone guidelines for this situation?Tone guidelinesA situation the guidelines do not cover; time pressure
PromisesDid the reply promise only what the business can deliver?Standards and remedy rulesAgents unsure of their limits
EffortDid the customer have to repeat themselves, chase or switch channel?Response and resolution standardsHandover without history; routing

The table is a layout, not a standard to copy. Write the lines from your own standards, and keep the column on what a "not met" points to, because it reminds reviewers that most failures have a cause beyond the reply. Where customers are answered in other languages, the guide to supporting customers in several languages and time zones covers who reviews translated replies; the same scorecard applies, read by a fluent reviewer. Every conversation reviewed is a test of the brand voice at the moment it matters most.

A craft kit shop starts reviewing its replies (illustrative)

Illustrative example — not a real result

Quabrin Craft Kits starts a weekly quality review

Quabrin Craft Kits, an invented online craft kit retailer, has 5 agents answering email and chat, and sells kits both singly and by monthly subscription. The support lead writes a 6-line scorecard from the team's standards and tone guidelines, the same lines as the layout above. Each week she and a senior agent review 4 randomly chosen conversations per agent, plus every complaint, escalation and rating of 1 or 2 out of 5: 31 conversations in the first week.

In the first calibration session, both reviewers score the same 8 conversations and disagree on "Resolved or next step" in 5 of them. One counted a conversation as resolved when the agent had explained the policy; the other only when the customer's problem had gone away. They agree on the second reading and add 2 examples to the scorecard notes.

Over the first month, 14 of the 118 reviewed conversations fail the "Correct" line, and 11 of those concern pausing or changing a subscription. They come from 4 different agents. Every one used the same saved reply, written before the shop let subscribers skip a month. The library owner rewrites it that day and links it to the subscription help article. In the following month, 3 conversations fail the "Correct" line, none about subscriptions, and the team reports the fixed reply rather than any agent's score.

Common mistakes

  • A scorecard nobody can trace to a standard. Agents see it as the reviewer's opinion, and it is.
  • Too many lines. Reviewers tick boxes and the important failures disappear.
  • Reviewing only the conversations that went wrong. Agents see the review as punishment, and the scores mean nothing about typical service.
  • Judging tone by taste. Without the written guidelines, every reviewer marks a different tone as right.
  • No calibration. An agent's score depends on who reviewed them.
  • Blaming agents for causes they do not own. Repeated failures across agents are a reply, article, policy or tool problem.
  • Ranking agents on small samples. A few conversations a week cannot separate a good agent from a lucky one.
  • A scorecard that never changes. Once agents learn it, it measures how well they satisfy the scorecard.

Frequently asked questions

How many support conversations should be reviewed each week?

Enough to cover every agent, channel and common contact reason over a few weeks, plus every complaint, escalation and low rating, and no more than the reviewers can read properly. The review is for finding problems and their causes, not for estimating a precise score, so a small sample read carefully is more useful than a large one skimmed. Read any score built on a handful of conversations as a rough signal, not a ranking.

Who should review support conversations?

Usually a support lead or experienced agents, with the scorecard owner joining calibration sessions. Agents reviewing each other's conversations, with the same scorecard and regular calibration, often learn more from the review than the people being reviewed. Whoever reviews needs time set aside for it; a review squeezed between tickets is rushed and unfair.

Should quality review scores be tied to agents' pay or targets?

This tutorial treats the review as a way to improve replies and remove causes, and that works best when agents trust it. Once a score becomes a target, agents learn to satisfy the scorecard rather than the customer. How reviews relate to performance management is a separate decision with its own employment considerations, which this page does not cover.

Terms used on this page

Frameworks

Where this page is used

Also related

Sources

  1. Nielsen Norman Group articles — Nielsen Norman Group

Related