Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Customer Experience, Retention & Loyalty › CX Measurement

Testing Whether Survey Scores Predict Churn

How to check in your own data whether satisfaction or effort scores warn of churn: join answers to later behaviour, compare groups, read small numbers well.
Tutorialadvanced11 min readUpdated

Why a score earns its place only if it warns you

The churn and retention metrics guide puts perception indicators, such as low transactional scores and negative comments, beside behavioural and service indicators as warning signs of churn, with one condition: they are useful when they have been shown to predict behaviour in the business's own data. The guide's advice for finding any indicator is the same: compare what customers who left did beforehand with what similar customers who stayed did, and drop indicators that do not predict churn. Looking at the customers who stayed without the ones who left is survivorship bias, which hides the evidence the departed carry, so both groups belong in every comparison.

The measurement hub states the cost of skipping this. If scores rise and churn does not fall, the score is measuring something the business does not need. The voice of the customer guide says what each survey measure asks and captures and ends its reading with the same instruction: join survey responses to what the same customers did afterwards and check whether low scorers leave more often. A relationship score that has already fallen for several waves raises the same question from the other end, and the diagnosis for a relationship survey score that is falling asks it last, after noise, sample and timing have been ruled out.

None of the existing guides runs that check, and it is easy to get wrong. A score read against churn in the same month says nothing about prediction. A comparison across all customers mixes new customers, who both score and leave differently, with long-standing ones. Customers who did not answer the survey may be the ones most likely to leave. And a gap seen once, in a small sample, may be chance. This tutorial runs the check carefully enough to base a decision on. It is an analysis in your own data, not a claim about any score in general; the site cites no research on whether any survey measure predicts churn, and quotes none. The academy checkpoint on growth models, growth accounting and leading indicators tests joining survey answers to later behaviour, reading small numbers and deciding whether a score earns its place.

Before you start

  • Read the two prerequisite guides. The voice of the customer guide explains what your score measures, when it is asked and how sampling and response bias shape it. The churn guide defines churn, voluntary and involuntary churn, and cohorts. This tutorial uses both without repeating them.
  • Have survey answers that can be linked to customers. You need each answer with a customer identifier and a date. The step on linking below sets the conditions.
  • Have a reliable churn or lapse definition. For a subscription, a cancellation date; for a business without subscriptions, an inactivity rule set from your own purchase gaps, as the churn guide describes. Write it down.
  • Be able to build a cohort table. The cohort analysis tutorial groups customers by acquisition period and follows them. You need the same skills here, applied to customers grouped by score as well as by cohort.
  • Know how to read small counts. The reading small numbers guide shows counts behind every rate and ranges around every figure. The comparison in this tutorial stays at that level: counts, groups and intervals, no modelling.
  • Agree who decides. Name the person who owns the retention dashboard and will decide what happens to the score once the result is in.

The steps

  1. Write the question and the decision it serves

    State exactly what is being tested: "Do customers who give a low effort score after a support contact cancel more often over the following months than customers who give a high one?" Name the score, the moment it is asked and the kind of churn. Then write the decision the answer feeds: whether the score becomes a Leading Indicator on the retention dashboard, a trigger for a follow-up call, or neither. A test with no decision attached tends to be run, admired and forgotten.

  2. Check that answers can be linked to customers

    Linking survey answers to accounts turns them into personal data about identifiable customers. The voice of the customer guide applies the GDPR, the EU framework for processing personal data, and ESOMAR's professional codes on honesty, consent and care with respondents' data: customers should be told how feedback linked to their account will be used, and anonymous surveys must stay anonymous. Check what each survey's invitation told respondents. Use only answers from surveys where linking was disclosed, and keep the joined table restricted to the people doing the analysis, with names and contact details removed. The survey and questionnaire design guide sets out the choice between anonymous, confidential and attributable surveys for the future. The guide to anonymous, confidential or attributable surveys works that choice through for the next survey, including how its invitation says that answers will be linked. This is general direction, not legal advice.

  3. Fix the outcome and the window before looking

    Decide, in writing, what counts as churn and over what period after the answer it is measured: cancelled within a stated number of months of the survey, or lapsed under the inactivity rule by a stated date. Choose a window long enough for leaving to happen in your business and short enough that the answer is still relevant. Separate voluntary from involuntary churn, as the churn guide advises; a failed card has nothing to do with a survey answer. Fixing these before you see the data stops the window being chosen because it makes the score look good.

  4. Build the table of answers and later behaviour

    Make one row per answer: customer identifier, survey date, score, the customer's acquisition date and cohort, their plan or segment, and the outcome at the end of the window. Exclude answers whose window has not yet closed. Where a customer answered several times, keep one answer per customer, such as the first in the period, so frequent respondents do not count twice. Record how many answers were dropped at each stage and why.

  5. Group scores into a few bands

    Group the scale into a small number of bands that match how the score would be used, such as low, middle and high, with the cut points written down before the comparison. Fine-grained bands spread the answers thin and make each band's churn rate jump around. If the score would trigger action only at the bottom of the scale, make sure the bottom band is defined the way the trigger would be.

  6. Compare churn across bands within comparable cohorts

    Calculate the churn rate for each band, with the count of customers and the count who left beside every rate. Then repeat the comparison within groups of comparable customers, such as each acquisition Cohort, plan or tenure band. New customers often both score differently and leave more often for reasons the survey did not touch, so a gap across all customers can come from mix rather than from the score. A gap that holds inside each cohort is more convincing than one that appears only in the total.

  7. Account for the customers who did not answer

    Add a row for customers who were invited and did not respond, with their churn rate over the same window. If non-responders leave more often than any band, the survey is missing the customers most at risk, and a score that looks predictive among respondents may cover few of the customers who matter. Compare respondents with the invited customers on tenure, value and segment, as the voice of the customer guide advises. The tutorial on drawing a survey sample from a customer list shows how to draw future samples so that groups are represented.

  8. Read the differences with intervals

    Put a range around each band's churn rate, using a Confidence Interval as the reading small numbers guide describes, and read the ranges together. Ranges that are clearly separate suggest a real difference; ranges that overlap widely mean the data cannot tell the bands apart yet. A band with few answers will have a wide range whatever the score does. Note the sample size behind each band, and resist reading a gap that only appears in a band too small to support it.

  9. Repeat on a later period

    A gap seen once can be chance or a feature of one period, such as a price change or a service outage that moved both scores and churn. Repeat the whole check, with the same definitions, bands and window, on answers from a later period. A score that warns of churn in both periods, and within cohorts in both, has earned more trust than one that did so once.

  10. Decide what the score is for

    Use the decision table below to settle where the score belongs. A score that passes can sit on the retention KPI tree as an input under the driver it reflects; the KPI tree for retention shows the voluntary churn branch that service, product and messaging inputs feed. A score that fails as a predictor may still identify customers to follow up. Record the result, the date and the definitions, and set a date to repeat the check, because the relationship can change when the product, the customers or the survey change.

Reading the result: a decision table

What the comparison showsWhat it meansWhat to do with the score
Low band churns clearly more than high, within cohorts, in both periodsThe score warns of churn in this businessPlace it on the retention KPI tree as a leading input, and use the low band as a follow-up trigger
Only the lowest band differs; middle and high bands overlapThe score warns about a small group, not across the scaleUse the lowest band as a trigger for follow-up; do not report the average as a retention measure
Gap appears in the total but not within cohortsMix, not the score, drives the gapDo not use it as a churn signal; read churn by cohort instead
Gap in one period, not the nextChance or a one-off eventTreat it as unproven; repeat on another period before deciding
Ranges overlap widely everywhereToo few answers to tellCollect for longer or pool periods; make no claim yet
Non-responders churn most of allThe survey misses the customers at riskFix sampling and invitations first; add behavioural indicators beside the score
No difference with enough dataThe score does not predict churn hereRemove it from the retention dashboard; keep it, if at all, as a diagnostic for a process

Two neighbours help with what follows. Operational measures may predict churn where perception does not, and the guide to customer service metrics defines repeat contacts and contact rate so they can be tested the same way. And a score can say that customers are leaving without saying why; the jobs-to-be-done framework for retention adds switch interviews with customers who left.

A refill subscription tests its effort score (illustrative)

Illustrative example — not a real result

Plazzuvo Refills checks whether its effort score warns of cancellations

Plazzuvo Refills, an invented cleaning-product refill subscription, asks a one-question effort survey after support contacts, and its invitation says answers are linked to the customer's account to improve service. The team links a year of answers, 2,140 customers after keeping one answer each, to whether each customer had voluntarily cancelled within 4 months of answering. Customers whose 4-month window had not closed are excluded.

Answers on the 1 to 7 scale are grouped into three bands written down in advance: 1 to 3 (low), 4 to 5 (middle) and 6 to 7 (high). Across all customers, 31% of the low band cancelled within 4 months (95% interval 26% to 36%), against 17% of the middle band (14% to 21%) and 14% of the high band (12% to 16%). Within signup cohorts the low band stays clearly higher in each cohort with enough answers, while the middle and high bands overlap in every cohort. Customers who were invited and did not answer cancelled at 19%.

The team repeats the check on the next half-year of answers, 980 customers. The low band again cancels at a clearly higher rate, 29% (22% to 37%), and the middle and high bands again overlap. The retention owner decides that a low effort answer becomes a trigger for a call from a senior agent within 2 working days, and that the average effort score stays off the board's retention dashboard, where it had been reported as a headline figure. The check is booked to repeat in 6 months.

Common mistakes

  • Reading the score against churn in the same month. Prediction needs the answer first and the behaviour afterwards, over a window fixed in advance.
  • Choosing the window after seeing the data. A window picked because it gives the clearest gap is a result manufactured, not found.
  • Mixing voluntary and involuntary churn. Failed payments have nothing to do with how a customer answered a survey.
  • Comparing across all customers only. New and long-standing customers differ in both scores and churn, so the gap may be mix. Compare within cohorts.
  • Ignoring non-responders. The customers who never answer may be the ones who leave.
  • Too many bands. Fine bands spread answers thin and every rate jumps around.
  • Single figures without ranges. A churn rate for a small band is a wide range, not a precise number.
  • Testing once. A gap seen in one period may be chance or a one-off event. Repeat it.
  • Linking answers that were promised anonymity. An anonymous survey stays anonymous, whatever the analysis would gain.
  • Generalising the result. A score that predicts churn in one business, for one survey moment, says nothing about another business or another moment.

Frequently asked questions

Does NPS predict churn?

This site makes no general claim either way, and the only answer that matters for a business is the one in its own data. Link each customer's survey answer to whether they were still a customer some time later, compare churn between score groups within comparable customers, and read the differences with ranges rather than single figures. If low scorers leave clearly more often, the score is a useful warning; if they do not, it is measuring something the business does not need for retention.

Can we link anonymous survey answers to churn?

No. An anonymous survey must stay anonymous, so its answers cannot be joined to accounts. The check works only with surveys where respondents were told their answers would be linked to their customer record and how they would be used, as the research and voice of the customer guides set out. For future surveys, decide in advance whether answers will be linked, and say so in the invitation. Treat this as general direction and agree the specifics with whoever handles data protection.

What if the score does not predict churn?

It may still be useful for something else, such as finding individual customers to follow up after a poor experience, or showing which part of a process frustrates people. What it should not be is a headline retention measure. Check first that the result is not an artefact of a short window, a small number of responses or a biased set of respondents; if it holds when repeated, move the score off the retention tree and keep it, if at all, as a diagnostic.

Terms used on this page

Frameworks

Where this page is used

Part of these learning paths

Also related

Sources

  1. Regulation (EU) 2016/679 (GDPR) — EUR-Lex
  2. ESOMAR — ESOMAR

Related