Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Marketing Analytics › Business Intelligence for Marketing

Reading Small Numbers in Marketing Reports

How to read marketing metrics built on small counts: why rates jump around, ranges instead of single figures, minimum volumes, pooling and control limits.
Guideintermediate11 min readUpdated

Why small counts mislead

Every conversion rate in a report is a sample. The visitors who arrived last week are some of the people who might have arrived, and whether each one bought depended on a long list of things nobody measured. Run the same week again with the same campaign, the same site and the same prices, and a different number of people would convert. With large counts the difference between the two runs is small relative to the total. With small counts it is large, because a few conversions more or less is a big share of a small number.

The effect is arithmetic, not a fault in the tracking, and it has three practical consequences:

  • Rates from small counts swing widely from period to period with nothing changed. A week-on-week fall in a campaign's conversion rate (CVR) or a rise in its cost per acquisition (CPA) can be entirely chance.
  • The most extreme figures in any list come from the smallest units. The best and worst landing pages by conversion rate are usually the ones with the fewest visits.
  • Looking often multiplies false alarms. Each time a figure is read and compared with a line, chance gets another opportunity to cross it. Evan Miller's essay on how not to run an A/B test shows how repeated looks at a test inflate the rate of false winners; a weekly report read for anything unusual, with action taken on the first figure that looks different, repeats the same pattern without the test.

None of this means small campaigns cannot be judged. It means they have to be judged with more data, over a longer period or alongside similar campaigns, and that their figures should be shown with an honest statement of how uncertain they are.

Counts behind every rate

The first habit is the simplest: never read a rate without the count beneath it. A conversion rate is conversions divided by sessions or clicks; a cost per lead is spend divided by leads; a return on spend is revenue divided by spend, and revenue itself is a few orders added up. The number that controls how much the rate can move by chance is usually the smaller count, the conversions, leads or orders, not the larger one.

Put the counts next to the rates in every table and dashboard. Two campaigns with the same rate can be in very different positions: one has hundreds of conversions behind its figure and the other has a few. A report that shows only the rates invites the reader to treat them as equally solid.

Averages need the same care. An average order value built on a few orders can be moved by one large basket, and a revenue figure for a small segment can double because of one customer. For averages, look at the number of orders and at the largest values; a single order that dominates the total is worth naming in the commentary.

Ranges, not single figures

A figure from a sample has a range of plausible true values around it. For a rate, the confidence interval is the standard way of stating that range: given what was observed, the true rate probably lies between a lower and an upper value. Its width depends mostly on the count of conversions. Because the width shrinks with the square root of the count, four times as many conversions are needed to halve the range, which is why small campaigns stay uncertain for a long time.

There are standard methods for computing the range of a rate; the Wilson interval is a common choice because it behaves sensibly when counts are small or the rate is near zero, and spreadsheet formulas for it are widely available. The conversion rate calculator handles the rate arithmetic itself. For a count of events, such as leads in a week, a range based on the count's own variation serves the same purpose.

Reading two ranges together gives a quick judgement:

What the ranges showWhat it meansWhat to do
They overlap widelyThe data cannot tell the two figures apartTreat them as the same until more data arrives
They are clearly separateThe difference is probably more than chanceInvestigate the cause; the change is worth explaining
They overlap a littleUnclear; overlapping ranges are a rough guide, not a testPool more data, or run a formal comparison

A formal comparison of two rates is a Statistical Significance question, and the sample size and significance guide explains how it is done and how a planned test avoids the problems of comparing whatever happened to occur. In routine reporting the range is used more modestly: to stop a single figure being read as more precise than it is. The same reading of overlapping ranges decides whether customers who give low survey scores really leave more often, which the tutorial on testing whether survey scores predict churn applies to churn rates by score band.

Illustrative example — not a real result

Tallowmint Candles nearly pauses its best ad set

Tallowmint Candles is an invented candle maker that sells through its own site. Its best-converting paid social ad set has a strong week followed by a weak one, and the media buyer pauses it on Monday because cost per order has more than doubled. Before the change goes live, the analyst puts ranges around the figures:

PeriodClicksOrdersConversion rateRange (95% Wilson interval)Cost per order
Strong week400143.5%2.1% to 5.8%£41
Weak week38061.6%0.7% to 3.4%£92
Previous eight weeks pooled3,100782.5%2.0% to 3.1%£45

The two weekly ranges overlap between 2.1% and 3.4%, and both contain the pooled rate. At the pooled rate, 380 clicks would be expected to bring about 9 or 10 orders, and for counts that small a week with 6 or with 14 is ordinary. The ad set had become the account's best on the strength of the strong week, so some fall was to be expected whatever happened next. The pause is cancelled. Looking back over a year of its own ad sets, the team finds that weekly rates from ad sets with fewer than 20 orders a week regularly moved by half in either direction with nothing changed. It agrees to judge ad sets on rolling four-week figures, to show the order count and range beside every rate in the weekly report, and to act on a single week only when it falls outside the range of the pooled figure. Over the next six weeks the ad set converts at 2.4%.

Pooling and minimum volumes

When a figure is too uncertain to act on, the answer is more data. A planned test fixes its sample size in advance; a routine report has to work with whatever arrived, and gathers more in one of two ways.

Longer windows. Read the figure over several weeks instead of one, or over a rolling window that moves forward each week. A rolling window keeps the report current while giving each figure a larger count behind it. The cost is responsiveness: a real change takes longer to show, which matters for fast-moving campaigns and much less for steady ones.

Grouping similar units. Combine campaigns, ad sets, keywords or pages that share an audience, a theme or a template, and judge the group. Individual units can then be compared with their group, and a unit whose figure sits far from the group over a long period is worth a look. A refinement used in more mature teams pulls each small unit's figure part of the way towards its group's figure, more for units with little data and less for units with plenty; the result ranks units more sensibly than raw rates do.

Minimum volumes. Agree a minimum count below which a figure is not judged alone, and set it from the team's own history rather than from a figure found elsewhere. A practical way to do it: take periods in which campaigns ran unchanged, look at how far their weekly figures moved at different volumes, and choose the volume at which that ordinary movement becomes smaller than the differences the team acts on. The minimum depends on the decision; pausing an ad set needs less evidence than reallocating a quarter's budget. The guide to using industry benchmarks makes the related point that a figure with a wide normal range cannot be judged against an outside number on one period's value either. When the small counts come from a website change rather than a campaign, the guide to conversion work without an A/B test applies the same habits to a before-and-after reading, with the reading date and the comparison period fixed before the change ships.

League tables and regression to the mean

Ranking units by a rate is the most common way small numbers mislead. Sort landing pages, campaigns, regions or sales reps by conversion rate, and the top and bottom of the list fill with the smallest units, because they have the widest spread. The page with the highest rate has a handful of visits; so has the page with the lowest. Acting on the extremes rewards and punishes luck.

Two fixes help. Sort or filter by volume first, so that the ranking covers units with comparable counts. Or plot each unit's rate against its volume with limits that narrow as volume grows, sometimes called a funnel plot; units outside the limits are unusual for their size, and units inside them are not, however high or low they rank.

Regression to the mean is the second trap. A unit picked because its figure was extreme in one period will usually be closer to average in the next, even if nothing is done, because part of what made it extreme was chance and chance does not repeat on demand. Pause the worst campaigns and the account's average improves partly because they were unlucky; reward the best performers and they seem to slip. Before crediting an action with an improvement in units chosen for being extreme, compare them with similar extreme units that were left alone, or check whether the improvement exceeds the movement those units show without any action. The method for explaining a change includes testing an explanation against data it did not come from, which is the general form of the same check.

Recurring metrics and control limits

For a metric read every week, such as leads, orders, cost per lead or a satisfaction score, the question is whether this week's figure is unusual compared with the metric's own normal spread. The control charts framework answers it: a centre line from a stable baseline, limits set from the metric's routine week-to-week movement, and rules agreed in advance for what counts as a signal. Two points from that framework matter most for small numbers. First, a rate whose denominator varies from week to week needs limits that widen in weeks with less traffic, or it will flag the quiet weeks as unusual. Second, when no rule fires, the discipline is to leave the metric alone, because reacting to routine variation adds variation rather than removing it.

Control limits also settle the weekly argument about whether a dip is real. If the team has agreed the limits and rules in advance, the dip either crosses them or it does not, and the meeting can move on.

Showing uncertainty in reports

Most readers of a report will not compute a range, so the report has to do it for them in a form they can read at a glance:

  • Counts beside rates in every table, so that a rate from a small count is visibly different.
  • Flags for low volume. Grey out, italicise or mark rates below the team's minimum volume, with a note explaining the rule.
  • Ranges for the figures that drive decisions, as an error bar on a chart or as "between" wording in the commentary.
  • Pooled views beside weekly ones, such as a rolling four weeks next to the latest week, so that the weekly figure is read in context.
  • Words that match the evidence. "Within its normal range" and "too few orders to tell" are findings, and they belong in the commentary as much as any change does.
  • No percentage changes on tiny bases. A move from two leads to five is better reported as counts.

The performance reporting guide sets the cadence at which these figures are read, with daily checks for breakages and decisions at the weekly and monthly reviews where counts are larger. The BI hub covers where flags and pooled views can be built once, in the reporting layer, rather than added by hand to every chart.

Common mistakes

  • Reading rates without counts. Two equal rates can rest on very different amounts of evidence.
  • Pausing or scaling on one week. A single week of a small campaign is mostly chance; use a rolling window.
  • Ranking units by raw rate. The extremes of the list are the smallest units, not the best and worst.
  • Crediting actions with regression to the mean. Units chosen for being extreme drift back without help.
  • Borrowing a minimum volume from somewhere else. Set it from your own history and the size of the decision.
  • Hiding small figures entirely. Mark them and offer a pooled view instead.
  • Treating overlapping ranges as proof of no difference. They show the data cannot yet tell; more data may.
  • Using test thresholds on routine reports. Reports read every week are a stream of comparisons, not one planned test; control limits suit them better.
  • Reporting percentage changes on tiny bases. Report the counts.

Frequently asked questions

How many conversions do I need before I can trust a conversion rate?

There is no universal number, and any figure quoted without your context is a guess. The useful question is how wide the range around your rate is at the count you have, and whether a range that wide would change the decision. Work it out from your own history: look at how much a campaign's weekly rate moves in periods when nothing was changed, and set the minimum volume where that movement becomes smaller than the differences you act on. Larger decisions deserve larger counts.

Is a confidence interval only for A/B tests?

No. Any rate or average computed from a sample of people has a range of plausible true values around it, whether the figure comes from a test, a campaign report or a landing page dashboard. In routine reporting the range is mostly used to read a single figure honestly and to see whether two figures can be told apart. Planning a test, choosing its sample size and deciding when to stop it are separate jobs, covered by the guide to sample size and significance.

Should we hide figures that are based on too few conversions?

Show them, but mark them. Hiding a small campaign's figures makes people go looking for them elsewhere, often in a platform interface without any context. A better pattern is to show the count beside every rate, grey out or flag rates below the team's minimum volume, and offer a pooled view over a longer period or a group of similar campaigns. Readers learn quickly that a flagged figure is not one to act on alone.

Terms used on this page

Tools to try

Frameworks

Where this page is used

Sources

  1. How Not To Run an A/B Test — Evan Miller

Related