What a control chart is
Every marketing metric moves from week to week for reasons nobody controls: the weather, the day a holiday falls on, a few large orders, the random spread of who happened to visit. Most reporting treats each movement as news, and teams spend meetings explaining changes that would have reversed themselves. A control chart separates two kinds of variation. Routine variation is the metric's ordinary spread, produced by the system as it stands. A signal is a change larger or more persistent than that spread can explain, which usually means something in the system changed: a price, a page, a channel, a competitor or the tracking.
The chart is built from the metric's own history rather than from a target or an external figure. That is the difference from a benchmark, which compares the business with others; the guide to using industry benchmarks warns that a metric with a wide normal range cannot be judged on one month's value against anything, and a control chart is how that normal range is measured. It is also different from a test. A statistical significanceA test result unlikely enough under 'no real difference' to pass a threshold chosen in advance. check compares two groups over the same period; a control chart compares one process with its own past. The comparison of control charts and A/B testing sets out what each can prove and when a change should be split rather than charted.
The parts
| Part | What it is | What it tells you |
|---|---|---|
| Centre line | The typical level of the metric over a baseline period, usually its average or median | Where the process sits when nothing unusual is happening |
| Upper and lower limits | Lines either side of the centre, set from the size of the metric's routine week-to-week movement | The range the metric can be expected to fall within if nothing changes; a point outside is unlikely by chance |
| Run rules | Patterns inside the limits that are also unlikely by chance, such as a long run on one side of the centre line or a steady climb | A smaller, lasting shift that no single point would reveal |
| Baseline period | The stretch of history the lines are calculated from | What "normal" means; a disrupted baseline gives misleading limits |
| Annotations | Notes on the chart marking known changes: launches, price moves, tracking fixes | Whether a signal lines up with something the team already knows about |
The limits are not targets and not a confidence intervalA range of values for an effect, such as a lift in conversion rate, that is consistent with the data at a chosen confidence level. around a single estimate. They describe the spread of the process itself. The method comes from statistical process control, which Walter Shewhart developed at Bell Laboratories in the 1920s to separate routine from unusual variation in manufacturing, and the version for one value per period is called an individuals chart (or XmR chart). It places the limits about three standard deviations of routine variation either side of the centre line, estimated from the average change between consecutive points rather than from the overall spread, so that a shift in level during the baseline does not widen the limits and hide itself. In symbols, with the change between consecutive points called the moving range:
Moving range = |this period's value − the previous period's value|
Limits = Centre line ± 2.66 × Average moving range
The centre line is the average of the baseline values, and 2.66 is the standard factor that turns the average moving range into three standard deviations of routine variation (it is 3 divided by 1.128). The exact convention matters less than using the same one consistently.
Building one for a marketing metric
Choose a metric that matters and has a stable definition: weekly orders, leads, cost per lead, a satisfaction score, branded search volume. A KPIA metric tied directly to an objective. whose definition changed halfway through the history needs either a restated history or a new baseline from the date of the change.
Pick a baseline of comparable periods. For a weekly metric, that means a run of ordinary weeks without launches, outages or tracking problems. Too short a baseline gives limits that jump around as each new point arrives; too long a baseline may include an older version of the business. Treat early limits as provisional and recompute them as history builds.
Then compute the lines. Take the average of the baseline values as the centre line. Subtract each value from the one before it, ignore the sign, and average those differences; there is one fewer difference than there are points. Multiply that average by 2.66, add the result to the centre line for the upper limit and subtract it for the lower limit. For a count or a rate that cannot go below zero, a lower limit under zero is simply shown as zero.
Deal with seasonalityPredictable, repeating changes in demand tied to the calendar, such as months, holidays, weekdays or hours. before drawing limits, because a seasonal metric is not a stable process and its peaks will look like signals. There are three common approaches. Build limits only from the same season in previous years. Chart the ratio of actual to expected, where expected comes from last year's pattern, so the seasonal shape is removed and only the surprise remains. Or chart the year-on-yearComparing a period with the same period a year earlier, aligned by weekday and by holidays that move, so that the seasonal pattern drops out of the change. change instead of the raw value. Campaign weeks need the same treatment: mark them and chart them separately, or the planned spikes will inflate the limits and hide real changes in the ordinary weeks.
Watch ratios with small denominators. A conversion rate from a small number of visitors, or a satisfaction score from a few dozen responses, swings widely for purely arithmetic reasons. Either use limits that widen when the denominator is small, or aggregate to a longer period until the base is large enough to read. The voice of the customer guide makes the same point about survey scores: a movement within the normal range of noise means nothing. For the one-off figures in a routine report that are not charted over time, such as a small campaign's conversion rate or a league table of landing pages, reading small numbers in marketing reports covers ranges, pooling and minimum volumes.
Reading signals and acting
Agree the rules before reading the chart. A typical set is short:
- A single point outside either limit.
- A run of consecutive points on the same side of the centre line, of a length the team fixes in advance.
- A steady trend of consecutive points rising or falling.
- An unusual absence of variation, which often points to a broken data feed repeating the same value.
When a rule fires, the first step is to check the data, because a tracking fault can look exactly like a change in the market. The second is to list what changed around that date: prices, delivery terms, site releases, campaigns, competitor activity, platform changes. Only then decide whether a response is needed. When no rule fires, the discipline is to leave the metric alone. Reacting to routine variation, by cutting a budget after an ordinary dip or celebrating an ordinary peak, adds variation rather than removing it.
Build the rules into the reporting rhythm rather than into everyone's head. The marketing reporting cadence playbook gives the weekly view the job of flagging anomalies; a control chart defines what counts as one. In a dashboard, the guide to dashboard design shows how to lay out a headline chart with supporting views, and the centre line and limits can be added as reference lines so the reader sees the normal range at a glance. For a launch, where there is no history yet, the guide to measuring launch success recommends a baseline from a comparable past launch, which is the closest thing to a centre line a new product has.
Once a lasting change is confirmed, whether good or bad, recompute the limits from the new level and annotate the date. The old limits describe a process that no longer exists.
Worked example (illustrative)
An online florist separates a real drop from an ordinary one
Mossgarten Florists, an invented online florist, charts weekly orders. Its raw orders swing with gifting dates and wedding season, so it does not chart the raw count: each week's orders are divided by the level expected for that week from the previous two years and scaled back to an average week. That flattens the seasonal peaks and leaves only the surprise. It builds its baseline from 30 ordinary weeks in a row of these adjusted orders, with no campaigns or outages. The centre line is 420 orders. The 29 differences between one week and the next, ignoring the sign, average 24 orders, so the limits are 420 ± 2.66 × 24 = 420 ± 63.84, which rounds to 356 and 484.
In March, adjusted orders fall to 371. The owner wants to cut the paid social budget, but the point sits inside the limits and no run rule has fired, so the team notes it and waits. The next week is 431.
In June, no single week falls outside the limits, but 7 consecutive weeks come in below the centre line, between 382 and 409. The team has agreed in advance that 7 in a row on one side is a signal. Tracking checks come back clean. The annotation log shows that the standard delivery fee rose from £4.50 to £5.95 in the week the run began. Basket abandonment at the delivery step has risen from 18% to 26%. The team restores the old fee for orders over £40, and the following 6 weeks return to either side of the centre line.
Limits and criticisms
The method assumes a stable process, and most marketing metrics are not stable without adjustment. Seasonal patterns, campaign bursts, pricing changes and channel shifts all move the level on purpose. A control chart drawn on the raw series will flag every planned peak and miss real changes hidden inside them. The adjustments described above help, but they add judgement, and judgement can be bent towards the answer someone wanted.
Limits drawn from a short history are unreliable. With few points, the centre line and the limits both move noticeably each time a value is added, and an early signal may be nothing more than an unsettled baseline. Young businesses and new metrics have to live with provisional limits for longer than is comfortable.
A chart shows that something changed, not why. A run below the centre line says the system is different; it does not say whether the cause is a price, a competitor, a tracking fault or a change in the mix of customers. The investigation still has to be done, and a team that treats the chart as the diagnosis will fix the wrong thing.
Ratios with small denominators swing widely, and limits that ignore the size of the base will either fire constantly or be set so wide they never fire. And because the chart only judges the process against itself, a metric that is stable at a poor level looks perfectly in control. Whether the level is good enough is a separate question for targets and benchmarks.
Related frameworks
A control chart says that a metric moved; the KPI tree helps find which branch moved it, by splitting the outcome into the drivers and inputs that produce it. The measurement plan framework settles which metrics deserve a chart in the first place and who owns them. When a metric on the chart is also a target, Goodhart's law explains why a sudden improvement may reflect gaming rather than real change. And for reversible changes that are shipped and watched, one-way and two-way door decisions relies on exactly this kind of chart to tell whether a change should be undone.
Common mistakes
- Reacting to every movement. A dip inside the limits with no run rule firing is routine variation.
- Drawing limits through the peaks. Seasonal and campaign weeks in the baseline inflate the limits and hide real changes.
- Never recomputing. After a confirmed, lasting change, the old limits describe a business that no longer exists.
- Treating limits as targets. The limits describe what the process does, not what it should do.
- Skipping the data check. A tracking fault can look exactly like a market change; rule it out before investigating the market.
- Charting tiny ratios weekly. Small denominators produce wide swings; aggregate or use limits that widen with a small base.
- Rules agreed after the fact. Choosing the run length once a pattern has appeared turns the chart into an argument.
