Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Comparisons

Control Charts vs A/B Testing

Control charts and A/B tests compared: telling a real shift from noise in a metric over time against proving cause with a split, and when each is right.
Comparisonintermediate11 min readUpdated

Control charts and A/B tests at a glance

The control charts framework describes the chart as a line chart of a metric with a centre line showing its typical level and upper and lower limits showing how far it normally strays, all three taken from the metric's own history. Values inside the limits, with no unusual pattern, are routine variation and left alone; a value outside them, or a run of values that should not happen by chance, is a signal. The rules for what counts as a signal are agreed before the chart is read, so the weekly argument about whether a dip matters becomes a check against something fixed.

The A/B testing guide describes a test as a randomised controlled experiment. Visitors are assigned at random to the control or the variant before they see anything, both run in the same period, and the metric, sample and decision rule are written down before launch. Because the groups differ only by chance and by the change, a difference too large to be chance is attributed to the change. The sample size and significance guide explains how large the sample has to be and what statistical significance does and does not mean, and the experimentation hub places the A/B test among the other designs, holdouts and staged rollouts included.

The framework page draws the line itself: a significance check compares two groups over the same period; a control chart compares one process with its own past. That is the whole difference in a sentence. A test removes the calendar from the comparison by running both experiences at once. A chart cannot, so anything else that changed in the same weeks is mixed into its signal. In return, a chart works on a change that has already shipped to everyone, on a metric too thin to split and on the many metrics nobody is testing at all.

Compare table

Control charts vs A/B testing

Row labelControl chartsA/B testing
Question answeredHas this metric moved by more than its routine variation, and when?Did this change cause a difference in this metric for this audience?
Comparison groupThe metric's own past, summarised as a centre line and limitsA randomly assigned control group running at the same time
Traffic neededEnough history to draw stable limits; small denominators need wider limits or longer periodsEnough visitors per arm to detect the smallest effect worth having
TimingRead every period, against rules agreed in advanceRead once, when the planned sample is reached, in whole weeks
What it provesThat the process changed around a dateThat the change caused the difference, within a measured range
What it cannot rule outAnything else that changed in the same weeks, including tracking faultsChance at small sizes; effects outside the test's audience, season or length
Effect of seasonalityMust be removed first, or seasonal peaks look like signalsHits both groups equally, because they run at the same time
Who can run itAnyone with a clean weekly series and agreed rulesA team that can split traffic, check the split and hold the plan
How results are readSignal or no signal, then an investigation of causesEffect and range against the minimum worth shipping, then guardrails
Typical misuseTreating a signal as proof the latest change workedWatching daily and stopping the first time it looks significant

The rows that decide most cases are the second and the sixth. A control group running at the same time is what lets a test attribute a difference to the change; its absence is what limits a chart to saying that something changed. The hierarchy of evidence puts the two at different levels for that reason: a before-and-after reading, which is what a control chart disciplines, cannot rule out seasonality, trend or other changes in the same weeks, while a randomised test can attribute the difference to the change, within its measured range.

When to choose a control chart

Choose a control chart when the change cannot be split. The guide to conversion work without an A/B test sorts those cases: fixes that should simply ship because they repair something broken, wrong or required; changes the traffic cannot test in any period the business will wait; and cheap, reversible changes not worth a test. Each of them is still shipped with a prediction, a reading date and a comparison chosen in advance, and for a metric read every week a control chart is the version of that routine that knows how far the metric normally moves. The one-way and two-way door framework names it as the way a team knows a reversible change needs walking back through.

Choose one, too, for metrics nobody is testing: weekly orders, leads, cost per lead, a satisfaction score, any KPI read in a routine report. Most movement in those figures is noise, and the chart's discipline is to leave the metric alone when no rule fires. The guide to reading small numbers applies the same control limits to recurring metrics and adds the warning that a rate whose denominator varies from week to week needs limits that widen in quieter weeks.

A chart is the weaker choice when the business needs to know what a specific change caused, and could split the traffic to find out. It is also weak on a seasonal metric until the seasonal shape has been removed, which the framework page does by building limits from the same season in earlier years, charting actual against expected, or charting the year-on-year change. And it describes a process against itself, so a metric that is stable at a poor level looks perfectly in control.

When to choose an A/B test

Choose a test when there is a genuine question about a change, the traffic can answer it and the answer is worth having. Those are the three conditions the conversion guide sets before a test earns its cost. The arithmetic comes first: the baseline rate of the step, which the conversion rate calculator turns from counts into a rate; the smallest effect worth having; and the Sample Size per arm that follows from them. If the resulting run is longer than the business will wait, the conversion guide lists the ways to make a test possible, from testing a template across many pages to testing a bolder change or an earlier step.

A test also needs discipline in running. The testing guides take a rule from Evan Miller's essay on how not to run an A/B test: a test checked repeatedly and stopped the first time it looks significant declares false winners far more often than the nominal error rate suggests. So the sample is fixed in advance, the test runs in whole weeks and the result is read once. Monitoring during the run is for defects, such as an uneven split or a broken variant, not for the result.

A test is the weaker choice when the change has to reach everyone at once, when it fixes a fault that should not be left in place for half the traffic, or when the page sees too few conversions to detect any effect the business cares about. Where the choice is between testing whole versions and testing several elements at once, the comparison of A/B and multivariate testing covers it; where the question is a channel's contribution rather than a page change, the comparison of marketing mix modelling and incrementality testing does.

Decision table

SituationChooseWhy
A page change that can be shown to a random half of visitors, on a page with ample conversionsA/B testA concurrent control attributes the difference to the change
A fix to a broken form, a wrong price or a legal requirementShip it, then watch the step with a control chartThere is no question to test; the chart catches a fix that did not work
A change that has to go live everywhere at onceControl chart with a prediction and rules set before releaseNo split is possible, so the reading has to be disciplined instead
A page whose test would take longer than the business will waitMake a test possible, or ship and chart itThe traffic cannot answer the question as first framed
Weekly leads, orders or cost per lead in a routine reportControl chartIt separates routine variation from shifts worth investigating
A strongly seasonal metricA test, or a chart drawn on seasonally adjusted figuresRaw seasonal peaks look like signals on a chart
A feature already live whose cumulative value is in doubtA holdout group kept away from it for a periodA randomised group still gives a causal answer after launch

Using both: watching the metric after a winning test ships

The two methods meet after a test is won. A test answers its question for its audience, its season and its length; once the variant ships to everyone, the metric it moved becomes a recurring figure again. A control chart of that metric, with the release date annotated, shows whether the gain holds at full traffic, fades or is swamped by something else. If a lasting shift is confirmed, the framework page's advice is to recompute the limits from the new level, because the old limits describe a process that no longer exists.

They also meet before a test. A signal on a routine chart, a run of weeks below the centre line with no change logged, is a question with no answer yet. The investigation the framework page sets out, checking the data first and then listing what changed around that date, often produces a hypothesis that a test can answer properly. Where a change has already shipped and the business wants a causal reading after the fact, a Holdout Group that keeps a random group on the old experience for a fixed period is the design the conversion guide offers between a full test and a plain before-and-after reading.

Two habits keep the pair honest. Label each result by the evidence behind it, so a chart signal is not later quoted as a tested win; the conversion guide records before-and-after results in the same learning log as tests, labelled with the strength of their evidence. And deal with seasonality in the chart before drawing limits, because a chart that flags every planned peak trains the team to ignore its signals.

Illustrative example — not a real result

An optician chain checks a booking form change it could not split

Nembyail Opticians, an invented chain of 24 opticians, shortens the eye-test booking form on every branch page from 11 fields to 6 in a single release; its booking system serves one form to all pages, so the traffic cannot be split. Before release, the team charts weekly bookings from 30 ordinary weeks, leaving out the weeks around December and the start of the school year. The centre line is 610 bookings. The average change from one week to the next is 30 bookings, which puts the limits at 530 and 690. The rules, agreed in advance, are one week outside the limits or seven consecutive weeks on the same side of the centre line.

The nine weeks after release record 648, 671, 639, 662, 684, 655, 667, 676 and 659 bookings. No week crosses 690, but the seventh consecutive week above the centre line fires the run rule. The team checks the data first: bookings are counted in the booking system, whose definition has not changed. The annotation log shows no price change, campaign or change to opening hours in those weeks, and the same weeks last year averaged 604 bookings, close to the centre line.

The nine weeks average about 662, roughly 50 bookings a week above the old centre line. The team records the result as a before-and-after reading consistent with the form change, not as a proven effect, keeps the shorter form because it is cheap to reverse and nothing points elsewhere, and recomputes the limits from the new level. For the next proposal, asking for a deposit at booking, which would be harder to undo, it asks the booking system's provider for a random split across branch pages.

Common mistakes

  • Reading a chart signal as proof. A signal says the process changed around a date; the cause still has to be found.
  • Watching a test like a chart. A test is read once at its planned sample; stopping on the first good day inflates false winners.
  • Drawing limits through seasonal peaks. Seasonal and campaign weeks in the baseline widen the limits and hide real changes.
  • Choosing the rules after the pattern appears. Run lengths and limits agreed afterwards turn the chart into an argument.
  • Testing a fix that should simply ship. A test keeps the fault in place for part of the traffic.
  • Running a test the traffic cannot finish. Do the sample-size arithmetic first, then make a test possible or chart the change instead.
  • Skipping the data check. A tracking fault can produce a signal on a chart and a false difference in a test.
  • Forgetting the metric after the test. Chart the tested metric after rollout to see whether the gain holds.

The comparison library lists the other pairs that are easy to confuse.

Frequently asked questions

Can a control chart prove that a change worked?

No. A control chart shows that a metric moved by more than its routine variation, and roughly when, which is strong evidence that something in the system changed. It cannot say what. A price change, a competitor, a campaign, a tracking fault or a shift in the mix of customers in the same weeks would produce the same signal. Read a signal as the start of an investigation: check the data, list what else changed around that date, and only then decide what the signal most likely means. Proof of cause needs a randomised comparison.

Is a control chart just a before-and-after comparison?

It is a disciplined one. A plain before-and-after comparison sets one period against another and invites reading whatever difference appears as the effect. A control chart first measures how far the metric normally moves from week to week, then fixes in advance the rules for what counts as a signal, so an ordinary wobble is not mistaken for a result. It still shares the weakness of every before-and-after reading: anything else that changed in the same weeks is mixed in.

Why not just watch the A/B test results every day like a chart?

Because a test read repeatedly and stopped the first time it looks significant declares false winners far more often than its threshold suggests, which is the problem Evan Miller's essay on how not to run an A/B test describes and the site's testing guides adopt as a rule. A test is monitored daily for defects, such as a broken variant or an uneven split, but its result is read once, when the planned sample is reached. A control chart is designed to be read every period, because its limits and rules were set for exactly that.

Terms used on this page

Tools to try

Frameworks

Where this page is used

Also related

Sources

  1. How Not To Run an A/B Test — Evan Miller

Related