Control charts and A/B tests at a glance
The control charts framework describes the chart as a line chart of a metric with a centre line showing its typical level and upper and lower limits showing how far it normally strays, all three taken from the metric's own history. Values inside the limits, with no unusual pattern, are routine variation and left alone; a value outside them, or a run of values that should not happen by chance, is a signal. The rules for what counts as a signal are agreed before the chart is read, so the weekly argument about whether a dip matters becomes a check against something fixed.
The A/B testing guide describes a test as a randomised controlled experiment. Visitors are assigned at random to the control or the variant before they see anything, both run in the same period, and the metric, sample and decision rule are written down before launch. Because the groups differ only by chance and by the change, a difference too large to be chance is attributed to the change. The sample size and significance guide explains how large the sample has to be and what statistical significanceA test result unlikely enough under 'no real difference' to pass a threshold chosen in advance. does and does not mean, and the experimentation hub places the A/B test among the other designs, holdouts and staged rolloutsReleasing a change to a growing share of visitors, pages or regions in steps and reading each before the next; a test only if groups are random and concurrent. included.
The framework page draws the line itself: a significance check compares two groups over the same period; a control chart compares one process with its own past. That is the whole difference in a sentence. A test removes the calendar from the comparison by running both experiences at once. A chart cannot, so anything else that changed in the same weeks is mixed into its signal. In return, a chart works on a change that has already shipped to everyone, on a metric too thin to split and on the many metrics nobody is testing at all.
Compare table
Control charts vs A/B testing
| Row label | Control charts | A/B testing |
|---|---|---|
| Question answered | Has this metric moved by more than its routine variation, and when? | Did this change cause a difference in this metric for this audience? |
| Comparison group | The metric's own past, summarised as a centre line and limits | A randomly assigned control group running at the same time |
| Traffic needed | Enough history to draw stable limits; small denominators need wider limits or longer periods | Enough visitors per arm to detect the smallest effect worth having |
| Timing | Read every period, against rules agreed in advance | Read once, when the planned sample is reached, in whole weeks |
| What it proves | That the process changed around a date | That the change caused the difference, within a measured range |
| What it cannot rule out | Anything else that changed in the same weeks, including tracking faults | Chance at small sizes; effects outside the test's audience, season or length |
| Effect of seasonality | Must be removed first, or seasonal peaks look like signals | Hits both groups equally, because they run at the same time |
| Who can run it | Anyone with a clean weekly series and agreed rules | A team that can split traffic, check the split and hold the plan |
| How results are read | Signal or no signal, then an investigation of causes | Effect and range against the minimum worth shipping, then guardrails |
| Typical misuse | Treating a signal as proof the latest change worked | Watching daily and stopping the first time it looks significant |
The rows that decide most cases are the second and the sixth. A control group running at the same time is what lets a test attribute a difference to the change; its absence is what limits a chart to saying that something changed. The hierarchy of evidence puts the two at different levels for that reason: a before-and-after reading, which is what a control chart disciplines, cannot rule out seasonality, trend or other changes in the same weeks, while a randomised test can attribute the difference to the change, within its measured range.
When to choose a control chart
Choose a control chart when the change cannot be split. The guide to conversion work without an A/B test sorts those cases: fixes that should simply ship because they repair something broken, wrong or required; changes the traffic cannot test in any period the business will wait; and cheap, reversible changes not worth a test. Each of them is still shipped with a prediction, a reading date and a comparison chosen in advance, and for a metric read every week a control chart is the version of that routine that knows how far the metric normally moves. The one-way and two-way door framework names it as the way a team knows a reversible change needs walking back through.
Choose one, too, for metrics nobody is testing: weekly orders, leads, cost per lead, a satisfaction score, any KPIA metric tied directly to an objective. read in a routine report. Most movement in those figures is noise, and the chart's discipline is to leave the metric alone when no rule fires. The guide to reading small numbers applies the same control limits to recurring metrics and adds the warning that a rate whose denominator varies from week to week needs limits that widen in quieter weeks.
A chart is the weaker choice when the business needs to know what a specific change caused, and could split the traffic to find out. It is also weak on a seasonal metric until the seasonal shape has been removed, which the framework page does by building limits from the same season in earlier years, charting actual against expected, or charting the year-on-year change. And it describes a process against itself, so a metric that is stable at a poor level looks perfectly in control.
When to choose an A/B test
Choose a test when there is a genuine question about a change, the traffic can answer it and the answer is worth having. Those are the three conditions the conversion guide sets before a test earns its cost. The arithmetic comes first: the baseline rate of the step, which the conversion rate calculator turns from counts into a rate; the smallest effect worth having; and the Sample SizeThe number of users or sessions an experiment needs to detect a given effect reliably. per arm that follows from them. If the resulting run is longer than the business will wait, the conversion guide lists the ways to make a test possible, from testing a template across many pages to testing a bolder change or an earlier step.
A test also needs discipline in running. The testing guides take a rule from Evan Miller's essay on how not to run an A/B test: a test checked repeatedly and stopped the first time it looks significant declares false winners far more often than the nominal error rate suggests. So the sample is fixed in advance, the test runs in whole weeks and the result is read once. Monitoring during the run is for defects, such as an uneven split or a broken variant, not for the result.
A test is the weaker choice when the change has to reach everyone at once, when it fixes a fault that should not be left in place for half the traffic, or when the page sees too few conversions to detect any effect the business cares about. Where the choice is between testing whole versions and testing several elements at once, the comparison of A/B and multivariate testing covers it; where the question is a channel's contribution rather than a page change, the comparison of marketing mix modelling and incrementality testing does.
Decision table
| Situation | Choose | Why |
|---|---|---|
| A page change that can be shown to a random half of visitors, on a page with ample conversions | A/B test | A concurrent control attributes the difference to the change |
| A fix to a broken form, a wrong price or a legal requirement | Ship it, then watch the step with a control chart | There is no question to test; the chart catches a fix that did not work |
| A change that has to go live everywhere at once | Control chart with a prediction and rules set before release | No split is possible, so the reading has to be disciplined instead |
| A page whose test would take longer than the business will wait | Make a test possible, or ship and chart it | The traffic cannot answer the question as first framed |
| Weekly leads, orders or cost per lead in a routine report | Control chart | It separates routine variation from shifts worth investigating |
| A strongly seasonal metric | A test, or a chart drawn on seasonally adjusted figures | Raw seasonal peaks look like signals on a chart |
| A feature already live whose cumulative value is in doubt | A holdout group kept away from it for a period | A randomised group still gives a causal answer after launch |
Using both: watching the metric after a winning test ships
The two methods meet after a test is won. A test answers its question for its audience, its season and its length; once the variant ships to everyone, the metric it moved becomes a recurring figure again. A control chart of that metric, with the release date annotated, shows whether the gain holds at full traffic, fades or is swamped by something else. If a lasting shift is confirmed, the framework page's advice is to recompute the limits from the new level, because the old limits describe a process that no longer exists.
They also meet before a test. A signal on a routine chart, a run of weeks below the centre line with no change logged, is a question with no answer yet. The investigation the framework page sets out, checking the data first and then listing what changed around that date, often produces a hypothesis that a test can answer properly. Where a change has already shipped and the business wants a causal reading after the fact, a Holdout GroupA randomly chosen part of an audience kept from exposure to measure what exposure caused. that keeps a random group on the old experience for a fixed period is the design the conversion guide offers between a full test and a plain before-and-after reading.
Two habits keep the pair honest. Label each result by the evidence behind it, so a chart signal is not later quoted as a tested win; the conversion guide records before-and-after results in the same learning log as tests, labelled with the strength of their evidence. And deal with seasonalityPredictable, repeating changes in demand tied to the calendar, such as months, holidays, weekdays or hours. in the chart before drawing limits, because a chart that flags every planned peak trains the team to ignore its signals.
An optician chain checks a booking form change it could not split
Nembyail Opticians, an invented chain of 24 opticians, shortens the eye-test booking form on every branch page from 11 fields to 6 in a single release; its booking system serves one form to all pages, so the traffic cannot be split. Before release, the team charts weekly bookings from 30 ordinary weeks, leaving out the weeks around December and the start of the school year. The centre line is 610 bookings. The average change from one week to the next is 30 bookings, which puts the limits at 530 and 690. The rules, agreed in advance, are one week outside the limits or seven consecutive weeks on the same side of the centre line.
The nine weeks after release record 648, 671, 639, 662, 684, 655, 667, 676 and 659 bookings. No week crosses 690, but the seventh consecutive week above the centre line fires the run rule. The team checks the data first: bookings are counted in the booking system, whose definition has not changed. The annotation log shows no price change, campaign or change to opening hours in those weeks, and the same weeks last year averaged 604 bookings, close to the centre line.
The nine weeks average about 662, roughly 50 bookings a week above the old centre line. The team records the result as a before-and-after reading consistent with the form change, not as a proven effect, keeps the shorter form because it is cheap to reverse and nothing points elsewhere, and recomputes the limits from the new level. For the next proposal, asking for a deposit at booking, which would be harder to undo, it asks the booking system's provider for a random split across branch pages.
Common mistakes
- Reading a chart signal as proof. A signal says the process changed around a date; the cause still has to be found.
- Watching a test like a chart. A test is read once at its planned sample; stopping on the first good day inflates false winners.
- Drawing limits through seasonal peaks. Seasonal and campaign weeks in the baseline widen the limits and hide real changes.
- Choosing the rules after the pattern appears. Run lengths and limits agreed afterwards turn the chart into an argument.
- Testing a fix that should simply ship. A test keeps the fault in place for part of the traffic.
- Running a test the traffic cannot finish. Do the sample-size arithmetic first, then make a test possible or chart the change instead.
- Skipping the data check. A tracking fault can produce a signal on a chart and a false difference in a test.
- Forgetting the metric after the test. Chart the tested metric after rollout to see whether the gain holds.
The comparison library lists the other pairs that are easy to confuse.
