What a hierarchy of evidence is
Every marketing claim is an argument about cause: the sponsorship brought new customers, the new page lifted sales, the cut channel was not adding anything. The evidence offered for these claims varies enormously. Some is a story someone found convincing. Some is a figure from an ad platform. Some is a chart that rose after a change. Some is a test in which a group was held back at random. These are not equally strong, and treating them as though they were is how budgets end up following the most confident voice in the room.
A hierarchy of evidence puts the kinds of proof in order by how many alternative explanations each can rule out. The question at each level is the same: if the claim were false, could this evidence still look like this? An anecdote could; so could an attributed figure, because AttributionAssigning conversion credit to touchpoints. credits conversions that would have happened anyway. A randomised holdout is much harder to fool, because the only systematic difference between the groups is the thing being tested.
The site's guides already reason this way. The guide to measuring offline media climbs a ladder of evidence from response codes to regional tests and chooses the evidence in proportion to the spend. The guide to affiliate reporting and incrementality calls a before-and-after comparison weaker evidence than a planned holdout. The guide to showing your contribution to results grades a marketer's own claims on a wording ladder from "caused" to "coincided with". This page sets out the general ranking those guides each apply to one kind of question. The tutorial on checking an agency's claimed results applies it from the buyer's side, grading the evidence behind each result an agency shows before the agency is appointed.
The levels
| Kind of evidence | What it can show | What it cannot rule out | Typical marketing source |
|---|---|---|---|
| Anecdote and opinion | That a claim is plausible to someone | Almost anything: selective memory, enthusiasm, a single unusual case | A sales story, a presenter's enthusiasm, a customer quote, an agency's view |
| Attributed figures and counts | That the activity touched or was used by some buyers | Buyers who would have bought anyway; credit taken from other channels | Platform conversions, discount code redemptions, tracked partner sales |
| Before-and-after reading | That a result changed when the activity started or stopped | Seasonality, trend, other changes in the same weeks | A chart of sales around a launch, a pause or a price change, read against a baseline |
| Matched comparison | That groups with and without the activity differed | Differences between the groups that the matching missed | Regions, stores or periods chosen to resemble each other |
| Randomised test or holdout | That the activity caused the difference, within a measured range | Chance at small sizes; effects outside the test's market, season or length | A/B tests, randomised audience holdouts, geographic tests with random assignment |
Each step up removes a class of alternative explanation. A before-and-after reading improves on an attributed figure because it looks at the total result rather than the credit one channel claims, but anything else that changed in the same weeks is mixed in with the activity. A matched comparison removes the calendar from the picture by comparing groups over the same period, but the groups may differ in ways nobody thought to match. Random assignment removes that last gap: the groups differ only by chance and by the activity, and the test's range says how large the chance part could be. The guide to incrementality testing covers the holdout, geographic and platform lift designs at the top of the ranking and how to read their uncertainty.
What people say when asked, through post-purchase surveys or "how did you hear about us" questions, sits low in the ranking, but it is a systematic version of anecdote rather than a single story. It is useful for channels that leave no clicks behind, and stronger when its answers agree with other levels.
Matching the evidence to the decision
The point of the ranking is not to demand the top level every time. It is to ask for evidence in proportion to what rides on the decision. Two questions set the level worth asking for.
How much money or reputation is at stake? The guide to cutting the marketing budget treats a test comparing exposed and unexposed groups as the best evidence available, which matters most on the largest lines of spend, where a wrong cut costs most. A small trial can run on weaker evidence; a large commitment should not.
How easy is it to undo? A change that can be reversed next week, such as a headline or an email subject line, can be shipped and watched. A year's sponsorship contract, a brand change or the closure of a channel that took years to build cannot. The one-way and two-way doors framework sorts decisions by reversibility, and the less reversible the door, the higher up the hierarchy the evidence should come from.
Where the top level is not available, use the best evidence that is, and label it. The guide to conversion work without A/B tests records results from before-and-after designs and staged rollouts in the same learning log as tests, labelled with the strength of their evidence. The guide to retargeting that gets all the credit puts any existing holdout or lift result beside the attributed figure and treats the attributed figure alone as an assumption. The guide to choosing an attribution approach combines attribution, experiments and mix models so that each answers the question it is suited to, which keeps the weaker sources in use without letting them settle questions only a test can answer.
Several sources at different levels that point the same way are more persuasive together than any one alone, and a higher-level result that disagrees with a lower-level one should usually win the argument, unless it was badly run.
Worked example: a pet food subscription weighs three claims about a sponsorship (illustrative)
A pet food subscription weighs three claims about a sponsorship
Skivvelto Pet Food, an invented subscription business, ran an 8-week podcast sponsorship costing £12,000 and must decide whether to sign a year's contract for £70,000.
Three claims are on the table. The podcast's presenter says listeners love the brand and the episodes drew record messages. The discount code was redeemed 1,240 times, 430 of them by new subscribers. And because the podcast's ads were inserted by region, Skivvelto had randomly chosen 6 of 12 regions to receive the ads and held the other 6 back.
The team ranks the claims. The presenter's view is anecdote. The code count shows the sponsorship touched buyers, but not how many would have subscribed anyway. The regional holdout compares new subscriptions in exposed and held-back regions over the same 8 weeks and estimates 310 additional subscribers, with a range of 190 to 430.
At the low end of that range the trial cost about £63 per additional subscriber, against the £75 Skivvelto can pay for one. Only the holdout is strong enough to justify a year's commitment, and it does. The team signs, with a second holdout in the contract's fourth month.
Limits and criticisms
A higher level is not automatically right. A randomised test that ran too briefly, used too small a group, changed halfway through or was read early can mislead more than a careful matched comparison. The ranking describes what each kind of evidence can rule out when done well, not the quality of a particular study.
The ranking ignores relevance. A strong test from another market, another season or another product may answer a different question from the one in front of the team, and a weaker reading from the business's own recent data may be more useful. The strength of evidence and its fit to the decision are separate judgements.
Some questions cannot be tested at all. Long-run brand effects, a once-only event or a channel too small to split leave a team with lower levels only, and the hierarchy offers no route to the top in those cases. Pretending a weaker reading is a test is worse than saying plainly what the evidence is.
And insisting on the top level for every decision slows teams down on choices that are cheap to reverse. Waiting weeks for a test on a change that could be shipped and rolled back in a day costs more than the test protects.
Related frameworks
The experimentation loop runs the cycle of research, hypothesis, prioritisation, test and learning that produces evidence at the top of the hierarchy; the hierarchy says how much weight each result in the loop's learning log can bear. One-way and two-way doors sorts decisions by how hard they are to reverse, which, together with the money at stake, sets the level of evidence worth asking for. The marketing frameworks library lists the other experimentation and measurement frameworks by the decision they serve.
Common mistakes
- Reading attributed conversions as caused conversions. Attribution shows what an activity touched, not what it added; IncrementalityConversions caused by marketing. needs a comparison.
- Treating a before-and-after chart as a test. Without a comparison group, seasonality and other changes are mixed into the result; set a BaselineAn estimate of what a metric would have done without a given change, campaign or promotion, against which the effect is measured. and say what else changed.
- Trusting a test because it is a test. Check its size, length and how it was read before letting it overrule other evidence.
- Demanding the top level for every decision. Reversible, low-cost choices can be shipped and watched.
- Accepting a report for a one-way door. Large or irreversible commitments deserve a test or a Holdout GroupA randomly chosen part of an audience kept from exposure to measure what exposure caused. where one is possible.
- Hiding the strength of the evidence. Label every result with the kind of evidence behind it.
- Using a strong result outside its context. A test from another market or season may answer a different question.
