Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Marketing Frameworks

Hierarchy of Evidence for Marketing Claims

A hierarchy of evidence for marketers: ranking proof from anecdote to randomised test and matching it to the size of the decision, with an example.
Frameworkintermediate8 min readUpdated

What a hierarchy of evidence is

Every marketing claim is an argument about cause: the sponsorship brought new customers, the new page lifted sales, the cut channel was not adding anything. The evidence offered for these claims varies enormously. Some is a story someone found convincing. Some is a figure from an ad platform. Some is a chart that rose after a change. Some is a test in which a group was held back at random. These are not equally strong, and treating them as though they were is how budgets end up following the most confident voice in the room.

A hierarchy of evidence puts the kinds of proof in order by how many alternative explanations each can rule out. The question at each level is the same: if the claim were false, could this evidence still look like this? An anecdote could; so could an attributed figure, because Attribution credits conversions that would have happened anyway. A randomised holdout is much harder to fool, because the only systematic difference between the groups is the thing being tested.

The site's guides already reason this way. The guide to measuring offline media climbs a ladder of evidence from response codes to regional tests and chooses the evidence in proportion to the spend. The guide to affiliate reporting and incrementality calls a before-and-after comparison weaker evidence than a planned holdout. The guide to showing your contribution to results grades a marketer's own claims on a wording ladder from "caused" to "coincided with". This page sets out the general ranking those guides each apply to one kind of question. The tutorial on checking an agency's claimed results applies it from the buyer's side, grading the evidence behind each result an agency shows before the agency is appointed.

The levels

Kind of evidenceWhat it can showWhat it cannot rule outTypical marketing source
Anecdote and opinionThat a claim is plausible to someoneAlmost anything: selective memory, enthusiasm, a single unusual caseA sales story, a presenter's enthusiasm, a customer quote, an agency's view
Attributed figures and countsThat the activity touched or was used by some buyersBuyers who would have bought anyway; credit taken from other channelsPlatform conversions, discount code redemptions, tracked partner sales
Before-and-after readingThat a result changed when the activity started or stoppedSeasonality, trend, other changes in the same weeksA chart of sales around a launch, a pause or a price change, read against a baseline
Matched comparisonThat groups with and without the activity differedDifferences between the groups that the matching missedRegions, stores or periods chosen to resemble each other
Randomised test or holdoutThat the activity caused the difference, within a measured rangeChance at small sizes; effects outside the test's market, season or lengthA/B tests, randomised audience holdouts, geographic tests with random assignment

Each step up removes a class of alternative explanation. A before-and-after reading improves on an attributed figure because it looks at the total result rather than the credit one channel claims, but anything else that changed in the same weeks is mixed in with the activity. A matched comparison removes the calendar from the picture by comparing groups over the same period, but the groups may differ in ways nobody thought to match. Random assignment removes that last gap: the groups differ only by chance and by the activity, and the test's range says how large the chance part could be. The guide to incrementality testing covers the holdout, geographic and platform lift designs at the top of the ranking and how to read their uncertainty.

What people say when asked, through post-purchase surveys or "how did you hear about us" questions, sits low in the ranking, but it is a systematic version of anecdote rather than a single story. It is useful for channels that leave no clicks behind, and stronger when its answers agree with other levels.

Matching the evidence to the decision

The point of the ranking is not to demand the top level every time. It is to ask for evidence in proportion to what rides on the decision. Two questions set the level worth asking for.

How much money or reputation is at stake? The guide to cutting the marketing budget treats a test comparing exposed and unexposed groups as the best evidence available, which matters most on the largest lines of spend, where a wrong cut costs most. A small trial can run on weaker evidence; a large commitment should not.

How easy is it to undo? A change that can be reversed next week, such as a headline or an email subject line, can be shipped and watched. A year's sponsorship contract, a brand change or the closure of a channel that took years to build cannot. The one-way and two-way doors framework sorts decisions by reversibility, and the less reversible the door, the higher up the hierarchy the evidence should come from.

Where the top level is not available, use the best evidence that is, and label it. The guide to conversion work without A/B tests records results from before-and-after designs and staged rollouts in the same learning log as tests, labelled with the strength of their evidence. The guide to retargeting that gets all the credit puts any existing holdout or lift result beside the attributed figure and treats the attributed figure alone as an assumption. The guide to choosing an attribution approach combines attribution, experiments and mix models so that each answers the question it is suited to, which keeps the weaker sources in use without letting them settle questions only a test can answer.

Several sources at different levels that point the same way are more persuasive together than any one alone, and a higher-level result that disagrees with a lower-level one should usually win the argument, unless it was badly run.

Worked example: a pet food subscription weighs three claims about a sponsorship (illustrative)

Illustrative example — not a real result

A pet food subscription weighs three claims about a sponsorship

Skivvelto Pet Food, an invented subscription business, ran an 8-week podcast sponsorship costing £12,000 and must decide whether to sign a year's contract for £70,000.

Three claims are on the table. The podcast's presenter says listeners love the brand and the episodes drew record messages. The discount code was redeemed 1,240 times, 430 of them by new subscribers. And because the podcast's ads were inserted by region, Skivvelto had randomly chosen 6 of 12 regions to receive the ads and held the other 6 back.

The team ranks the claims. The presenter's view is anecdote. The code count shows the sponsorship touched buyers, but not how many would have subscribed anyway. The regional holdout compares new subscriptions in exposed and held-back regions over the same 8 weeks and estimates 310 additional subscribers, with a range of 190 to 430.

At the low end of that range the trial cost about £63 per additional subscriber, against the £75 Skivvelto can pay for one. Only the holdout is strong enough to justify a year's commitment, and it does. The team signs, with a second holdout in the contract's fourth month.

Limits and criticisms

A higher level is not automatically right. A randomised test that ran too briefly, used too small a group, changed halfway through or was read early can mislead more than a careful matched comparison. The ranking describes what each kind of evidence can rule out when done well, not the quality of a particular study.

The ranking ignores relevance. A strong test from another market, another season or another product may answer a different question from the one in front of the team, and a weaker reading from the business's own recent data may be more useful. The strength of evidence and its fit to the decision are separate judgements.

Some questions cannot be tested at all. Long-run brand effects, a once-only event or a channel too small to split leave a team with lower levels only, and the hierarchy offers no route to the top in those cases. Pretending a weaker reading is a test is worse than saying plainly what the evidence is.

And insisting on the top level for every decision slows teams down on choices that are cheap to reverse. Waiting weeks for a test on a change that could be shipped and rolled back in a day costs more than the test protects.

The experimentation loop runs the cycle of research, hypothesis, prioritisation, test and learning that produces evidence at the top of the hierarchy; the hierarchy says how much weight each result in the loop's learning log can bear. One-way and two-way doors sorts decisions by how hard they are to reverse, which, together with the money at stake, sets the level of evidence worth asking for. The marketing frameworks library lists the other experimentation and measurement frameworks by the decision they serve.

Common mistakes

  • Reading attributed conversions as caused conversions. Attribution shows what an activity touched, not what it added; Incrementality needs a comparison.
  • Treating a before-and-after chart as a test. Without a comparison group, seasonality and other changes are mixed into the result; set a Baseline and say what else changed.
  • Trusting a test because it is a test. Check its size, length and how it was read before letting it overrule other evidence.
  • Demanding the top level for every decision. Reversible, low-cost choices can be shipped and watched.
  • Accepting a report for a one-way door. Large or irreversible commitments deserve a test or a Holdout Group where one is possible.
  • Hiding the strength of the evidence. Label every result with the kind of evidence behind it.
  • Using a strong result outside its context. A test from another market or season may answer a different question.

Frequently asked questions

Are platform-reported conversions evidence that a channel works?

They are evidence that the channel touched those conversions, not that it caused them. Attribution assigns credit for sales that happened, including sales that would have happened anyway, so a channel that reaches people already on their way to buying can look strong while adding little. Treat the attributed figure as a starting point and ask whether any test or comparison supports it before moving significant budget.

What should a team do when a test is not possible?

Use the strongest comparison that is possible and say which one it is. A before-and-after reading against a trend and similar periods, or a comparison between matched regions or groups, can still narrow the question, and several weaker sources that agree are more persuasive than one that stands alone. Record the strength of the evidence beside the result, so that later readers know how much the decision rested on and when it is worth revisiting.

Does every marketing decision need a randomised test?

No. Tests cost time, traffic and sometimes sales, and many decisions are cheap to reverse if they turn out wrong, so shipping and watching is often the better choice. The case for the strongest evidence grows with the money at stake and with how hard the decision is to undo. A year's commitment, a large budget cut or a change that cannot be rolled back deserves more proof than a headline or a button colour.

Terms used on this page

Frameworks

Where this page is used

Also related

Related