Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Marketing Analytics › Business Intelligence for Marketing

Reviewing an Analysis Before It Is Shared

A review routine for one-off marketing analyses: restate the question, rerun the numbers, test comparisons and claims, and record the sign-off before sharing.
Tutorialintermediate9 min readUpdated

What a review is for

Authors read their own work through what they meant to do. A filter they intended to apply looks applied, a definition they had in mind looks like the one in the query, and a conclusion they reached early colours every chart after it. A review exists to read the work as a decision maker will meet it, from the outside, and to catch the errors that change the decision before the decision is made.

Those errors are rarely arithmetic. The common ones, and the guides that explain each, are:

ErrorWhat it looks likeOwning guide
A different definitionLeads that include spam, revenue that includes taxMetrics layer and definitions
A join that multiplies rowsRevenue counted once per item or per touchSQL for marketers
An unfair comparisonA holiday month against a normal oneComparing periods
A rate on a handful of casesA conversion rate from a few dozen visitsReading small numbers
Mix read as performanceA blended rate moves because the segments changed shareExplaining metric changes
A causal claim from a correlation"The campaign drove sales" from a before-and-after chartIncrementality testing

A review is not the place to redo the analysis or to argue for a different question. It answers one thing: can this go to the decision maker as written, and if not, what has to change first. The recurring monthly pack has its own review in the monthly reporting pack SOP, and research studies are written up under the guide to reporting research findings; this routine is for the one-off analysis in between.

Before you start

  • The request. The intake record from the request process, with the decision, the decider and the date.
  • The analysis as it will be sent: the note, the slides or the table, exactly as the reader will see them.
  • The working files. Saved queries, exploration links, exports and any spreadsheet steps, with the date each was run. An analysis whose numbers cannot be traced to a saved query or export is not ready for review.
  • The written definitions of every metric used, from the metrics layer or the team's dictionary.
  • Access to the same data the author used, at the same level, so the reviewer can rerun rather than trust.
  • Time agreed in advance, sized to the decision, and a record to write the outcome into.

The steps

  1. Restate the question and the decision it serves

    Before opening any numbers, write in a sentence what the analysis was asked to answer and which decision it informs, from the intake record, then read the analysis's first paragraph and check it answers that question. Kaushik's measurement model, listed in the sources, starts from the business objective and derives measures from it; the review starts from the same place. An analysis that answers a nearby question, such as which campaign had the most leads when the decision was which campaign to cut, is sent back at this step, however good its numbers are. Note anything the decider will need that the analysis does not cover.

  2. Reproduce the headline numbers from the saved queries or exports

    Pick the figures the conclusion rests on, usually a few, and produce them yourself from the saved queries, exploration links or exports, not from the author's spreadsheet. Then check one of them against the source of truth for that measure, such as orders from the order system rather than from analytics. If a figure comes from a GA4 exploration, read its sampling and threshold indicators as the explorations tutorial describes, because a sampled or thresholded figure reproduced on a different day can differ for that reason alone. A number nobody else can reproduce does not go out.

  3. Check every metric against its written definition

    For each metric in the analysis, compare what the query actually counts with the written definition: which events or records, which statuses, which dates, which currency and whether tax and refunds are in or out. The metrics layer guide sets out the parts a definition needs. Where the analysis invents a metric for the occasion, such as a blended cost per qualified lead, check that its definition is written in the note itself. Where the analysis uses a platform's own conversions, check whether the claim needs them or the business's own outcomes; the guide to platform attribution and analytics explains why the two disagree and which decision each suits.

  4. Check grain, joins and filters

    Read every query for the three faults the SQL guide names: a join that fans out because the tables are at a different grain, nulls that vanish from counts or comparisons, and filters that drop or keep the wrong rows, such as test orders, internal traffic or cancelled leads. Compare row counts before and after each join. The routine for checking AI-written SQL works through grain, joins, filters and reconciliation for a single query and applies just as well to SQL a person wrote. Spreadsheet steps get the same reading: a lookup that matched the first of several rows is a join that went wrong quietly.

  5. Check that each comparison is fair

    For every "up", "down", "better" or "worse", identify what is being compared with what. The guide to comparing periods ends with a checklist: aligned weekdays and holidays, complete periods, late conversions allowed for, breaks in tracking marked, growth separated from seasonality. Run the analysis's comparisons through it. Comparisons between segments, such as campaigns, regions or customer groups, need the same care: were they the same size, reached at the same time, measured the same way? A comparison that fails is either rebuilt against a fair Baseline or removed.

  6. Check the counts behind every rate and the uncertainty around it

    Write the count under every rate and percentage in the analysis. The guide to reading small numbers shows how wide the range around a rate is at a given count and how to put a range around a figure instead of quoting it alone; add one wherever the decision depends on the figure. Where a blended rate moved, check whether the segments' rates moved or their shares did, the mix shift the guide to explaining metric changes separates from rate. Where the analysis reads a test, check that the sample size was set in advance and that the test was not stopped when it first looked good; Evan Miller's article, listed in the sources, explains why repeated looks produce false winners, and the experiment results report template lists what a test write-up should state.

  7. Match each claim to the evidence that supports it

    List every claim the analysis makes, especially in its summary, and write beside each the evidence offered and how strong it is. The hierarchy of evidence ranks proof from anecdote to randomised test, and the claim's wording should match its level:

    Claim as writtenEvidence offeredWhat the evidence supports
    "The new landing page increased leads"Leads rose after launch"Leads rose after the launch; other changes in the period are listed"
    "Paid social drives our new customers"Platform-reported conversions"Paid social reports these conversions under its own attribution"
    "Segment A converts better"A rate on a small count"Segment A's rate is higher, with a wide range; worth testing"

    Causal claims need a test or a strong design behind them, as the incrementality testing guide explains. Where the evidence is weaker, the claim is narrowed rather than the analysis abandoned.

  8. Check the charts and the wording against the numbers

    Read every chart as the decision maker will: does its axis start where a reader assumes, are the periods labelled, are the units stated, does the title say what the chart shows rather than what the author hopes? Check that every number in the text matches the tables, including rounded figures and percentages written as words. Read the commentary against the what, so what, now what structure: what happened is stated as data, what it means is labelled as interpretation, and the recommendation follows from both. Words such as "proves", "caused", "always" and "significant" are checked against the claims table from the previous step.

  9. Record the outcome and the sign-off

    Choose one outcome and write it down with the reviewer's name, the date and what changed:

    OutcomeWhen it applies
    Approved as writtenThe numbers reproduce, the comparisons are fair and the claims match the evidence
    Approved with changesCorrections or narrower wording agreed and made before sending
    Returned for more workA number does not reproduce, a definition is wrong or the question was not answered
    Not to be shared yetThe data cannot support any answer to the question in time; the decider is told so

    Keep the record with the working files, so anyone who later questions the number can see how it was checked. If an error still gets through, the tutorial on correcting a number after a report has gone out takes over. Points of judgement where author and reviewer still differ go into the note's caveats. Patterns across reviews, such as the same join fault recurring, belong in the team's standards rather than in each review.

A climbing centre chain's membership analysis reviewed (illustrative)

Illustrative example — not a real result

Nurvolt Climbing narrows a claim before it reaches the board

Nurvolt Climbing, an invented chain of 4 indoor climbing centres, launches a beginners' course in July. An analysis for the board says the course "doubled membership sign-ups". The reviewer, a second analyst, takes 3 hours.

CheckWhat the review found
ReproduceAugust sign-ups reproduce at 186 from the membership system; the comparison month, May, at 92
Fair comparisonAugust is a school holiday month and May is not; last August had 151 sign-ups, so the year-on-year rise is about 23%
Counts behind rates"48% of course attendees joined" rests on 15 of 31 attendees, all at 1 centre; the range runs from roughly 30% to 65%
Claim and evidence"Doubled" rested on a before-and-after comparison across a seasonal peak; no other centre ran the course

Outcome: approved with changes. The analysis goes to the board saying that August sign-ups were about 23% above the previous August, that early course conversion at 1 centre is promising but rests on 31 people, and that the course will run at 2 centres and not at 2 comparable ones through the autumn term to measure its effect.

Common mistakes

  • Reading instead of reproducing. A reviewer who looks at the author's numbers checks the arithmetic, not the analysis.
  • Reviewing the polish. Fonts and colours are fixed while the definition and the join go unread.
  • Skipping the question. A careful answer to the wrong question still misleads the decision.
  • Rates without counts. A striking percentage on a few cases passes because nobody wrote the count down.
  • Letting causal words through. "Drove" and "caused" survive from the draft into the board paper.
  • The commissioner as the only reviewer. The person who wanted the answer checks that it suits them, not that it is right.
  • No record. When the number is questioned later, nobody can show how it was checked.

The business intelligence hub lists the other guides in this section.

Frequently asked questions

Who should review an analysis?

Someone who did not write it and who can run the queries or open the exports themselves: another analyst, or a marketer with enough data skill to reproduce the headline numbers. The person who commissioned the analysis reviews something different, whether it answers their question, and that check comes on top of the review, not instead of it. When nobody else is available, the author can review their own work after a pause, working through the same steps from the saved files rather than from memory.

Does every analysis need a full review?

The depth should follow the decision. A quick factual answer for a small, reversible choice needs the numbers reproduced and the definition checked; an analysis that will move budget, change a target or reach the board deserves every step. Decide the depth when the request is accepted, so the review is planned time rather than an afterthought squeezed in on the day the answer is due.

What if the reviewer and the author disagree?

Settle it on the evidence. The author shows how the number was produced, the reviewer shows how the reproduction differs, and the difference is traced to a query, a definition or a filter. Where it is a matter of judgement, such as how strongly to word a claim, the narrower wording goes out and the disagreement is recorded in the sign-off, so the reader sees the uncertainty rather than a compromise that hides it.

Terms used on this page

Frameworks

Sources

  1. How Not To Run an A/B Test — Evan Miller
  2. [GA4] Get started with Explorations — Google Analytics Help
  3. Digital Marketing and Measurement Model — Avinash Kaushik

Related