What a review is for
Authors read their own work through what they meant to do. A filter they intended to apply looks applied, a definition they had in mind looks like the one in the query, and a conclusion they reached early colours every chart after it. A review exists to read the work as a decision maker will meet it, from the outside, and to catch the errors that change the decision before the decision is made.
Those errors are rarely arithmetic. The common ones, and the guides that explain each, are:
| Error | What it looks like | Owning guide |
|---|---|---|
| A different definition | Leads that include spam, revenue that includes tax | Metrics layer and definitions |
| A join that multiplies rows | Revenue counted once per item or per touch | SQL for marketers |
| An unfair comparison | A holiday month against a normal one | Comparing periods |
| A rate on a handful of cases | A conversion rate from a few dozen visits | Reading small numbers |
| Mix read as performance | A blended rate moves because the segments changed share | Explaining metric changes |
| A causal claim from a correlation | "The campaign drove sales" from a before-and-after chart | Incrementality testing |
A review is not the place to redo the analysis or to argue for a different question. It answers one thing: can this go to the decision maker as written, and if not, what has to change first. The recurring monthly pack has its own review in the monthly reporting pack SOP, and research studies are written up under the guide to reporting research findings; this routine is for the one-off analysis in between.
Before you start
- The request. The intake record from the request process, with the decision, the decider and the date.
- The analysis as it will be sent: the note, the slides or the table, exactly as the reader will see them.
- The working files. Saved queries, exploration links, exports and any spreadsheet steps, with the date each was run. An analysis whose numbers cannot be traced to a saved query or export is not ready for review.
- The written definitions of every metric used, from the metrics layer or the team's dictionary.
- Access to the same data the author used, at the same level, so the reviewer can rerun rather than trust.
- Time agreed in advance, sized to the decision, and a record to write the outcome into.
The steps
Restate the question and the decision it serves
Before opening any numbers, write in a sentence what the analysis was asked to answer and which decision it informs, from the intake record, then read the analysis's first paragraph and check it answers that question. Kaushik's measurement model, listed in the sources, starts from the business objective and derives measures from it; the review starts from the same place. An analysis that answers a nearby question, such as which campaign had the most leads when the decision was which campaign to cut, is sent back at this step, however good its numbers are. Note anything the decider will need that the analysis does not cover.
Reproduce the headline numbers from the saved queries or exports
Pick the figures the conclusion rests on, usually a few, and produce them yourself from the saved queries, exploration links or exports, not from the author's spreadsheet. Then check one of them against the source of truthThe system whose figure is treated as authoritative for a given number, such as billing for revenue or the CRM for pipeline, with other tools reconciled to it. for that measure, such as orders from the order system rather than from analytics. If a figure comes from a GA4 exploration, read its sampling and threshold indicators as the explorations tutorial describes, because a sampled or thresholded figure reproduced on a different day can differ for that reason alone. A number nobody else can reproduce does not go out.
Check every metric against its written definition
For each metric in the analysis, compare what the query actually counts with the written definition: which events or records, which statuses, which dates, which currency and whether tax and refunds are in or out. The metrics layer guide sets out the parts a definition needs. Where the analysis invents a metric for the occasion, such as a blended cost per qualified lead, check that its definition is written in the note itself. Where the analysis uses a platform's own conversions, check whether the claim needs them or the business's own outcomes; the guide to platform attribution and analytics explains why the two disagree and which decision each suits.
Check grain, joins and filters
Read every query for the three faults the SQL guide names: a join that fans out because the tables are at a different grainWhat one row of a table represents, such as one order, one order line, one session or one customer per day; queries and metric definitions must match it., nulls that vanish from counts or comparisons, and filters that drop or keep the wrong rows, such as test orders, internal traffic or cancelled leads. Compare row counts before and after each join. The routine for checking AI-written SQL works through grain, joins, filters and reconciliation for a single query and applies just as well to SQL a person wrote. Spreadsheet steps get the same reading: a lookup that matched the first of several rows is a join that went wrong quietly.
Check that each comparison is fair
For every "up", "down", "better" or "worse", identify what is being compared with what. The guide to comparing periods ends with a checklist: aligned weekdays and holidays, complete periods, late conversions allowed for, breaks in tracking marked, growth separated from seasonality. Run the analysis's comparisons through it. Comparisons between segments, such as campaigns, regions or customer groups, need the same care: were they the same size, reached at the same time, measured the same way? A comparison that fails is either rebuilt against a fair BaselineAn estimate of what a metric would have done without a given change, campaign or promotion, against which the effect is measured. or removed.
Check the counts behind every rate and the uncertainty around it
Write the count under every rate and percentage in the analysis. The guide to reading small numbers shows how wide the range around a rate is at a given count and how to put a rangeA range of values for an effect, such as a lift in conversion rate, that is consistent with the data at a chosen confidence level. around a figure instead of quoting it alone; add one wherever the decision depends on the figure. Where a blended rate moved, check whether the segments' rates moved or their shares did, the mix shiftA change in a blended metric caused by a change in the mix of segments, such as devices or sources, rather than by any segment performing differently. the guide to explaining metric changes separates from rate. Where the analysis reads a test, check that the sample sizeThe number of users or sessions an experiment needs to detect a given effect reliably. was set in advance and that the test was not stopped when it first looked good; Evan Miller's article, listed in the sources, explains why repeated looks produce false winners, and the experiment results report template lists what a test write-up should state.
Match each claim to the evidence that supports it
List every claim the analysis makes, especially in its summary, and write beside each the evidence offered and how strong it is. The hierarchy of evidence ranks proof from anecdote to randomised test, and the claim's wording should match its level:
Claim as written Evidence offered What the evidence supports "The new landing page increased leads" Leads rose after launch "Leads rose after the launch; other changes in the period are listed" "Paid social drives our new customers" Platform-reported conversions "Paid social reports these conversions under its own attribution" "Segment A converts better" A rate on a small count "Segment A's rate is higher, with a wide range; worth testing" Causal claims need a test or a strong design behind them, as the incrementality testing guide explains. Where the evidence is weaker, the claim is narrowed rather than the analysis abandoned.
Check the charts and the wording against the numbers
Read every chart as the decision maker will: does its axis start where a reader assumes, are the periods labelled, are the units stated, does the title say what the chart shows rather than what the author hopes? Check that every number in the text matches the tables, including rounded figures and percentages written as words. Read the commentary against the what, so what, now what structure: what happened is stated as data, what it means is labelled as interpretation, and the recommendation follows from both. Words such as "proves", "caused", "always" and "significant" are checked against the claims table from the previous step.
Record the outcome and the sign-off
Choose one outcome and write it down with the reviewer's name, the date and what changed:
Outcome When it applies Approved as written The numbers reproduce, the comparisons are fair and the claims match the evidence Approved with changes Corrections or narrower wording agreed and made before sending Returned for more work A number does not reproduce, a definition is wrong or the question was not answered Not to be shared yet The data cannot support any answer to the question in time; the decider is told so Keep the record with the working files, so anyone who later questions the number can see how it was checked. If an error still gets through, the tutorial on correcting a number after a report has gone out takes over. Points of judgement where author and reviewer still differ go into the note's caveats. Patterns across reviews, such as the same join fault recurring, belong in the team's standards rather than in each review.
A climbing centre chain's membership analysis reviewed (illustrative)
Nurvolt Climbing narrows a claim before it reaches the board
Nurvolt Climbing, an invented chain of 4 indoor climbing centres, launches a beginners' course in July. An analysis for the board says the course "doubled membership sign-ups". The reviewer, a second analyst, takes 3 hours.
| Check | What the review found |
|---|---|
| Reproduce | August sign-ups reproduce at 186 from the membership system; the comparison month, May, at 92 |
| Fair comparison | August is a school holiday month and May is not; last August had 151 sign-ups, so the year-on-year rise is about 23% |
| Counts behind rates | "48% of course attendees joined" rests on 15 of 31 attendees, all at 1 centre; the range runs from roughly 30% to 65% |
| Claim and evidence | "Doubled" rested on a before-and-after comparison across a seasonal peak; no other centre ran the course |
Outcome: approved with changes. The analysis goes to the board saying that August sign-ups were about 23% above the previous August, that early course conversion at 1 centre is promising but rests on 31 people, and that the course will run at 2 centres and not at 2 comparable ones through the autumn term to measure its effect.
Common mistakes
- Reading instead of reproducing. A reviewer who looks at the author's numbers checks the arithmetic, not the analysis.
- Reviewing the polish. Fonts and colours are fixed while the definition and the join go unread.
- Skipping the question. A careful answer to the wrong question still misleads the decision.
- Rates without counts. A striking percentage on a few cases passes because nobody wrote the count down.
- Letting causal words through. "Drove" and "caused" survive from the draft into the board paper.
- The commissioner as the only reviewer. The person who wanted the answer checks that it suits them, not that it is right.
- No record. When the number is questioned later, nobody can show how it was checked.
The business intelligence hub lists the other guides in this section.
