Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Templates

Experiment Results Report Template

An experiment results report template: hypothesis recap, design as run, primary metric and guardrails, confidence, segments, the decision and the learning.
Templateintermediate10 min readUpdated

What this template is for

Use the report for every concluded test: an A/B Test on a page, an email variant, a pricing change, a bid strategy split, or a test that compares an exposed group with a holdout group. Winners, losers and inconclusive results all get one, in the same format, because the inconclusive tests are usually the majority and they tell the next person which ideas have already been tried at what sample.

The report sits at the end of a chain of documents. Before the test, the experiment hypothesis recorded the evidence, the change, the expected effect, the metric, the guardrails and the decision rule. The experiment library holds ready-made designs for common tests. After the test, the experimentation programme playbook keeps the learning log that this report feeds and runs the read-out meeting where decisions are committed. The results report is not a place to redesign the test or to argue for a new decision rule; it reads the test that ran against the rule that was written.

The structure follows the order a careful reader checks a result in, which the A/B testing guide sets out: first whether the test ran as planned, then the primary metric with its interval, then the guardrails, and only then, carefully, the segments. Putting validity first is deliberate. A strong-looking result from a test that did not reach its sample or that split traffic unevenly is not a result, and a report that leads with the headline number invites the reader to skip the checks.

The template

One report per test, linked from the hypothesis document and from the learning log entry. Keep the field names so reports can be compared across the programme.

FieldWhat to write
Test name and link to the planThe test ID, its short name and a link to the hypothesis document it was run from
Design as runVariants, unit of randomisation, split, start and end dates, and every deviation from the plan
Sample and duration checkWhether the planned sample and duration were reached, and whether the test was stopped early or peeked at
Primary metric resultControl and variant values, the difference, its confidence interval and the significance result
GuardrailsEach guardrail metric with its bound and whether it held
Segments and validity checksPre-registered segments only, the sample ratio check and the tracking checks
DecisionShip, iterate or stop, with the decision rule quoted from the plan
LearningWhat the result says about the customer, beyond this page, email or ad
Follow-ups and log entryNext tests, roll-out tasks with owners and the learning log reference

Keep the report to one page plus an appendix. The analysis workbook, screenshots of each arm and incident notes go in the appendix or behind links, so the reader meets the validity checks and the decision first.

How to fill each section

Use the same ID the test carried in the backlog and the hypothesis document. Link to the plan rather than restating it: the report's reader needs to see what was predicted and what rule was set, and the plan is the evidence that both were fixed before any data arrived. If no plan exists, say so at the top of the report; a test without a written rule can still be described, but its decision carries less weight.

Design as run

Record what actually ran, which is not always what was planned. List the variants, the unit of randomisation (user, session, account or region), the allocation split, and the exact start and end dates. Then list every deviation: a variant changed mid-test, a traffic source added, a promotion that overlapped, a bug that showed the wrong content for a period. The A/B testing guide explains why each of these matters. A deviation does not always invalidate a test, but it must be visible so the reader can judge it.

Sample and duration check

Compare the sample reached in each arm with the planned sample size, and the days run with the planned duration in whole weeks. State plainly whether anyone looked at results before the end and whether the test was stopped early. Repeatedly checking a running test and stopping when it looks significant inflates false positives, a problem Evan Miller's note on how not to run an A/B test explains clearly; the sample size and significance guide covers the same ground and the sequential designs that allow planned interim looks. If the sample was not reached, the honest label for the result is indicative at best.

Primary metric result

Report the control value, the variant value, the absolute and relative difference, the confidence interval on the difference and the Statistical Significance result against the threshold set in the plan. Lead with the interval, not the point estimate: the observed lift is the single most likely value, but the result is the range. The sample size and significance guide explains the three readings of an interval: wholly above zero, wholly below, or straddling it. The conversion rate calculator turns counts into rates when the report needs them stated consistently.

Guardrails

List each guardrail from the plan with its bound and the observed value in each arm, and mark it held or breached. Do not add guardrails at this stage; a metric chosen after the data is in can be used to justify almost any decision. A win on the primary metric with a breached guardrail is a decision for the business owner, not an automatic rollout, and the A/B testing guide treats it that way.

Segments and validity checks

Report only the segments named in the plan. Segments found after the fact are hypotheses for the next test, and the report should list them under follow-ups rather than here. The validity checks are the sample ratio check (whether the observed split matches the planned split closely enough to rule out an assignment fault) and the tracking checks (whether the events that define the metric fired in both arms throughout). The data quality guide covers the tracking failures that most often distort a test, from duplicate events to consent changes mid-test. For a test read in Google Analytics, the A/B test analysis tutorial shows how to confirm that the exposure event and its variant parameter arrived in both arms before any figure goes into this section. An exposure event missing from one arm leaves that arm without a count, so the comparison cannot be read.

Decision

Quote the decision rule from the plan, then apply it. The outcome is one of three: ship, iterate or stop. If the rule produces a decision the team dislikes, the report still records what the rule says and then records any override separately, with the name of the person who made it and why. The experimentation programme playbook explains how decisions are committed at the read-out with an owner and a date, and why the "implemented" field in the learning log matters as much as the result.

Learning

Write what the team now believes about its customers that it did not believe before, in one or two sentences. This is the field that turns a results list into learning, because it generalises beyond the page or ad that was tested. A flat result still has a learning: the barrier the hypothesis named was not the one holding people back, or not at this size. The experimentation cadence guide shows how these sentences are reviewed as a body each quarter to draw out themes for the backlog.

Follow-ups and log entry

List the next tests the result suggests, including any post-hoc segment worth a proper test, the roll-out tasks with owners and dates, and the learning log reference. If the variant ships, add a date to revisit the shipped change against later data, since test lifts often shrink once a change meets the whole audience. The playbook keeps the log; the report is where the entry's content comes from. The tutorial on shipping a winning test variant turns the roll-out tasks into steps, from rebuilding the variant in the site's code to retiring the test.

Filled example

Below, the report is completed for an invented business. The figures, dates and names are illustrative.

Illustrative example — not a real result

Cindermoor Houseplants: delivery date promise on product pages

FieldCindermoor's results report
Test name and link to the planEXP-027, "Delivery date promise on product pages", run from hypothesis document EXP-027 v2 (approved 29 April 2026)
Design as runControl: product pages showing "Delivered in 2–4 working days". Variant: a dated promise ("Order in the next 3 h 12 min for delivery Thursday 14 May") calculated from courier cut-offs and plant size. Randomised by user, 50/50, all product pages, 6 May to 26 May 2026 (21 days). Deviation: on 9 May, from 10:00 to 12:05, the variant showed next-day dates for large plants that ship in two days; fixed, left in the data and checked separately (excluding the window changes the result by less than 0.1 percentage points)
Sample and duration checkPlanned 21,000 users per arm over three full weeks. Reached 22,140 (control) and 22,060 (variant) over 21 days. No interim reads of the primary metric; health checks only, as the plan required
Primary metric resultProduct page to order within the session: control 712 orders (3.22%), variant 787 orders (3.57%). Absolute difference +0.35 percentage points, relative +10.9%; 95% interval on the absolute difference +0.01 to +0.69 points (about +0.4% to +21% relative). Significant at the 5% level set in the plan (p ≈ 0.04)
GuardrailsAverage order value (bound: no fall beyond £1.00): £31.10 control, £31.40 variant, held. "Where is my order" contacts per 1,000 orders (bound: rise of no more than 5): 14 control, 17 variant, held. Product page load time on mobile (bound: under 2.5 s): 1.9 s both arms, held
Segments and validity checksPre-registered segments: device (mobile, desktop) and plant size (small, posted; large, couriered). Direction positive in all four; none has enough sample to be read on its own. Sample ratio 50.1% / 49.9%, within the check's tolerance. Order event and variant parameter verified in both arms on days 1, 8 and 15
DecisionRule from the plan: "Ship if the primary metric is significantly higher at the 5% level with all guardrails held; iterate if positive but not significant; stop if flat or negative." Result: ship. The interval's lower end is close to zero, so the expected benefit is stated as a range in the roll-out note
LearningUncertainty about when a living plant will arrive is a purchase barrier on product pages; a concrete date reduces it more than a range of days does. The contact rise, though inside its bound, suggests a dated promise raises expectations that operations must meet
Follow-ups and log entryRoll out to all product pages by 12 June (owner: e-commerce lead), with courier cut-offs maintained by the operations manager. Next test: dated promise in the basket and on category pages (EXP-031). Watch "where is my order" contacts monthly. Revisit against August order data. Learning log entry LL-044

Common mistakes

  • Leading with the headline lift. A report that opens with the point estimate primes the reader to skip the validity checks. Put design as run and the sample check first.
  • Reporting the point estimate as the result. The result is the interval. A win stated as a single number will disappoint when the shipped change lands nearer the lower end.
  • Calling an inconclusive test a loss. An interval that spans both harm and a large gain means the test lacked the traffic to answer. Label it inconclusive and record the sample it reached.
  • Rescuing a lost test with segments. With enough segments, one will look like a winner by chance. Report pre-registered segments only and move the rest to follow-ups.
  • Changing the decision rule after the data. The rule exists to be applied, not negotiated. Record an override separately with a name and a reason.
  • Hiding deviations. A bug, an overlapping promotion or a mid-test change belongs in the report even when it seems minor; the reader decides what it means.
  • A learning that only describes the page. "The variant won" is not a learning. Say what it reveals about the customer, so the next test can build on it.
  • No owner for the roll-out. Winning tests that never ship are a common failure of programmes. Name the owner and date, and check them at the next read-out.

The hypothesis document this report closes, and the monthly report where concluded tests are summarised, are in the templates library.

Terms used on this page

Tools to try

Frameworks

Where this page is used

Also related

Sources

  1. How Not To Run an A/B Test — Evan Miller

Related