What this playbook covers and when to run it
The playbook covers the governance and operation of a marketing experimentation programme across websites, landing pages, email, paid media and product-led marketing. It runs from an idea arriving to a decision being made and recorded. It does not teach how to calculate sample size, how to write a hypothesis or how to design an incrementality study; the sample size and significance guide, the experiment hypothesis template and the incrementality testing guide own those.
Run it when tests are happening in more than one place and nobody can say what was learned last quarter, when results are disputed after the fact, or when the team wants to scale testing beyond the occasional A/B TestControlled comparison of two variants. on a landing page. It needs a minimum of traffic or audience to be worth running formally; where volumes are very low, many marketing questions are better answered by research than by split tests.
The playbook is part of the playbooks library. It builds on the experimentation cadence guide, which explains why rhythm matters for growth teams, and on the experimentation loop framework, which describes the cycle this playbook operationalises. To make sure the backlog serves the year's plan rather than whatever is easiest to test, the tutorial on turning plan assumptions into a learning agenda ranks the assumptions the plan rests on and sends only the ones that need an experiment into this programme.
Roles and responsibilities
- Experimentation lead. Owns the programme: intake, the roadmap, prioritisation, the stand-up and the learning log. Has the authority to stop a test that breaks the rules.
- Analyst. Owns test design quality and results: metrics, sample size, duration, stopping rulesA rule set in advance that says when to stop: when a test has run long enough or must halt, or when a flexible budget line stops increasing., the analysis and the read-out. Signs off that a result is reliable.
- Developer. Owns the build: implementing variants, targeting, tracking of test events, and the technical QA before launch.
- Designer. Owns the variant experience: designs that express the hypothesis clearly and work across devices.
- Channel owner. Owns the area being tested, whether a page, a journey or a campaign. Brings ideas, approves tests in their area and implements winners.
The analyst's sign-off is the programme's quality gate. If channel owners can declare their own winners, the programme will slowly fill with false positives that do not hold when shipped. Agree at the outset that only the analyst declares a result.
Cadence, inputs and outputs
| Cadence | Owner | Inputs | Outputs |
|---|---|---|---|
| Continuous intake | Experimentation lead | Ideas from channel owners, research, analytics, customer feedback | Ideas logged with a problem statement and evidence |
| Weekly stand-up | Experimentation lead, developer, analyst | Tests in design, build, QA and live | Launch approvals, QA sign-offs, live tests checked for problems |
| Fortnightly read-outs | Analyst, channel owners | Finished tests with full data | Read-outs, decisions to ship, iterate or drop; learning log entries |
| Quarterly roadmap review | Experimentation lead | Learning log, backlog, win rate, business priorities | Re-prioritised roadmap, process changes |
Keep the weekly stand-up and the fortnightly read-out separate. The stand-up is about whether tests are healthy and ready; the read-out is about what finished tests mean. Mixing them invites people to discuss live results before a test has finished. A smaller growth team that has to merge them can follow the tutorial on running a weekly growth meeting, which checks live tests for health only and calls finished tests against their decision rules.
The playbook in phases
Intake ideas
Every idea enters through one door: a simple form or backlog that asks for the problem, the evidence for it, the proposed change, the audience affected and the metric it should move. Evidence can be analytics, user research, customer feedback or a previous test; the Nielsen Norman Group publishes extensive usability research that is often a useful starting point for website ideas. Ideas without a problem statement go back to the submitter. The experimentation lead reviews intake weekly, merges duplicates and asks for missing evidence. Encourage ideas from everywhere, including sales and support, because the people closest to customers often see problems analytics cannot. Every idea, accepted or not, stays in the log.
Prioritise
Score the backlog using a consistent framework; the prioritising tests guide compares the common ones. Whatever model is used, it should weigh the expected impact on the primary metric, the strength of the evidence, the reach of the change and the effort to build and run it. Check feasibility early: some ideas cannot reach a readable result in a reasonable time with the available traffic, and those are better answered another way. The experimentation lead builds a roadmap from the top of the list, avoiding tests that would overlap on the same audience and page. Share the roadmap so channel owners know when their ideas will run.
Design and QA
For each test, the analyst and the channel owner write the design using the hypothesis template: the hypothesis, the primary metric, guardrail metrics, the audience, the variants, the expected effect size, the required sample, the duration and the stopping rule. The duration should cover full business cycles, usually at least whole weeks. The designer and developer build the variants. QA checks that variants render correctly across devices and browsers, that users are assigned randomly and consistently, that test events record, and that nothing else on the page changed. The weekly stand-up approves the launch only when design and QA are both signed off. The A/B test launch checklist expands this phase into the lines to tick for each test, from the stopping rule to the first day's traffic split.
Run and monitor
Once launched, a test runs for its planned duration without changes to the variants, the targeting or the traffic sources. The developer and analyst check in the first days for problems: a sample ratioA persistent gap between the share of visitors an A/B test was meant to assign to each variant and the share each actually received. that differs from the planned split, a variant that errors on a browser, tracking that stopped. Problems like these can justify stopping and restarting a test; an encouraging early result cannot. The weekly stand-up reviews every live test for health only, not for results. Resist pressure to peek and call a winner. If a business event, such as a large promotion, disrupts the test, note it and decide with the analyst whether the data is still usable.
Read out and log learnings
When a test reaches its planned end, the analyst analyses it against the design: the primary metric, the guardrails and any segments that were specified in advance. Segments found after the fact are hypotheses for the next test, not results. The read-out states the result, the confidence, what it means for the hypothesis and the decision: ship, iterate or drop. The channel owner commits to the decision with a date. The experimentation lead adds the test to the learning log, linked to its design and read-out. The experiment results report template gives the read-out a fixed format, from the sample check to the decision and the learning. For tests that measure channel IncrementalityConversions caused by marketing., the read-out also notes how the result should change budget or reporting.
The learning log and how decisions follow
The learning log is the programme's memory and its main product. Each entry records the question, the hypothesis, the design, the result, the decision and whether the decision was carried out. Over time it answers questions like "have we tested social proof on the pricing page?" and "what do we know about how our audience responds to urgency?" without anyone relying on memory.
| Field | Why it matters |
|---|---|
| Question and hypothesis | Lets future readers judge whether a new idea is genuinely new |
| Area and audience | Shows where the learning applies and where it might not |
| Primary metric and result | Records what moved, usually a CVRShare of visits or clicks that convert. or a downstream measure |
| Confidence and caveats | Shows how much weight the result can bear |
| Decision and owner | Makes clear what was supposed to happen next |
| Implemented? | Closes the loop between learning and action |
The "implemented" field matters most. Programmes often generate winning tests that never ship because the change sat in a development queue behind other work. At each fortnightly read-out, review the open decisions from previous read-outs. At each quarterly review, report the share of winning tests that have shipped. A learning that never changes anything is an expensive curiosity.
A subscription meal-kit company building its log
A meal-kit subscription company has run about 40 tests over two years across its site, emails and paid social, with results scattered across slide decks. The new experimentation lead rebuilds a learning log from what can be recovered and finds that 9 tests were declared winners from early peeks and that 6 genuine winners were never implemented. The team introduces analyst sign-off and a fortnightly read-out that reviews open decisions. In the next two quarters it runs 18 tests; 5 win, 4 lose clearly and 9 show no reliable difference. All 5 winners ship within a month, and one losing test on discount framing stops a planned sitewide change.
Measuring the playbook
Measure the programme on throughput, quality and use. Throughput signals include tests launched per quarter, time from idea to launch and the share of the roadmap delivered. Quality signals include the share of tests that passed QA first time, the number of tests stopped for technical problems, and the share of tests that reached their planned sample. Use signals include the share of winners shipped, the time from read-out to implementation, and how often new ideas cite a previous learning.
Be careful with win rate as a measure. A programme with a very high share of winners may be testing timid changes or calling winners too early; one with very few may be testing without enough evidence. The quarterly review should look at win rate alongside effect sizes and the quality of evidence behind ideas. An outside provider paid by the number of winning tests faces the same pressure, as the guide to hiring and evaluating a CRO agency explains. The conversion rate calculator is useful for quick checks on the baseline figures that feed test designs.
Common mistakes
- Peeking and stopping early. Calling a winner when results first look good inflates false positives. Fix duration and stopping rules before launch.
- Channel owners declaring their own winners. The person who proposed the idea wants it to win. Only the analyst declares results.
- Ideas without evidence. Tests of opinions rarely teach much. Ask for the problem and the evidence at intake.
- Post-hoc segments presented as findings. Slicing results until something looks significant produces noise. Treat unplanned segments as new hypotheses.
- Winners that never ship. A result sitting in a development queue has no value. Track implementation in the log.
- Testing where traffic cannot support it. Tests that cannot reach a readable result waste time. Check feasibility at prioritisation and use research instead where needed.
