Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Playbooks

Experimentation Programme Playbook

A playbook for running a marketing experimentation programme: idea intake, prioritisation, test design, QA, reading results and sharing what was learned.
Playbookadvanced9 min readUpdated

What this playbook covers and when to run it

The playbook covers the governance and operation of a marketing experimentation programme across websites, landing pages, email, paid media and product-led marketing. It runs from an idea arriving to a decision being made and recorded. It does not teach how to calculate sample size, how to write a hypothesis or how to design an incrementality study; the sample size and significance guide, the experiment hypothesis template and the incrementality testing guide own those.

Run it when tests are happening in more than one place and nobody can say what was learned last quarter, when results are disputed after the fact, or when the team wants to scale testing beyond the occasional A/B Test on a landing page. It needs a minimum of traffic or audience to be worth running formally; where volumes are very low, many marketing questions are better answered by research than by split tests.

The playbook is part of the playbooks library. It builds on the experimentation cadence guide, which explains why rhythm matters for growth teams, and on the experimentation loop framework, which describes the cycle this playbook operationalises. To make sure the backlog serves the year's plan rather than whatever is easiest to test, the tutorial on turning plan assumptions into a learning agenda ranks the assumptions the plan rests on and sends only the ones that need an experiment into this programme.

Roles and responsibilities

  • Experimentation lead. Owns the programme: intake, the roadmap, prioritisation, the stand-up and the learning log. Has the authority to stop a test that breaks the rules.
  • Analyst. Owns test design quality and results: metrics, sample size, duration, stopping rules, the analysis and the read-out. Signs off that a result is reliable.
  • Developer. Owns the build: implementing variants, targeting, tracking of test events, and the technical QA before launch.
  • Designer. Owns the variant experience: designs that express the hypothesis clearly and work across devices.
  • Channel owner. Owns the area being tested, whether a page, a journey or a campaign. Brings ideas, approves tests in their area and implements winners.

The analyst's sign-off is the programme's quality gate. If channel owners can declare their own winners, the programme will slowly fill with false positives that do not hold when shipped. Agree at the outset that only the analyst declares a result.

Cadence, inputs and outputs

CadenceOwnerInputsOutputs
Continuous intakeExperimentation leadIdeas from channel owners, research, analytics, customer feedbackIdeas logged with a problem statement and evidence
Weekly stand-upExperimentation lead, developer, analystTests in design, build, QA and liveLaunch approvals, QA sign-offs, live tests checked for problems
Fortnightly read-outsAnalyst, channel ownersFinished tests with full dataRead-outs, decisions to ship, iterate or drop; learning log entries
Quarterly roadmap reviewExperimentation leadLearning log, backlog, win rate, business prioritiesRe-prioritised roadmap, process changes

Keep the weekly stand-up and the fortnightly read-out separate. The stand-up is about whether tests are healthy and ready; the read-out is about what finished tests mean. Mixing them invites people to discuss live results before a test has finished. A smaller growth team that has to merge them can follow the tutorial on running a weekly growth meeting, which checks live tests for health only and calls finished tests against their decision rules.

The playbook in phases

  1. Intake ideas

    Every idea enters through one door: a simple form or backlog that asks for the problem, the evidence for it, the proposed change, the audience affected and the metric it should move. Evidence can be analytics, user research, customer feedback or a previous test; the Nielsen Norman Group publishes extensive usability research that is often a useful starting point for website ideas. Ideas without a problem statement go back to the submitter. The experimentation lead reviews intake weekly, merges duplicates and asks for missing evidence. Encourage ideas from everywhere, including sales and support, because the people closest to customers often see problems analytics cannot. Every idea, accepted or not, stays in the log.

  2. Prioritise

    Score the backlog using a consistent framework; the prioritising tests guide compares the common ones. Whatever model is used, it should weigh the expected impact on the primary metric, the strength of the evidence, the reach of the change and the effort to build and run it. Check feasibility early: some ideas cannot reach a readable result in a reasonable time with the available traffic, and those are better answered another way. The experimentation lead builds a roadmap from the top of the list, avoiding tests that would overlap on the same audience and page. Share the roadmap so channel owners know when their ideas will run.

  3. Design and QA

    For each test, the analyst and the channel owner write the design using the hypothesis template: the hypothesis, the primary metric, guardrail metrics, the audience, the variants, the expected effect size, the required sample, the duration and the stopping rule. The duration should cover full business cycles, usually at least whole weeks. The designer and developer build the variants. QA checks that variants render correctly across devices and browsers, that users are assigned randomly and consistently, that test events record, and that nothing else on the page changed. The weekly stand-up approves the launch only when design and QA are both signed off. The A/B test launch checklist expands this phase into the lines to tick for each test, from the stopping rule to the first day's traffic split.

  4. Run and monitor

    Once launched, a test runs for its planned duration without changes to the variants, the targeting or the traffic sources. The developer and analyst check in the first days for problems: a sample ratio that differs from the planned split, a variant that errors on a browser, tracking that stopped. Problems like these can justify stopping and restarting a test; an encouraging early result cannot. The weekly stand-up reviews every live test for health only, not for results. Resist pressure to peek and call a winner. If a business event, such as a large promotion, disrupts the test, note it and decide with the analyst whether the data is still usable.

  5. Read out and log learnings

    When a test reaches its planned end, the analyst analyses it against the design: the primary metric, the guardrails and any segments that were specified in advance. Segments found after the fact are hypotheses for the next test, not results. The read-out states the result, the confidence, what it means for the hypothesis and the decision: ship, iterate or drop. The channel owner commits to the decision with a date. The experimentation lead adds the test to the learning log, linked to its design and read-out. The experiment results report template gives the read-out a fixed format, from the sample check to the decision and the learning. For tests that measure channel Incrementality, the read-out also notes how the result should change budget or reporting.

The learning log and how decisions follow

The learning log is the programme's memory and its main product. Each entry records the question, the hypothesis, the design, the result, the decision and whether the decision was carried out. Over time it answers questions like "have we tested social proof on the pricing page?" and "what do we know about how our audience responds to urgency?" without anyone relying on memory.

FieldWhy it matters
Question and hypothesisLets future readers judge whether a new idea is genuinely new
Area and audienceShows where the learning applies and where it might not
Primary metric and resultRecords what moved, usually a CVR or a downstream measure
Confidence and caveatsShows how much weight the result can bear
Decision and ownerMakes clear what was supposed to happen next
Implemented?Closes the loop between learning and action

The "implemented" field matters most. Programmes often generate winning tests that never ship because the change sat in a development queue behind other work. At each fortnightly read-out, review the open decisions from previous read-outs. At each quarterly review, report the share of winning tests that have shipped. A learning that never changes anything is an expensive curiosity.

Illustrative example — not a real result

A subscription meal-kit company building its log

A meal-kit subscription company has run about 40 tests over two years across its site, emails and paid social, with results scattered across slide decks. The new experimentation lead rebuilds a learning log from what can be recovered and finds that 9 tests were declared winners from early peeks and that 6 genuine winners were never implemented. The team introduces analyst sign-off and a fortnightly read-out that reviews open decisions. In the next two quarters it runs 18 tests; 5 win, 4 lose clearly and 9 show no reliable difference. All 5 winners ship within a month, and one losing test on discount framing stops a planned sitewide change.

Measuring the playbook

Measure the programme on throughput, quality and use. Throughput signals include tests launched per quarter, time from idea to launch and the share of the roadmap delivered. Quality signals include the share of tests that passed QA first time, the number of tests stopped for technical problems, and the share of tests that reached their planned sample. Use signals include the share of winners shipped, the time from read-out to implementation, and how often new ideas cite a previous learning.

Be careful with win rate as a measure. A programme with a very high share of winners may be testing timid changes or calling winners too early; one with very few may be testing without enough evidence. The quarterly review should look at win rate alongside effect sizes and the quality of evidence behind ideas. An outside provider paid by the number of winning tests faces the same pressure, as the guide to hiring and evaluating a CRO agency explains. The conversion rate calculator is useful for quick checks on the baseline figures that feed test designs.

Common mistakes

  • Peeking and stopping early. Calling a winner when results first look good inflates false positives. Fix duration and stopping rules before launch.
  • Channel owners declaring their own winners. The person who proposed the idea wants it to win. Only the analyst declares results.
  • Ideas without evidence. Tests of opinions rarely teach much. Ask for the problem and the evidence at intake.
  • Post-hoc segments presented as findings. Slicing results until something looks significant produces noise. Treat unplanned segments as new hypotheses.
  • Winners that never ship. A result sitting in a development queue has no value. Track implementation in the log.
  • Testing where traffic cannot support it. Tests that cannot reach a readable result waste time. Check feasibility at prioritisation and use research instead where needed.

Frequently asked questions

How many tests should a programme run at once?

As many as the traffic, the team and the QA process can support without tests interfering with each other. Running more tests than the team can design and read properly produces inconclusive results and false wins. Start with the number the team can QA thoroughly each week, and grow from there.

What counts as a successful test?

A test that produced a reliable answer to its question, whether the variant won, lost or made no difference. A clear loss teaches as much as a win and prevents a bad change from shipping. The programme should be judged on the quality and use of its learnings, not only on its share of winners.

Can a test be stopped early if the result looks clear?

Only if the stopping rule was set in advance and allows it. Peeking at results and stopping when they look good is one of the most common causes of false positives. The design phase fixes the duration and the stopping rule before launch, and the run phase respects them.

Terms used on this page

Tools to try

Frameworks

Where this page is used

Part of these learning paths

Sources

  1. Nielsen Norman Group articles — Nielsen Norman Group

Related