What these experiment designs are for
Most marketing tests fail before they start. The idea is sound, but nobody wrote down which metric would settle it, the test is read on the third day because the dashboard looked exciting, and the "winner" is rolled out with no check on whether lead quality or margin quietly fell. A design fixes those decisions in advance, while nobody has a stake in the answer. That is the whole point of these pages.
Each design answers one recurring question a team asks. Does a sharper headline convert more of the right visitors? Does a shorter form bring more qualified leads, or just more leads? Does extra budget buy extra sales, or only sales the business would have had anyway? The pages assume you already know how an A/B TestControlled comparison of two variants. works and why significanceA test result unlikely enough under 'no real difference' to pass a threshold chosen in advance. matters; they do not re-teach the statistics. What they add is the specific shape of each test: the unit that gets randomisedWhat a test assigns at random to control or variant, such as a visitor, session, account or region; it should match the level at which the metric is counted., the guardrails that matter for that kind of change and the traps that are particular to it.
The designs also sit between two other kinds of page on the site. The experiments resource page is the short reference for the ideas behind testing, and the templates library holds the blank hypothesis document you fill in for your own test. A design here is the worked middle step: a filled-in plan for a common situation that you copy into the template and adjust to your traffic, your buying cycle and your definitions.
How each design is structured
Every design follows the same order, so you can compare two tests side by side or hand one to a colleague without explanation. It opens with the hypothesis, written as a change, an expected effect and a reason, because a hypothesis without a reason teaches you nothing when it fails. It then names one primary metric, the number that decides the test, and a short list of guardrails, the numbers that must not deteriorate for the change to ship.
The design section states the unit of randomisation (such as a visitor, recipient, user, customer, quote, product or region), the variants, and the reasoning behind the duration, usually full weekly cycles or a full buying cycle. A table summarises those choices. The sample size section explains, in words, what drives the number of people or regions needed and points to the sample size and significance guide for the arithmetic; there is deliberately no figure on any page, because the right figure depends on your baseline CVRShare of visits or clicks that convert., your traffic and the smallest effect worth acting on.
The last sections are the ones teams skip. "Reading the results" says what to look at, in what order and what to do with a flat or mixed outcome. The pitfalls section names the failure modes particular to that test, such as inflated email opens or regions that leak into each other. Common mistakes closes each page. The structure follows the experimentation loop: hypothesis, design, run, read, decide and record.
Which experiment when
Start from the decision you are trying to make, not from the test you have seen others run. If the question is whether a channel causes sales at all, a page-level test will not answer it. The table routes the common situations to the design that fits.
| Situation | Run this design | Why this one |
|---|---|---|
| Paid traffic lands but few visitors convert, and the ad promise differs from the page | Landing page headline test | Tests message match with the cheapest change on the page |
| Forms collect plenty of leads but sales reject many, or the form feels long | Lead form length test | Judges fields on qualified leads, not raw submissions |
| Visitors reach pricing but ask sales to explain the plans | Pricing page layout test | Tests presentation without touching the prices |
| Open rates look healthy but clicks are weak | Email subject line test | Reads subject lines on clicks, not inflated opens |
| A regular campaign goes out at a fixed time chosen long ago | Email send time test | Holds content constant and repeats across sends |
| Video ads get views but acquisition cost is high | Ad creative hook test | Changes only the opening seconds of the edit |
| Responsive search ads are rated well but conversions lag | Search ad asset test | Reads assets on conversions per impression |
| You want to move to automated bidding but cannot trust a platform experiment | Bidding strategy geo split test | Compares matched regions at equal spend |
| Leadership asks whether more budget will buy more sales | Budget increase incrementality test | Measures marginal cost, not platform-reported returns |
| A shop debates where to set the free shipping threshold | Free shipping threshold test | Judges on margin per visitor, not order value |
| New sign-ups receive a fixed onboarding sequence and many never activate | Onboarding email cadence test | Randomises at sign-up and reads on activation |
| Remarketing claims a large share of sales and you doubt it | Remarketing holdout test | Holds out a random group to measure IncrementalityConversions caused by marketing. |
| You need to choose who a referral programme rewards: the referring customer, the friend or both | Referral reward test | Reads each reward structure on referral margin per invited customer |
| The team debates whether the pricing page should show annual or monthly billing first | Billing period default test | Keeps prices and plans identical and reads contracted revenue per visitor |
| New app users meet an account wall before they reach any value | Deferred account creation test | Assigns each install at first open and reads activation by install cohort |
| Prospecting campaigns keep reaching people who are already customers | Existing customer exclusion test | Compares matched regions on cost per new customer from your own records |
| Product pages raise questions shoppers cannot get answered before buying | Product page chat test | Counts everyone in each arm, not only those who chatted |
| Quotes go out and then go quiet | Quote follow-up test | Assigns each quote when it is sent and reads acceptance by send-month cohort |
| The mobile site pushes visitors into the app and nobody knows whether it helps | Web-to-app switch test | Reads orders across web and app combined, not installs |
| Shoppers will not risk a full size of a product they have never tried | Trial size product test | Reads margin per first-time visitor after the cost of the trial sizes |
| Someone asks how much branding performance ads should carry | Brand cue ad test | Splits matched regions so each runs only one version of every ad |
| Customers have learned to wait for a code before they buy | Full-price protection test | Reads gross profit per customer across several promotion cycles |
| Shopping feed titles were copied from the page titles | Shopping feed title test | Retitles half a category and reads each half against its own pre-period |
| Booked appointments are often missed | Booking deposit test | Reads attended appointments, not bookings made |
| A sign-up source brings addresses of unclear quality | Double opt-in test | Reads email revenue per form submission, not per subscriber |
| The team must choose which piece to mail a customer segment | Direct mail format test | Compares a catalogue, a postcard and no mail on net incremental margin |
| A loyalty programme is proposed with no evidence it changes behaviour | Loyalty programme pilot test | Invites customers at random or holds them out and reads incremental margin after reward costs |
| Someone proposes letting an AI assistant answer product questions in chat | AI chat assistant test | Reads margin per chat visitor against staff-only chat, not chats handled |
| An existing customer reaches a plan limit and meets a block that only says to upgrade | Upgrade prompt test | Assigns each account when it first reaches the limit and reads upgrades beside downgrades, refunds and cancellations |
The pages beneath this introduction are listed automatically, so new designs appear here as they are published. If two rows fit, run the test whose wrong answer would cost more first.
Experiments and the guides behind them
A design is only as good as the method under it, and each method has its own home. The testing guides own the mechanics of split tests: randomisation, significance, stopping rules and the pitfalls common to every page test. The growth marketing guides own the cadence and the backlog: how many tests a team can run, how to prioritise them and how to keep a learning log that outlives the people who ran the tests. Attribution and incrementality guides own the theory behind holdouts and geo tests. The designs here link to those pages rather than repeating them.
Some questions are not experiments at all. Choosing between two platforms, two pricing models or two measurement approaches is often better answered by a structured comparison, and the comparison library sets those out side by side. Use an experiment when you can change one thing, split the audience fairly and wait long enough to see a difference; use a comparison when the decision is strategic and cannot be split.
When a test finishes, the result belongs in the hypothesis document and the learning log, whatever it says. A flat result is information. It tells you the change was not worth making at the effect size you could detect, and that frees the next slot in the backlog for a bolder idea.
Common experiment problems
Most failed tests fail for one of a handful of reasons, and each has a page that explains the fix. Find the symptom you recognise and follow the link. One false win worth knowing by name is the novelty effectA temporary change in behaviour caused by a change being new, which fades as users get used to it., an early lift that fades once returning users get used to the change.
| Problem | What usually went wrong | Where to look |
|---|---|---|
| The test "won" but nothing improved after rollout | Read too early, or a novelty effect | A/B testing guide |
| Results swing every time you look | Too little traffic for the effect size | Sample size and significance |
| Leads rose but sales complained | Raw submissions used as the metric | Lead form length test |
| Platform reports a lift your sales data does not show | Attribution credit mistaken for causation | Remarketing holdout test |
| Nobody remembers why a past test was run | No written hypothesis | Experiment hypothesis template |
