Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Collection

Marketing Experiment Library

A library of experiment designs that are ready to adapt and run, each fixing the hypothesis, the metric that decides it, the guardrails that must hold and the way to read the result. The designs cover the pages, offers and channels marketing teams test most often, such as landing pages, forms, pricing pages, email, ad creative, bidding, budgets, remarketing, referrals, offers and promotions, apps, bookings, product feeds, quotes, direct mail, loyalty programmes, chat assistants and upgrade prompts, and each one links to the guide that owns the underlying method.

Looking for how to use this library? Read the guide below the list.

All 60 pages in this collection

What these experiment designs are for

Most marketing tests fail before they start. The idea is sound, but nobody wrote down which metric would settle it, the test is read on the third day because the dashboard looked exciting, and the "winner" is rolled out with no check on whether lead quality or margin quietly fell. A design fixes those decisions in advance, while nobody has a stake in the answer. That is the whole point of these pages.

Each design answers one recurring question a team asks. Does a sharper headline convert more of the right visitors? Does a shorter form bring more qualified leads, or just more leads? Does extra budget buy extra sales, or only sales the business would have had anyway? The pages assume you already know how an A/B Test works and why significance matters; they do not re-teach the statistics. What they add is the specific shape of each test: the unit that gets randomised, the guardrails that matter for that kind of change and the traps that are particular to it.

The designs also sit between two other kinds of page on the site. The experiments resource page is the short reference for the ideas behind testing, and the templates library holds the blank hypothesis document you fill in for your own test. A design here is the worked middle step: a filled-in plan for a common situation that you copy into the template and adjust to your traffic, your buying cycle and your definitions.

How each design is structured

Every design follows the same order, so you can compare two tests side by side or hand one to a colleague without explanation. It opens with the hypothesis, written as a change, an expected effect and a reason, because a hypothesis without a reason teaches you nothing when it fails. It then names one primary metric, the number that decides the test, and a short list of guardrails, the numbers that must not deteriorate for the change to ship.

The design section states the unit of randomisation (such as a visitor, recipient, user, customer, quote, product or region), the variants, and the reasoning behind the duration, usually full weekly cycles or a full buying cycle. A table summarises those choices. The sample size section explains, in words, what drives the number of people or regions needed and points to the sample size and significance guide for the arithmetic; there is deliberately no figure on any page, because the right figure depends on your baseline CVR, your traffic and the smallest effect worth acting on.

The last sections are the ones teams skip. "Reading the results" says what to look at, in what order and what to do with a flat or mixed outcome. The pitfalls section names the failure modes particular to that test, such as inflated email opens or regions that leak into each other. Common mistakes closes each page. The structure follows the experimentation loop: hypothesis, design, run, read, decide and record.

Which experiment when

Start from the decision you are trying to make, not from the test you have seen others run. If the question is whether a channel causes sales at all, a page-level test will not answer it. The table routes the common situations to the design that fits.

SituationRun this designWhy this one
Paid traffic lands but few visitors convert, and the ad promise differs from the pageLanding page headline testTests message match with the cheapest change on the page
Forms collect plenty of leads but sales reject many, or the form feels longLead form length testJudges fields on qualified leads, not raw submissions
Visitors reach pricing but ask sales to explain the plansPricing page layout testTests presentation without touching the prices
Open rates look healthy but clicks are weakEmail subject line testReads subject lines on clicks, not inflated opens
A regular campaign goes out at a fixed time chosen long agoEmail send time testHolds content constant and repeats across sends
Video ads get views but acquisition cost is highAd creative hook testChanges only the opening seconds of the edit
Responsive search ads are rated well but conversions lagSearch ad asset testReads assets on conversions per impression
You want to move to automated bidding but cannot trust a platform experimentBidding strategy geo split testCompares matched regions at equal spend
Leadership asks whether more budget will buy more salesBudget increase incrementality testMeasures marginal cost, not platform-reported returns
A shop debates where to set the free shipping thresholdFree shipping threshold testJudges on margin per visitor, not order value
New sign-ups receive a fixed onboarding sequence and many never activateOnboarding email cadence testRandomises at sign-up and reads on activation
Remarketing claims a large share of sales and you doubt itRemarketing holdout testHolds out a random group to measure Incrementality
You need to choose who a referral programme rewards: the referring customer, the friend or bothReferral reward testReads each reward structure on referral margin per invited customer
The team debates whether the pricing page should show annual or monthly billing firstBilling period default testKeeps prices and plans identical and reads contracted revenue per visitor
New app users meet an account wall before they reach any valueDeferred account creation testAssigns each install at first open and reads activation by install cohort
Prospecting campaigns keep reaching people who are already customersExisting customer exclusion testCompares matched regions on cost per new customer from your own records
Product pages raise questions shoppers cannot get answered before buyingProduct page chat testCounts everyone in each arm, not only those who chatted
Quotes go out and then go quietQuote follow-up testAssigns each quote when it is sent and reads acceptance by send-month cohort
The mobile site pushes visitors into the app and nobody knows whether it helpsWeb-to-app switch testReads orders across web and app combined, not installs
Shoppers will not risk a full size of a product they have never triedTrial size product testReads margin per first-time visitor after the cost of the trial sizes
Someone asks how much branding performance ads should carryBrand cue ad testSplits matched regions so each runs only one version of every ad
Customers have learned to wait for a code before they buyFull-price protection testReads gross profit per customer across several promotion cycles
Shopping feed titles were copied from the page titlesShopping feed title testRetitles half a category and reads each half against its own pre-period
Booked appointments are often missedBooking deposit testReads attended appointments, not bookings made
A sign-up source brings addresses of unclear qualityDouble opt-in testReads email revenue per form submission, not per subscriber
The team must choose which piece to mail a customer segmentDirect mail format testCompares a catalogue, a postcard and no mail on net incremental margin
A loyalty programme is proposed with no evidence it changes behaviourLoyalty programme pilot testInvites customers at random or holds them out and reads incremental margin after reward costs
Someone proposes letting an AI assistant answer product questions in chatAI chat assistant testReads margin per chat visitor against staff-only chat, not chats handled
An existing customer reaches a plan limit and meets a block that only says to upgradeUpgrade prompt testAssigns each account when it first reaches the limit and reads upgrades beside downgrades, refunds and cancellations

The pages beneath this introduction are listed automatically, so new designs appear here as they are published. If two rows fit, run the test whose wrong answer would cost more first.

Experiments and the guides behind them

A design is only as good as the method under it, and each method has its own home. The testing guides own the mechanics of split tests: randomisation, significance, stopping rules and the pitfalls common to every page test. The growth marketing guides own the cadence and the backlog: how many tests a team can run, how to prioritise them and how to keep a learning log that outlives the people who ran the tests. Attribution and incrementality guides own the theory behind holdouts and geo tests. The designs here link to those pages rather than repeating them.

Some questions are not experiments at all. Choosing between two platforms, two pricing models or two measurement approaches is often better answered by a structured comparison, and the comparison library sets those out side by side. Use an experiment when you can change one thing, split the audience fairly and wait long enough to see a difference; use a comparison when the decision is strategic and cannot be split.

When a test finishes, the result belongs in the hypothesis document and the learning log, whatever it says. A flat result is information. It tells you the change was not worth making at the effect size you could detect, and that frees the next slot in the backlog for a bolder idea.

Common experiment problems

Most failed tests fail for one of a handful of reasons, and each has a page that explains the fix. Find the symptom you recognise and follow the link. One false win worth knowing by name is the novelty effect, an early lift that fades once returning users get used to the change.

ProblemWhat usually went wrongWhere to look
The test "won" but nothing improved after rolloutRead too early, or a novelty effectA/B testing guide
Results swing every time you lookToo little traffic for the effect sizeSample size and significance
Leads rose but sales complainedRaw submissions used as the metricLead form length test
Platform reports a lift your sales data does not showAttribution credit mistaken for causationRemarketing holdout test
Nobody remembers why a past test was runNo written hypothesisExperiment hypothesis template