Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Marketing Frameworks

The Experimentation Loop and ICE

The experimentation loop as a framework: research, hypothesis, prioritisation with ICE, test, learn, the artefacts at each step, and a worked cycle.
Frameworkintermediate7 min readUpdated

What the loop is

Most teams test things. Fewer run a loop. The difference is that a loop closes: the result of one test changes what the next research looks for, so the programme accumulates knowledge about customers rather than a list of winners and losers. The loop has five steps. Research gathers evidence about where and why people fail to do what the site or campaign wants. Hypothesis turns an observation into a falsifiable statement about a change and its effect. Prioritisation orders the candidate hypotheses against each other, which is where ICE comes in. Test runs the experiment, typically an A/B Test, to a pre-agreed sample and decision rule. Learn records what happened, what it means and what the team now believes, and feeds the next round of research.

What the loop makes visible is the quality of the thinking behind each test. A hypothesis that cannot be traced to a research observation is a guess; a result with no written learning is a number nobody will remember by next quarter. What the loop hides, if a team is not careful, is the opportunity cost: a loop can run smoothly for a year on trivial tests while the biggest problems stay untouched, because they were never in the backlog. The CRO hub covers the research methods and the testing practice in depth; this page is the shape that connects them.

The steps and artefacts

Each step produces one artefact, and the artefacts are how a manager can tell whether the loop is running or the team is only shipping. The table lists them.

StepQuestion it answersArtefactWhat a weak artefact looks like
ResearchWhere and why do people fail, hesitate or leave?Insight log: observation, source, segment, dateA list of ideas with no observation attached
HypothesisIf we change this, what will happen and why?Hypothesis card: change, expected effect on a named KPI, rationale, success metric, guardrails"Let's try a new headline"
PrioritiseWhich hypothesis first, given what it could be worth and what it costs?Scored backlog with ICE values and the reason for each scoreScores with no rationale, or everything scored high
TestDoes the change produce the effect?Test spec: unit, allocation, metric, sample size, run length, QA checklist; then the test itselfA test started without a sample size
LearnWhat do we now believe, and what do we test next?Result write-up: outcome, interpretation, decision, follow-up researchA dashboard screenshot and the word "winner"

The hypothesis card deserves the most attention because it links two ends of the loop. It names the research observation it comes from, which keeps the backlog honest, and it names the metric and the minimum effect that would count, which keeps the test honest. The A/B testing guide covers the test spec in detail: choosing the unit and metric, randomisation, QA and running to the planned sample.

ICE scoring and its limits

ICE scores each hypothesis on three dimensions. Impact is the size of the effect if the hypothesis is right, usually judged by how much of the audience the change touches and how close it sits to the conversion. Confidence is how strong the evidence behind the hypothesis is: a change that addresses a pain seen in session recordings, surveys and support tickets scores higher than one that came from a competitor's site. Ease is the inverse of effort: design, build, QA and any legal or brand sign-off. Each is scored on a short scale, the scores are combined, usually by multiplying, and the backlog is sorted.

ICE's value is speed and shared language. Its limits are plain and worth stating. The scores are subjective, so two people score the same idea differently, and teams drift towards scoring their own ideas high. Confidence inflates over time as people learn that high confidence gets tests run. Ease biases the backlog towards small, cheap changes, which is fine for a while and then produces a year of button colours. The product of three rough scores carries false precision; a score of twenty-four is not meaningfully different from twenty. And ICE says nothing about the hard constraint on any test, which is whether the traffic can detect the effect the hypothesis predicts in a sensible time; the prioritising tests guide treats that constraint, along with risk and dependencies, as the filter that runs after scoring. A sound practice is to use ICE to sort, then to check the top of the list against detectable effect and against the question of whether the biggest problem from research is represented at all.

Worked cycle

The scenario below follows one hypothesis through every step and artefact. Scores and results are invented to show the shape of the loop, not to suggest typical values.

Illustrative example — not a real result

One cycle on a software pricing page

Research: session recordings and a short exit survey on a software company's pricing page showed that 38% of visitors scrolled to the plan comparison and left, and 14 of 40 survey responses mentioned not knowing which plan fitted a small team. Hypothesis: adding a "recommended for teams of 2 to 10" label on the middle plan will raise clicks from the pricing page to sign-up from 11% to at least 13%, because it resolves the choice the survey described. ICE: impact 8 (every pricing visitor, one step from conversion), confidence 7 (two research sources agree), ease 9 (a label and a short test), score 504, the top of a backlog of nine. Test spec: 50/50 split by visitor, primary metric pricing-to-sign-up click rate, guardrail sign-up completion rate, sample size 9,200 visitors per variant for the minimum effect, estimated run of three weeks. Result after the planned sample: 11.2% versus 13.9%, with the guardrail unchanged. Write-up: the label worked, the team now believes plan-choice anxiety is a general problem, and the next research step is to interview trial users about how they chose. The team logged the two-week delay caused by a legal review of the word "recommended" as an ease lesson for future scoring.

Running the loop as a cadence

A loop that runs once is a project; the framework only pays off when it runs on a schedule. A practical cadence has a weekly review of the backlog and running tests, a fortnightly or monthly research session that adds to the insight log, and a quarterly look at what the programme has learned and whether the biggest problems are being addressed. The experimentation cadence guide sets out how teams of different sizes schedule this. The cadence also protects the tests themselves. The cited piece on how not to run an A/B test explains why checking a result repeatedly and stopping as soon as it looks significant produces false winners; a fixed review rhythm with a pre-agreed sample size removes the temptation.

The loop also needs a place in the wider measurement system. Each hypothesis should name which stage of the funnel it serves, and the AARRR framework is a common way to tag a backlog so that a programme does not spend a year on acquisition tests while activation leaks. Over quarters, the measure of the loop is not the win rate; it is whether the team's beliefs about customers have changed and whether the CVR at the stages tested has moved in the direction the learning predicted.

Lead tests are easy to win on submissions, so the experimentation loop for lead generation judges them on qualified leads and pipeline, with a second readout after the sales cycle. Because retention builds slowly, the experimentation loop for retention relies on persistent holdouts, fixed cohort-age readouts and leading metrics proven to predict staying.

Common mistakes

  • Hypotheses without research. An idea with no observation behind it is a guess wearing a card.
  • Scoring ICE in private. Scores need a stated rationale and a second opinion, or confidence inflates and ease dominates.
  • Letting ease run the backlog. A year of small wins can leave the largest problem untested.
  • Peeking and stopping early. Decide the sample and the rule before launch, and review on the cadence, not on the dashboard.
  • Recording winners, not learning. The write-up is the artefact that changes the next research; the uplift is a footnote.
  • Running the loop as a project. Without a cadence the backlog goes stale and the insight log stops.

Frequently asked questions

What is the difference between ICE and PIE or RICE scoring?

They are variations on the same idea of scoring ideas on a few dimensions and sorting. PIE uses potential, importance and ease; RICE adds reach as a separate dimension and divides by effort rather than multiplying by ease. All of them share the same limits: subjective scores, false precision and no account of whether the traffic can detect the predicted effect, so the choice between them matters less than scoring with a stated rationale.

How many tests should a team run at once?

As many as the traffic can power without the tests interfering with each other, which for most sites is one or two on any given page or flow. Tests on different pages can overlap if their metrics are independent; tests on the same funnel step should run in sequence or as a designed multivariate test. Running more than the traffic supports produces tests that never reach their sample size.

What should happen to a test that loses?

The same as one that wins: a write-up that records the outcome, what the team now believes and what it will research next. A loss is evidence that the hypothesis, or the research behind it, was wrong in a specific way, and that is often more informative than a win. Teams that bury losses relearn the same lesson a year later.

Terms used on this page

Applied to an objective

Where this page is used

Part of these learning paths

Also related

Sources

  1. How Not To Run an A/B Test — Evan Miller

Related