What the loop is
Most teams test things. Fewer run a loop. The difference is that a loop closes: the result of one test changes what the next research looks for, so the programme accumulates knowledge about customers rather than a list of winners and losers. The loop has five steps. Research gathers evidence about where and why people fail to do what the site or campaign wants. Hypothesis turns an observation into a falsifiable statement about a change and its effect. Prioritisation orders the candidate hypotheses against each other, which is where ICE comes in. Test runs the experiment, typically an A/B TestControlled comparison of two variants., to a pre-agreed sample and decision rule. Learn records what happened, what it means and what the team now believes, and feeds the next round of research.
What the loop makes visible is the quality of the thinking behind each test. A hypothesis that cannot be traced to a research observation is a guess; a result with no written learning is a number nobody will remember by next quarter. What the loop hides, if a team is not careful, is the opportunity cost: a loop can run smoothly for a year on trivial tests while the biggest problems stay untouched, because they were never in the backlog. The CRO hub covers the research methods and the testing practice in depth; this page is the shape that connects them.
The steps and artefacts
Each step produces one artefact, and the artefacts are how a manager can tell whether the loop is running or the team is only shipping. The table lists them.
| Step | Question it answers | Artefact | What a weak artefact looks like |
|---|---|---|---|
| Research | Where and why do people fail, hesitate or leave? | Insight log: observation, source, segment, date | A list of ideas with no observation attached |
| Hypothesis | If we change this, what will happen and why? | Hypothesis card: change, expected effect on a named KPIA metric tied directly to an objective., rationale, success metric, guardrails | "Let's try a new headline" |
| Prioritise | Which hypothesis first, given what it could be worth and what it costs? | Scored backlog with ICE values and the reason for each score | Scores with no rationale, or everything scored high |
| Test | Does the change produce the effect? | Test spec: unit, allocation, metric, sample size, run length, QA checklist; then the test itself | A test started without a sample size |
| Learn | What do we now believe, and what do we test next? | Result write-up: outcome, interpretation, decision, follow-up research | A dashboard screenshot and the word "winner" |
The hypothesis card deserves the most attention because it links two ends of the loop. It names the research observation it comes from, which keeps the backlog honest, and it names the metric and the minimum effect that would count, which keeps the test honest. The A/B testing guide covers the test spec in detail: choosing the unit and metric, randomisation, QA and running to the planned sample.
ICE scoring and its limits
ICE scores each hypothesis on three dimensions. Impact is the size of the effect if the hypothesis is right, usually judged by how much of the audience the change touches and how close it sits to the conversion. Confidence is how strong the evidence behind the hypothesis is: a change that addresses a pain seen in session recordingsA reconstruction of one visitor's visit, replaying their clicks, scrolling, taps and form interactions, used to see how a page is actually used., surveys and support tickets scores higher than one that came from a competitor's site. Ease is the inverse of effort: design, build, QA and any legal or brand sign-off. Each is scored on a short scale, the scores are combined, usually by multiplying, and the backlog is sorted.
ICE's value is speed and shared language. Its limits are plain and worth stating. The scores are subjective, so two people score the same idea differently, and teams drift towards scoring their own ideas high. Confidence inflates over time as people learn that high confidence gets tests run. Ease biases the backlog towards small, cheap changes, which is fine for a while and then produces a year of button colours. The product of three rough scores carries false precision; a score of twenty-four is not meaningfully different from twenty. And ICE says nothing about the hard constraint on any test, which is whether the traffic can detect the effect the hypothesis predicts in a sensible time; the prioritising tests guide treats that constraint, along with risk and dependencies, as the filter that runs after scoring. A sound practice is to use ICE to sort, then to check the top of the list against detectable effect and against the question of whether the biggest problem from research is represented at all.
Worked cycle
The scenario below follows one hypothesis through every step and artefact. Scores and results are invented to show the shape of the loop, not to suggest typical values.
One cycle on a software pricing page
Research: session recordings and a short exit survey on a software company's pricing page showed that 38% of visitors scrolled to the plan comparison and left, and 14 of 40 survey responses mentioned not knowing which plan fitted a small team. Hypothesis: adding a "recommended for teams of 2 to 10" label on the middle plan will raise clicks from the pricing page to sign-up from 11% to at least 13%, because it resolves the choice the survey described. ICE: impact 8 (every pricing visitor, one step from conversion), confidence 7 (two research sources agree), ease 9 (a label and a short test), score 504, the top of a backlog of nine. Test spec: 50/50 split by visitor, primary metric pricing-to-sign-up click rate, guardrail sign-up completion rate, sample size 9,200 visitors per variant for the minimum effect, estimated run of three weeks. Result after the planned sample: 11.2% versus 13.9%, with the guardrail unchanged. Write-up: the label worked, the team now believes plan-choice anxiety is a general problem, and the next research step is to interview trial users about how they chose. The team logged the two-week delay caused by a legal review of the word "recommended" as an ease lesson for future scoring.
Running the loop as a cadence
A loop that runs once is a project; the framework only pays off when it runs on a schedule. A practical cadence has a weekly review of the backlog and running tests, a fortnightly or monthly research session that adds to the insight log, and a quarterly look at what the programme has learned and whether the biggest problems are being addressed. The experimentation cadence guide sets out how teams of different sizes schedule this. The cadence also protects the tests themselves. The cited piece on how not to run an A/B test explains why checking a result repeatedly and stopping as soon as it looks significant produces false winners; a fixed review rhythm with a pre-agreed sample sizeThe number of users or sessions an experiment needs to detect a given effect reliably. removes the temptation.
The loop also needs a place in the wider measurement system. Each hypothesis should name which stage of the funnel it serves, and the AARRR framework is a common way to tag a backlog so that a programme does not spend a year on acquisition tests while activation leaks. Over quarters, the measure of the loop is not the win rate; it is whether the team's beliefs about customers have changed and whether the CVRShare of visits or clicks that convert. at the stages tested has moved in the direction the learning predicted.
Lead tests are easy to win on submissions, so the experimentation loop for lead generation judges them on qualified leads and pipeline, with a second readout after the sales cycle. Because retention builds slowly, the experimentation loop for retention relies on persistent holdouts, fixed cohort-age readouts and leading metrics proven to predict staying.
Common mistakes
- Hypotheses without research. An idea with no observation behind it is a guess wearing a card.
- Scoring ICE in private. Scores need a stated rationale and a second opinion, or confidence inflates and ease dominates.
- Letting ease run the backlog. A year of small wins can leave the largest problem untested.
- Peeking and stopping early. Decide the sample and the rule before launch, and review on the cadence, not on the dashboard.
- Recording winners, not learning. The write-up is the artefact that changes the next research; the uplift is a footnote.
- Running the loop as a project. Without a cadence the backlog goes stale and the insight log stops.
