Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Glossary term

Learning Log

A single, searchable record of every experiment, losers and inconclusive results included, with the hypothesis, design, result, decision and what it taught.
AnalyticsconceptUpdated

Definition

A learning log is a single, searchable record of the experiments a team has run, kept in a format that lets someone years later find out whether an idea was tried and what happened. The guide to experimentation cadence calls it the programme's memory, and the experimentation programme playbook goes further and makes it the programme's main product: the thing that remains when individual tests, and the people who ran them, are gone.

Two features set it apart from a list of results. It is complete, holding every test whatever the outcome. And it is written for retrieval, with each entry tagged by area and audience, so that a question such as "have we tried social proof on the pricing page?" has an answer that does not depend on who is in the room.

What an entry holds, and why the interpretation matters

The two guides agree on the core fields, in slightly different words:

  • The question and hypothesis, as written before the test, not as remembered after it.
  • Where it applied: the area, audience and stage of the funnel or growth loop it targeted.
  • The design: variants, split, duration and sample.
  • The results on the primary metric and each guardrail, with the confidence and any caveats.
  • The call and its owner: ship, iterate or drop, and who made it.
  • Whether it was implemented. The playbook rates this field highly, because winning tests that never ship are common, and the log is where the gap shows.
  • The interpretation: a sentence or two on what this result taught the team about its customers.

The interpretation is the field that turns a record of outcomes into learning. A result says that a change won, lost or made no reliable difference; it does not say why. A losing test with a clear note often teaches more than a winner, and the cadence guide reviews the whole log each quarter to draw out themes, such as the stages where changes rarely move anything or the kinds of offers customers ignore, which then shape the next quarter's backlog.

In practice: tests, before-and-after changes and weekly meetings

The log is not only for split tests. Where traffic cannot support a controlled test, the guide to conversion work without A/B tests still asks for a written record: changes read before and after, and staged rollouts, go into the same log as the tests, each labelled with the strength of its evidence, so a weaker reading is never mistaken for a randomised one.

Entries are written while the result is fresh. The tutorial on running a weekly growth meeting makes log entries one of the meeting's only outputs and has them written in the log itself, interpretation included, before the hour ends, rather than in meeting notes that nobody searches.

Planning draws on the log too. A learning agenda starts from what the business already knows, including past test results, and gives findings that went against the hypothesis the same care as those that confirmed it, which keeps both kinds in the log for the next planning cycle.

Illustrative example — not a real result

A meal-kit company checks its log before retesting an old idea

Zestaur Meal Kits, a weekly recipe box company, planned a test of a cheaper first box, dropping its introductory price from £34 to £24. A search of the learning log for "first box price" found an entry from 2 years earlier: the same cut had raised first-box orders by 18%, but second-box retention fell from 58% to 47%, and the test was called a loss on the guardrail. The interpretation read: "Price brought in people who wanted a cheap dinner, not a recipe habit." Instead of rerunning it, the team tested the reason. It kept the £34 price and added a choice of 3 starter recipes rated quick to cook, aimed at the hesitation its exit survey kept naming. That test was logged with a link to the old entry.

Every entry starts from a Hypothesis, recorded as it was written before launch, because a hypothesis rewritten after the result can no longer be wrong. The Test Plan fixes the design, metrics and decision rule in advance, and the log keeps it beside the outcome. A Decision Log records decisions of every kind, such as budget moves, paused campaigns and replaced pages, with the reasoning and when the result is due to be checked; the learning log is narrower, covering tests and changes and what each one taught. An A/B Test is the most common source of entries and the strongest evidence a log holds, which is why weaker readings are labelled as such.

Common confusions

  • A learning log is a list of wins. A log of winners cannot stop a team retesting an idea that already lost. Losers and inconclusive results belong in it with the same care.
  • The results report is the log. Each test's read-out is one document; the log is the searchable collection of all of them, with the interpretation pulled out so it can be found.
  • The learning agenda and the learning log are the same. The agenda lists the questions a plan needs answered; the log records what tests and changes actually found.
  • Recording the outcome is enough. Without the interpretation, a future reader knows what happened but not what it meant, and cannot tell whether a new idea is genuinely new.

More terms to look up

Where this term is used

Related