Definition
A novelty effect is a short-lived difference between a changed experience and the old one that comes from the change itself rather than from its merit. Its cause is attention. A regular visitor has learned where things are, and when something moves, changes colour or appears for the first time, it stands out against that memory. Standing out earns clicks, opens and interactions, and in a test those count exactly like the behaviour the change was meant to produce.
The effect fades because familiarity returns. Once the new element is part of the expected page, it competes on what it actually offers. If that is no better than before, the variant's lead disappears; if it is better, a smaller, steadier lead remains. The novelty was never the result, only noise with a direction.
Why returning users drive it
New visitors have nothing to compare against. To someone arriving for the first time, the variant is simply the page, so nothing about it is novel. Returning users carry the old version in their heads, and only they can react to the difference. That asymmetry is what makes the effect detectable: a lift that appears among returning users and not among new ones, and that shrinks over time, is the signature.
The opposite reaction also happens. Some changes force regular users to relearn where things are, and they perform worse at first before adapting. That is usually called a primacy effect or change aversion, and it makes a good change look bad early. The guide to common testing pitfalls describes both shapes and why each argues for running long enough for the trend to flatten.
In practice: effect over time and new versus returning users
Two checks expose the effect, and both work on data any test already collects.
- Plot the lift by day or week across the run. A real effect wobbles around a level. A novelty effect starts high and slopes down. A cumulative chart hides the slope, so the daily or weekly difference is the one to read.
- Split the result into new and returning visitors. If the lift exists only among returning users, treat it with suspicion; if new users show a similar lift, the change is probably doing something real. The new visitor result is generally the better guide to the long-run effect, because it is free of the comparison that drives novelty.
Neither check works on a test stopped early. A run long enough to see whether the early lift holds is part of the design, which is why the A/B testing guide fixes duration in the plan rather than leaving it to whoever looks at the dashboard first. In email and lifecycle tests the same idea appears as reading at fixed cohort ages: the onboarding email cadence test waits for every enrolled user's window to close so that early excitement in the first cohorts does not decide the result.
Novelty effect versus a real lift
A real improvement and a novelty effect can produce identical numbers in the first days. What separates them is persistence and who shows the gain. A change that removes friction, clarifies an offer or answers an objection tends to help new and returning users alike and to keep helping. A change whose only advantage is that it looks different tends to help returning users briefly.
The difference also shows after launch. When a rolled-out winner drifts back to the old baseline, the guide to testing and scaling performance campaigns treats that reversion as a finding in itself: either the test was misread or conditions changed, and early, large wins are the ones most likely to have been novelty. The experimentation section lists novelty among the standard reasons a test can mislead, and the marketing experiment library points to it when a test won but nothing improved after rollout.
A recipe box redesign that won for one week
Brackenholt Recipe Box tested a redesigned weekly menu page against the old one, measuring the share of visitors who added a recipe to their box. After the first week the variant led by 14% relative, and almost all of the lift came from returning customers, who make up about 70% of visits. The team kept the test running as planned. The weekly lift fell to 8% in week two, 3% in week three and 1% in week four, and among new customers it was close to zero throughout. The final four-week result was a lift of about 6% with an interval that included zero. The redesign was kept for unrelated design reasons but recorded as having no measured effect on additions, and the forecast that assumed the early lift was withdrawn.
Related terms
An A/B TestControlled comparison of two variants. is where the novelty effect does its damage, by inflating the variant early in the run. Statistical SignificanceA test result unlikely enough under 'no real difference' to pass a threshold chosen in advance. does not protect against it, because a novelty lift is a real difference in behaviour while it lasts. Sample SizeThe number of users or sessions an experiment needs to detect a given effect reliably. calculations fix how long a test runs, which is usually long enough to see the fade if the test is not stopped early. A Holdout GroupA randomly chosen part of an audience kept from exposure to measure what exposure caused. kept after launch shows whether the gain persisted once the change stopped being new.
Common confusions
- An early lift is evidence of a winner. Early leads are the ones most often inflated by novelty.
- Significance rules it out. A significant result can still be a temporary one.
- The effect only affects design changes. New subject lines, new messages and new offers can all draw attention for being new.
- Running longer always cures it. Only if new visitors arrive and the trend is read; a cumulative figure can still carry the early spike.
- Novelty and primacy are the same. One flatters a change early, the other penalises it; both fade.
