Definition
Flicker, also called the flicker effect or a flash of original content, is the original version of a page showing briefly before a browser-side test applies its variant. It follows from how such tests work. The page arrives as normal, and a script then decides which version this visitor gets and edits the page to match, sometimes after rendering is already under way. When that edit lands late, the original is on screen first.
The A/B testing guide describes it as a short flash of the control ahead of the variant, and makes looking for it part of the quality check before a test goes live, on the devices that carry the traffic, slowest included.
A test assigned on the server cannot flicker, because the server builds the assigned version before the page is sent. Flicker is therefore a risk of where a test runs, though among browser tests some changes, such as blocks moved after the page renders, are more likely to flash than others.
Why it matters to a result
Visitors in the control arm see an ordinary page. Visitors in the variant arm may watch the page change in front of them, which looks like something going wrong. That difference reaches one arm only, so the comparison is no longer a clean test of the change. Whether the flash helps or hurts the variant, the result no longer isolates what was changed.
For reading results, the consequence is simple: an outcome gathered while the variant flickered cannot be pulled apart from the flicker, and where it was common during a run, the guides' answer is to remove the cause and run the test again. How to check for it on a slow device, and which remedies exist, are questions for the testing guides.
In practice: where a test runs, hypotheses, shipping and headline tests
The guide to deciding where a test runs counts flicker among the costs of a browser test, beside the wait a page-hiding script adds and winners left outside the site's code. A headline or image change on a campaign page may run in the browser once it has been loaded on a slow phone without the original showing; anything the server calculates, and any speed test, runs on the server.
When a hypothesis is written up, flicker belongs among its technical risks, as the guide to writing hypotheses lists them, so the check is planned rather than remembered.
After a win, the tutorial on shipping a winning variant rebuilds the change in the site's own templates, partly because a testing script left serving it can keep flickering, and removes any page-hiding snippet that was there only for the test.
The landing page headline test design asks the tester to watch for the control headline appearing for a moment before the challenger, because that flash lands on the challenger's arm.
A mattress retailer's mobile win disappears on the server
Baddoox Mattresses, an invented online mattress retailer, ran a landing page headline test through a browser-side script for 3 weeks, with about 21,000 visitors per arm. On desktop the two headlines converted at the same rate, 3.1%. On mobile the challenger converted at 2.4% against 2.0% for the control, and the team was ready to ship it. First, the analyst loaded the page on a mid-range phone with its connection slowed and saw the control headline on screen for about 0.5 seconds before the challenger replaced it. Because that flash appeared only on slower phones and only in the challenger arm, the mobile result could not be credited to the headline. The test was rebuilt with server-side assignment and rerun for 4 weeks: on mobile both headlines converted at 2.1%, and the challenger was not shipped.
Related terms
An A/B TestControlled comparison of two variants. is the comparison flicker spoils, and flicker is a fault of the browser-side way of serving one. A Sample Ratio MismatchA persistent gap between the share of visitors an A/B test was meant to assign to each variant and the share each actually received. is another fault that leaves the arms unequal, found in the visitor counts rather than on screen. A Novelty EffectA temporary change in behaviour caused by a change being new, which fades as users get used to it. is a real reaction to something new that fades as people get used to it; flicker is a reaction to a fault, and it lasts as long as the script keeps flashing. The Test PlanThe written design of a test, fixed before launch: hypothesis, unit of randomisation, primary metric, guardrails, planned sample, duration and decision rule. is fixed before launch, so the flicker check and the choice of where the test runs belong before launch too. Scripts that hide the page until they decide can stop the flash, but the wait they add falls on every visit and shows up in loading measures such as Core Web VitalsGoogle's user experience metrics., not in the test result.
Common confusions
- Flicker is cosmetic. It changes what the variant arm experiences, so it changes the result.
- Hiding the page until the script decides removes the cost. It can remove the flash, but every visitor waits, control included, and the comparison never shows that wait.
- A server-side test can flicker too. The server sends the assigned version complete, so there is no original to flash.
- Flicker means any flashing on a page. Here the word means only the original showing before a browser-side test applies its variant; flashing effects in accessibility guidance are a separate subject.
