Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Glossary term

Flicker

The brief flash of the original version before a browser-side test script applies the variant, which turns a result about the change into one about the fault.
AnalyticsconceptUpdated

Definition

Flicker, also called the flicker effect or a flash of original content, is the original version of a page showing briefly before a browser-side test applies its variant. It follows from how such tests work. The page arrives as normal, and a script then decides which version this visitor gets and edits the page to match, sometimes after rendering is already under way. When that edit lands late, the original is on screen first.

The A/B testing guide describes it as a short flash of the control ahead of the variant, and makes looking for it part of the quality check before a test goes live, on the devices that carry the traffic, slowest included.

A test assigned on the server cannot flicker, because the server builds the assigned version before the page is sent. Flicker is therefore a risk of where a test runs, though among browser tests some changes, such as blocks moved after the page renders, are more likely to flash than others.

Why it matters to a result

Visitors in the control arm see an ordinary page. Visitors in the variant arm may watch the page change in front of them, which looks like something going wrong. That difference reaches one arm only, so the comparison is no longer a clean test of the change. Whether the flash helps or hurts the variant, the result no longer isolates what was changed.

For reading results, the consequence is simple: an outcome gathered while the variant flickered cannot be pulled apart from the flicker, and where it was common during a run, the guides' answer is to remove the cause and run the test again. How to check for it on a slow device, and which remedies exist, are questions for the testing guides.

In practice: where a test runs, hypotheses, shipping and headline tests

The guide to deciding where a test runs counts flicker among the costs of a browser test, beside the wait a page-hiding script adds and winners left outside the site's code. A headline or image change on a campaign page may run in the browser once it has been loaded on a slow phone without the original showing; anything the server calculates, and any speed test, runs on the server.

When a hypothesis is written up, flicker belongs among its technical risks, as the guide to writing hypotheses lists them, so the check is planned rather than remembered.

After a win, the tutorial on shipping a winning variant rebuilds the change in the site's own templates, partly because a testing script left serving it can keep flickering, and removes any page-hiding snippet that was there only for the test.

The landing page headline test design asks the tester to watch for the control headline appearing for a moment before the challenger, because that flash lands on the challenger's arm.

Illustrative example — not a real result

A mattress retailer's mobile win disappears on the server

Baddoox Mattresses, an invented online mattress retailer, ran a landing page headline test through a browser-side script for 3 weeks, with about 21,000 visitors per arm. On desktop the two headlines converted at the same rate, 3.1%. On mobile the challenger converted at 2.4% against 2.0% for the control, and the team was ready to ship it. First, the analyst loaded the page on a mid-range phone with its connection slowed and saw the control headline on screen for about 0.5 seconds before the challenger replaced it. Because that flash appeared only on slower phones and only in the challenger arm, the mobile result could not be credited to the headline. The test was rebuilt with server-side assignment and rerun for 4 weeks: on mobile both headlines converted at 2.1%, and the challenger was not shipped.

An A/B Test is the comparison flicker spoils, and flicker is a fault of the browser-side way of serving one. A Sample Ratio Mismatch is another fault that leaves the arms unequal, found in the visitor counts rather than on screen. A Novelty Effect is a real reaction to something new that fades as people get used to it; flicker is a reaction to a fault, and it lasts as long as the script keeps flashing. The Test Plan is fixed before launch, so the flicker check and the choice of where the test runs belong before launch too. Scripts that hide the page until they decide can stop the flash, but the wait they add falls on every visit and shows up in loading measures such as Core Web Vitals, not in the test result.

Common confusions

  • Flicker is cosmetic. It changes what the variant arm experiences, so it changes the result.
  • Hiding the page until the script decides removes the cost. It can remove the flash, but every visitor waits, control included, and the comparison never shows that wait.
  • A server-side test can flicker too. The server sends the assigned version complete, so there is no original to flash.
  • Flicker means any flashing on a page. Here the word means only the original showing before a browser-side test applies its variant; flashing effects in accessibility guidance are a separate subject.

More terms to look up

Where this term is used

Related