Why analyse a test in analytics at all
An A/B testControlled comparison of two variants. is usually run by a testing tool that assigns each visitor to an arm, applies the change and reports the goal it was configured with. That report answers one question. Analytics answers three more. It shows behaviour the tool was never told about, such as basket additions, later visits and revenue. It gives a second, independent count of the outcome, which is the fastest way to notice that a conversion tag behaves differently in one arm. And through the export it lets the test be joined to the order or lead system, where the outcome the business cares about actually lives.
What analytics does not do is the inference. Explorations will show that one arm converted at a higher rate than the other; they will not say whether that difference is larger than chance. That judgement comes from the method the plan fixed in advance, and Evan Miller's essay on how not to run an A/B test shows why it must be fixed in advance: a team that checks repeatedly and stops at the first good-looking moment will declare winners that do nothing.
| System | What it is for in a test | What it should not be used for |
|---|---|---|
| Testing tool | Assignment, applying the change, its own goal and statistics | The full picture of behaviour in each arm |
| GA4 interface | Exposure counts, the split check, the primary metric and guardrails per arm | Significance; exact counts for a close decision |
| BigQuery export | Exact, unsampled counts per arm and the join to orders or leads | Daily monitoring by people without SQL |
| Order or lead system | The outcome as the business records it | Knowing which arm a customer saw, unless analytics tells it |
Before you start
- A written test planThe written design of a test, fixed before launch: hypothesis, unit of randomisation, primary metric, guardrails, planned sample, duration and decision rule.. Hypothesis, unit of randomisationWhat a test assigns at random to control or variant, such as a visitor, session, account or region; it should match the level at which the metric is counted., primary metric, guardrail metricsA metric an experiment must not worsen beyond an agreed bound, watched beside the primary metric that decides the test., planned sample sizeThe number of users or sessions an experiment needs to detect a given effect reliably., duration and the analysis method, as the A/B testing guide sets out. This tutorial implements the plan; it does not write it.
- Access. Edit and publish rights on the Tag Manager container, and a property role that can create custom definitions and explorations.
- The testing tool's documentation for how it tells the page which arm a visitor received: a callback, a data layer push or a readable value. Follow the vendor's own documentation for this part.
- The tracking plan open, so the exposure event and its parameters are added as a row before anything is built; the events and parameters guide describes the naming rules.
- The BigQuery export linked well before launch if the decision is a large one. The BigQuery export basics guide notes that nothing is backfilled, so a link created mid-test misses the first days.
- Working knowledge of explorations, segments and funnels, from the explorations tutorial.
Agree the metric and its unit before launch
Translate the plan's primary metric into the exact terms analytics will count. Name the numerator (for example, users who purchased after their first exposure), the denominator (users exposed to the arm), the event behind each, whether events or users are counted, and the window. The unit must match the randomisation: if the tool assigns visitors, the analysis counts users, not sessions, because a session-level denominator lets one person count several times. Write the guardrails the same way, and name the method that will turn the counts into a decision, whether that is the testing tool's statistics or a calculator. Put all of it in the test record before launch, so nobody chooses a metric after seeing which one looks best.
Send an exposure event with test and variant
Fire one event when a visitor actually sees an arm, in both arms including the control, carrying a test identifier and a variant identifier. The moment matters: an event fired on assignment, on a page the change does not touch, adds people who never saw anything and dilutes the difference. An exposure eventAn analytics event fired when a visitor sees a test arm, carrying the test and variant, so each arm's behaviour can be counted outside the testing tool. sent when the arm is shown records only people who could have reacted to the change. Have the testing tool's callback push to the data layer, and build a custom event trigger and a GA4 event tag in the container that read both keys as event parametersA named value sent with an analytics event that describes it, such as a page location, form name or item price.. Use short, controlled values agreed in the tracking plan, never anything typed by a visitor.
window.dataLayer = window.dataLayer || []; window.dataLayer.push({ event: "experiment_exposure", experiment_id: "pdp_short_2026_11", variant_id: "b" });Register the variant for reporting
Parameters that are not registered do not appear in reports or explorations, and the events guide explains that registration is not retroactive. Before launch, register
experiment_idandvariant_idas event-scoped custom dimensionsA GA4 reporting field created by registering an event, user or item parameter so it appears in reports and explorations.. Event scope fits because the events guide reserves it for values that describe the event they travel with, and an exposure is exactly that: this person saw this arm at this moment. Reuse the same two parameters for every future test rather than registering new ones each time, which keeps the property inside its documented limits on custom definitions. If a report also needs the arm on later events, a user property can carry it, but it holds one value at a time, so give each test its own name and retire it afterwards.Verify exposure in DebugView
Use the testing tool's preview or forcing option to load each arm in turn, with Tag Manager preview running. In Tag Assistant, confirm the data layer push arrives with both keys and that the GA4 tag fires once on it. In DebugView, confirm the event name and both parameter values for each arm, as the debugging tutorial describes. Then walk to the outcome in each arm and confirm the purchase or lead event fires once with the same parameters as outside the test. The test pitfalls guide lists broken tracking as the commonest cause of a confident wrong answer, and this walk is where it is caught, before any visitor is in the test.
Check the split every day
Build a free-form exploration with
variant_idin rows, filtered to the test'sexperiment_id, with users and the exposure event count as metrics and date as a second dimension. Compare each day's share per arm with the planned split, overall and by device category and browser. A persistent gap is a sample ratio mismatchA persistent gap between the share of visitors an A/B test was meant to assign to each variant and the share each actually received., which the pitfalls guide treats as a defect that removes visitors from one arm selectively, so stop, find the mechanism, fix it and restart. Compare the exposure counts with the testing tool's assignment counts too. This daily look is monitoring, not reading the result: it checks that the experiment is working, and it never looks at the primary metric.Build the variant comparison in explorations
Create one user segment per arm: users who had an
experiment_exposureevent with thisexperiment_idand thisvariant_id. Then build a closed funnel exploration whose first step is the exposure event narrowed to the test by its parameter and whose second step is the agreed outcome, indirectly followed, and apply the two segments as comparisons. A closed funnel counts only people who completed the first step, so a purchase made before the exposure is not counted, which is the trap a plain user segment falls into. Set the date range to the test's run, check the samplingWhen an analytics tool calculates a report from a subset of the data and scales the result up, rather than processing every event. indicator before reading, and name the exploration with the test identifier.Add the guardrail metrics
Each guardrail from the plan gets the same treatment: a defined numerator and denominator, read per arm, from the first exposure onward. Revenue per exposed user comes from a free-form table with the two segments as comparisons and purchase revenue and users as metrics; refunds come from the
refundevent where the shop sends it; engagement and error events come from their own events. Keep the guardrails few and named in advance. A variant that wins the primary metric and breaches a guardrail goes back to the decision makers rather than straight to rollout, which is the rule the A/B testing guide sets.Read the result with the method agreed before launch
When the planned sample is reached, and not before, take the counts per arm from the funnel: users exposed and users who reached the outcome. The conversion rate calculator turns them into rates; the significance judgement comes from the method named in the plan, applied as the sample size and significance guide explains. Read the guardrails alongside. Record the counts, the rates, the decision and any caveat, such as a sampled exploration or a consent gap, in the experiment results report, which also records the check that the variant parameter arrived.
Confirm in the BigQuery export when the decision is large
For a change that is expensive to reverse, recount in the export, which holds every event without the sampling and thresholds the interface applies. Take each user's first exposure to the test, keep the arm they saw, and count outcomes after that timestamp. Users who appear in both arms point at an assignment fault and should be investigated, not quietly dropped. Where the purchase carries a transaction identifier, the same query can join to the order system, which is how returns or cancellations become a guardrail the interface cannot show.
WITH exposures AS ( SELECT user_pseudo_id, (SELECT value.string_value FROM UNNEST(event_params) WHERE key = 'variant_id') AS variant_id, MIN(event_timestamp) AS first_seen FROM `project.analytics_property.events_*` WHERE _TABLE_SUFFIX BETWEEN @start_date AND @end_date AND event_name = 'experiment_exposure' AND (SELECT value.string_value FROM UNNEST(event_params) WHERE key = 'experiment_id') = 'pdp_short_2026_11' GROUP BY user_pseudo_id, variant_id ) SELECT e.variant_id, COUNT(DISTINCT e.user_pseudo_id) AS exposed_users, COUNT(DISTINCT IF(p.event_timestamp > e.first_seen, p.user_pseudo_id, NULL)) AS purchasers FROM exposures e LEFT JOIN `project.analytics_property.events_*` p ON p.user_pseudo_id = e.user_pseudo_id AND p.event_name = 'purchase' AND p._TABLE_SUFFIX BETWEEN @start_date AND @end_date GROUP BY e.variant_id;Close the test and clean up
When the decision is made, end the test in the testing tool and remove the losing arm's code, so the exposure event stops for that test identifier. Keep the event and its two registered parameters for the next test. Rename or archive the explorations and segments with the test identifier and the result, and retire any user property created for this test. Update the tracking plan row with the test's end date. Write the outcome into the results report and the backlog, so the next hypothesis in the experimentation loop starts from what this one learned rather than from memory.
A test read in GA4 (illustrative)
Tidewren Mattresses tests a shorter product page
Tidewren Mattresses, an invented mattress retailer, tests a product page with the specifications collapsed into tabs (arm b) against the current long page (arm a), on a planned even split for four full weeks. The primary metric is users who purchase after their first exposure, divided by users exposed; the guardrail is the share of orders returned, read from the order system through the export.
| Row label | Arm a (control) | Arm b (shorter page) |
|---|---|---|
| Users exposed (GA4) | 18,420 | 18,310 |
| Share of exposed users | 50.1% | 49.9% |
| Users who purchased after exposure | 412 | 455 |
| Order rate | 2.24% | 2.48% |
| Orders returned within the order system's window | 31 (7.5%) | 36 (7.9%) |
| Visitors assigned (testing tool) | 21,690 | 21,560 |
The daily split stayed between 49.6% and 50.4% for arm a, so no sample ratio mismatch. The testing tool counted about 18% more visitors than GA4 in both arms. The team traced the gap to consent: the tool assigned every visitor, while the analytics tag, under the site's basic consent mode implementation, sent nothing for visitors who declined analytics. Because the gap was the same proportion in each arm, the comparison held. At the planned sample the testing tool's statistics, the method named in the plan, judged the difference in order rate significant at the agreed level. The returns guardrail rose by 0.4 points, inside the tolerance the plan had set at one point. An export recount matched the interface to within a few users, and the shorter page was rolled out with a note to watch returns for two further months.
When analytics and the testing tool disagree
Two systems counting the same test will never agree exactly, and they do not need to. What matters is that any gap is the same in both arms. The table lists the usual causes.
| What you see | Likely cause | What to check |
|---|---|---|
| Analytics shows fewer people in both arms by a similar share | Consent choices or blockers stop the analytics tag for some visitors | Consent configuration; the gap's share per arm |
| Analytics shows more outcomes than the tool | The tool counts its goal inside its own window or session; analytics counts every later outcome | Each system's definition of a conversion |
| The gap differs between arms | A tag, trigger or page element behaves differently in one arm | Walk the outcome path in each arm in preview |
| Exposure counts in analytics exceed the tool's assignments | Exposure fires on every view rather than once, or on pages outside the test | Trigger conditions; count users, not events |
| Rates differ although counts look close | Different denominators, users in one system and sessions in the other | The metric definition written in the first step |
| Interface and export disagree slightly | Estimated user counts and thresholds in the interface | Treat the export as the exact record |
Common mistakes
- Firing the exposure event on assignment. People who never saw the change dilute the difference until it disappears.
- Sending the event from the variant only. Without a control exposure there is no denominator for the control arm.
- Registering the dimension after launch. Registration is not retroactive in the interface; the first days of the test are missing from every exploration.
- A plain user segment for the outcome. Purchases made before the exposure are counted; a closed funnel or a sequence condition fixes it.
- Reading the primary metric daily. Checking the split is monitoring; deciding from the primary metric before the planned sample is the peeking that inflates false winners.
- Treating a difference in rates as a result. Explorations supply counts; the agreed method supplies the judgement.
- Session denominators for a visitor-level test. One person counted several times in one arm.
- Ignoring the sampling icon. A sampled exploration is a poor basis for a close call; recount in the export.
- Leaving finished tests running in the container. Old exposure events clutter the next test's analysis.
The GA4 hub places this tutorial among the other GA4 guides, and the explorations tutorial covers the funnel and segment techniques used here in more depth.
