Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Experiment Library

Product Page Chat Test

An experiment design for offering chat on product pages: visitors split between chat and the usual contact routes, read on orders per visitor and returns.
Experimentintermediate8 min readUpdated

Hypothesis

If visitors to product pages are offered chat with staff who know the products, orders per product-page visitor will rise, because questions that would end the visit are answered at the moment of choice, unless the answers raise returns or the staff time costs more than the extra orders earn. The decision behind it: offer staffed chat on product pages, or keep the existing contact routes only.

The guide to answering pre-purchase questions treats those questions as a sales channel. It sets who answers, the briefing agents need, what they must not promise and how each question is recorded, and it says plainly that no general figure tells a business how much answering them is worth; a controlled test of a new channel, such as chat on product pages, is a separate piece of work designed for the business's own traffic. This is that design. Chat has a cost the page elements in the library do not: someone has to be there to answer, for every hour the window is open.

Metrics and guardrails

The primary metric is orders, or revenue, per product-page visitor, with each visitor held in one arm across sessions. Every visitor assigned to an arm counts in its denominator, whether they opened chat or not, so the metric measures what offering chat does to the whole audience, not what happens to the minority who use it. Orders per visitor is a conversion rate (CVR); a shop whose order values vary widely can use Revenue per Visitor instead, chosen before launch.

Three guardrails, each with a limit written in advance:

  • Return rate on orders from each arm. An answer that promises more than the product delivers wins the order and loses it again. Read the Return Rate of each arm's orders once the return period has passed.
  • Staff time per extra order. Divide the staff hours spent on chat by the extra orders the treatment arm produced over the control. It is the cost that decides whether a reliable lift is worth keeping, and the limit should come from the margin on an order.
  • Phone and email contacts per visitor. Chat may replace other contacts rather than add sales; a fall here is a service gain, but it is not the sales effect the test is meant to find. A rise suggests chat answers are sending people elsewhere for detail.

Read the Cart Abandonment Rate for the same products alongside, as the pre-purchase guide suggests: chat that removes a last doubt should show up there too.

Design

This is an A/B test at visitor level. Assign each visitor at random at their first product-page view and hold them in their arm across sessions, using a first-party identifier, because buyers of considered products often return several times before ordering.

Design elementChoiceReason
Unit of randomisationVisitor, assigned at the first product-page view, held across sessionsA visitor who sees chat on one visit and not the next blurs both arms
ArmsExisting contact routes only; the same routes plus staffed chat on product pagesThe decision is whether to add the channel
Staffing and hoursFixed in advance and the same across the runChanging either mid-test changes the treatment
Held constantPhone, email, help centre, page content, prices, offers and delivery termsOnly chat differs, so the result belongs to it
Agent briefingThe pre-purchase guide's briefing and approved wordingAnswers must not promise more than the page
Primary metricOrders or revenue per product-page visitorCounts everyone offered chat, not only those who used it

Choosing and staffing a channel is its own work. The guide to support channels and self-service describes live chat as fast and able to rescue a sale, but needing real-time staffing; set the hours to those chat would keep after the test, not a showcase rota. Outside the hours, the chat option should be absent in the treatment arm rather than leaving a form nobody answers.

What agents say is part of the treatment. The pre-purchase guide records that in the UK the Advertising Standards Authority's codes are the reference for marketing claims, and that an agent's answer carries the same risk as the claim on the page: if it promises a result the product cannot deliver, the customer has been misled. Its rules apply in full here: approved wording, no promise the page does not make, say when you do not know, and no pressure. This is general direction, not legal advice.

Count chats properly. The guide to measuring calls and chats separates a chat started from a chat that became an enquiry and explains how to tie a conversation to its source; use the same definitions here, and record the arm on each conversation so returns can be traced back.

Sample size reasoning

Most visitors never open chat, so its effect on orders is diluted across everyone in the arm. The inputs are the current conversion rate on product pages, their traffic and the smallest rise worth paying the staffing for. The sample size and significance guide shows how those combine and how dilution raises the number needed: a real effect on the few who chat becomes a small effect on the arm as a whole. The conversion rate calculator turns each arm's orders and visitors into the rates the guide works with. A team that cannot reach the number can test chat on the range that raises the most questions, where the share of visitors who need an answer is highest.

Duration

Run whole weekly cycles covering the opening hours chat would keep after the test, with staffing fixed in advance and the same across the run. Avoid sales and launches, when the questions people ask change. Then keep the reading open for a further window, until orders from both arms have passed the return period, because the return guardrail is the one that catches answers that overpromise.

Reading the results and the decision rule

Read once, after the return window. Compare orders per visitor between arms with the interval around the difference. Then read the guardrails: the return rate of each arm's orders, staff hours divided by the extra orders, and other contacts per visitor. Look at the chat record last, for what people asked. Write the rule before launch:

ResultDecision
Orders per visitor rise reliably and returns and staff time per extra order stay within their limitsKeep chat on product pages, at the hours tested
Orders per visitor rise but returns pass their limitReview the answers given before deciding; keep the current routes meanwhile
Orders rise but staff time per extra order passes its limitKeep chat only on the range that raised most questions, or not at all
Orders do not move reliablyKeep the current routes and put the recurring answers on the pages

Remember what the test cannot say. It measures chat at the hours and staffing tested; a cheaper rota or a different team is a different treatment.

Pitfalls

Reading only visitors who chatted is the first trap. They were closer to buying before they opened the window, and their conversion rate says nothing about what chat added. Compare the arms whole.

Staffing that changes mid-test is the second. A busy week with slow answers and a quiet week with instant ones are different treatments; fix the rota and log any gaps.

Answers that promise more than the product are the third. A delivery date the business cannot meet or a capability the page does not claim wins an order and loses it to a return or a complaint. Review a sample of transcripts against the briefing every week.

Returns read too early are the fourth. The orders chat wins on the strength of an answer are the ones most likely to come back, so wait for the return window to close.

Two simpler page tests sit beside this one and need no staff: the social proof placement test and the exit-intent popup test. If a recurring question can be answered on the page, the product page optimisation guide sets out where.

Example: an invented sewing machine shop tests chat on its product pages (illustrative)

Illustrative example — not a real result

Sauvrekt Sewing Machines tests staffed chat on its product pages

Sauvrekt Sewing Machines, an invented online sewing machine shop, offers chat with two trained staff to half of its product-page visitors from 9am to 6pm, Monday to Saturday, for six weeks. Machines carry a 30-day trial period, so the reading closes 30 days after the last order. Before launch the team sets two limits: the return rate in the chat arm may not exceed the control's, and staff time may not exceed four hours per extra order.

ArmVisitorsOrdersOrders per visitorReturned in trialPhone and email contacts
Existing routes20,0003801.90%23 (6.1%)180
With chat20,0004442.22%24 (5.4%)130

About 800 visitors in the chat arm open a conversation. Orders per visitor rise by 0.32 points, with an interval of roughly 0.05 to 0.59 points. The 240 staff hours give 64 extra orders, 3.75 hours each, inside the limit, and returns from the chat arm do not rise. Sauvrekt keeps chat at the hours tested and adds the three most-asked answers to its product pages.

Common mistakes

Chat tests usually fail on whom they count. The experiment library lists the other page and offer designs that sit beside this one.

  • Comparing chatters with non-chatters. They differ before chat starts; compare the arms whole.
  • Judging on chats started. A busy chat window is a cost, not a result.
  • Changing hours or staffing mid-test. The treatment is the chat as it will actually run.
  • Letting agents improvise claims. Use the approved wording and never promise beyond the page.
  • Reading before the return period ends. Overpromised orders come back later.
  • Ignoring other contacts. Chat that only replaces phone calls is a service change, not a sales gain.

Frequently asked questions

Why not compare visitors who chatted with visitors who did not?

Because people who open chat are different before they ever type. They are often closer to buying, more uncertain about one detail, or buying something expensive. Comparing them with everyone else measures that difference, not the effect of chat. Randomising visitors before anyone sees the chat option, and comparing everyone in one arm with everyone in the other, is the only reading that isolates what chat adds.

Does the test need a chatbot or a person?

This design tests staff who know the products, because the hypothesis is about answering real questions at the moment of choice. An automated assistant is a different treatment with different risks, and it would need its own test with its own accuracy checks. Whatever answers, it must not promise more than the product page and the terms do.

What if chat does not raise orders?

Then the questions are still worth having. Keep the existing contact routes, take the questions chat recorded during the test, and put the recurring answers on the product pages, where every visitor can find them without asking. A chat channel that does not lift orders can still have taught the business what its pages leave out.

Terms used on this page

Tools to try

Frameworks

Where this page is used

Also related

Sources

  1. Advertising Standards Authority — ASA

Related