What a conversion optimisation provider does
A conversion agency, consultant or freelancer sells some or all of the loop the experimentation hub describes: research that finds where and why visitors leave, hypotheses written from that research, a prioritised backlog, controlled experiments planned and built, results read and recorded, and recommendations for changes that should simply ship. Some providers also design and build pages; some only research and advise; some run an A/B testControlled comparison of two variants. programme on a retainer. The conversion research hub and the testing guides set out what good practice looks like at each stage, and they are the standard to hold any provider to, whatever it calls its method.
No provider can settle what the business is trying to achieve. What counts as a conversion, what an order or a qualified lead is worth, which harms a change must not cause and what is allowed to ship are business decisions. A provider can advise on all of them, but the answers belong to the people who will live with the consequences after the provider has gone.
This guide adds what is specific to conversion work. Shortlists, pitches, fee models, contracts, service levels and exits for any agency are covered in the guide to choosing, briefing and reviewing agencies, and whether to hire at all is the subject of the in-house versus agency comparison. The hiring guide for SEO agencies and the hiring guide for Google Ads agencies do the same job for search, and their principles carry across: the business owns what it pays for, defines the work before signing up to it, and judges the provider on results it can check for itself. Where the work reaches beyond page and funnel tests into activation, retention and acquisition economics, the guide to hiring and evaluating a growth marketing agency adds what stays with the business for the model, the definitions and the log.
What the business keeps: accounts, research and the test log
The agencies guide puts the ad and analytics accounts, customer data and paid-for creative in the business's hands. Conversion work adds several assets to that list, and each is worth settling before any proposal is compared, as general practice.
| Asset | Why the business keeps it | What goes wrong otherwise |
|---|---|---|
| Analytics property and tag container | Every test and every report depends on the same counting | A provider who rebuilds tracking in its own setup leaves no history behind |
| Testing tool account | Running and past tests, their settings and their data stay available | Leaving the provider means losing every test and starting again |
| Recording, heatmap and poll accounts | The research evidence can be re-read by the next team | Findings survive only as slides, without the sessions behind them |
| The research backlog | Problems, locations, evidence and decisions in one place | The next provider repeats research the business has already paid for |
| The test log | Hypotheses, plans, results and decisions for every test | Nobody can say what was tried, at what sample, or why it shipped |
| The code of shipped winners | Changes live in the site's own templates | Improvements disappear when a testing script is switched off |
The accounts are created in the business's name and the provider is given access, at the lowest level that does the job, with the business's own administrators kept in place. The access review tutorial for marketing tools keeps an access register that includes agency and freelancer access, so that access can be withdrawn person by person when the team or the arrangement changes.
The test log deserves particular care. The experimentation programme playbook calls the Learning LogA single, searchable record of every experiment, losers and inconclusive results included, with the hypothesis, design, result, decision and what it taught. the programme's memory: one entry per test, from the question and hypothesis through the design and result to the decision and whether it was acted on. A provider that keeps that record in its own systems is keeping the most valuable thing the business paid for.
Research brings personal data with it. The recordings and heatmaps tutorial handles a recording tool the way it handles every other tag: it loads only under the consent decision the site has made, every input is masked by default, and the tutorial points out that the GDPR, one of this page's sources, gives personal data a broad definition. A Session RecordingA reconstruction of one visitor's visit, replaying their clicks, scrolling, taps and form interactions, used to see how a page is actually used. that captures what people type is personal data nobody needed, and the guide to keeping personal data out of analytics shows where else it leaks. Who may see recordings, how long they are kept and what the contract says about them are decisions for the business's data protection lead, not for the provider or the marketing team alone.
Questions to ask before hiring
What counts is how specific each answer is. A careful provider answers with how it works on sites like yours; a careless one answers with reassurance or with results from somebody else's site.
| Question | What a careful answer contains |
|---|---|
| Who will do the research, the analysis and the build, and how much of their time do we get? | Named people and roles, including who reads results and who writes the code for variants |
| How do you decide what to test? | Research before tests, with each backlog entry located on a page and step and seen by more than one method |
| How do you size a test, and what happens when our traffic cannot support one? | A baseline from our own analytics, the smallest effect worth shipping, a sample fixed before launch, and research, fixes or bolder changes where the arithmetic rules a test out |
| When do you read a test, and who declares the result? | Once, at the planned sample, against a stopping rule written before launch, by someone other than the person who proposed the idea |
| How will results be reported? | The plan quoted, the primary metric as a range, every guardrail, and inconclusive and losing tests listed alongside winners |
| How does a winner reach the site? | Rebuilt in our own code by our developers or theirs, with the testing script used only as a short bridge |
| Whose accounts will the work run in? | Ours, with access granted to the provider |
| How will you handle recordings and personal data? | Our consent decision respected, inputs masked, access limited and agreed with our data protection lead |
| What happens when we part? | The accounts, the backlog, the test log and the research stay with us; scripts and variants are removed in an agreed order |
The sizing question separates providers more quickly than any other. The prioritising tests guide makes the effect a test can detect with the traffic available the binding limit on any backlog, and the guide to sample size and significance shows why a page with little traffic cannot detect a small change, however sound the reasoning behind it. A provider who promises a steady stream of tests to a site with few conversions each week is promising tests that cannot answer their questions. The honest answer for such a site is the one given in the guide to improving conversion without A/B tests: fix what is broken, research the rest, test bold changes or whole templates where the traffic allows, and read shipped changes against a prediction written in advance.
The reading question is the second filter. Evan Miller's essay, listed in the sources, shows that a test watched continually and stopped at the first significant-looking result will declare winners among changes that do nothing far more often than the significance level promises. The A/B testing guide turns that into procedure: the metric, the sample size and the decision rule are all set before launch, and the result is read a single time. A provider who reads results daily and calls them when they look good will report more winners than one who does not, and fewer of them will be real.
Warning signs
- Accounts in the provider's name. A testing, recording or analytics account the business does not own means the history leaves with the provider.
- Promised uplift before the site has been read. Nobody can say in advance which changes will help a particular site, which is the point the conversion rate optimisation pillar makes about tricks; a figure in a proposal is a sales line.
- A fee tied to the number of winning tests. The playbook warns that win rate is a poor measure on its own, and Goodhart's law describes what happens to any measure once it becomes a target. Paid per winner, a provider is paid to call results early and to test changes too timid to lose.
- Tests without research. A plan that starts with a list of changes rather than with where visitors leave is testing opinions, which the experimentation playbook warns rarely teaches much.
- No written plan before launch. Without one, nobody can tell later whether the metric, the audience or the end date moved after the data arrived.
- Few inconclusive results. The A/B testing guide notes that most results in most programmes are inconclusive. A report that shows mostly winners is reporting selectively or reading early.
- Segments presented as wins. A test that lost overall but "won on mobile returning visitors" has found a hypothesis, not a result, as the testing pitfalls guide explains.
- Single figures added into an annual claim. Point estimates from several tests, summed and projected over a year, describe nothing the business will see.
- Winners left in the testing tool. A change served indefinitely by a script has not shipped.
The diagnosis for glowing agency reports beside flat sales traces how several of these signs work together in any agency relationship: the agency's own figures in the report, credit claimed for outcomes the business would have had anyway, and a fee that grows with activity.
Scoping the opening piece of work
It helps to tell four kinds of work apart, since each fits a different situation and leaves something different behind.
| Scope | What it delivers | When it fits |
|---|---|---|
| Tracking check and expert review | Verified conversion counts and a list of issues across a journey, each with the evidence that would confirm it | Any site, and always before tests are planned |
| Research sprint | A ranked backlog with evidence from several methods for one funnel and one question | Before a testing programme, or when results have stalled |
| Testing programme | Plans, builds, quality checks, read-outs and the log, on a rhythm | Once the tracking is trusted, a backlog exists and the traffic can support tests |
| Fixes and page builds | Changes that should simply ship, built into the site | When research finds faults rather than open questions |
Start with the first or the second. The expert review tutorial and the research sprint tutorial show what good work of each kind contains, and the landing page audit scores a single page where that is the job. A provider who wants to start testing before anyone has checked that the purchase or lead event fires once is skipping the step that most often turns a test into a confident wrong answer.
The agency request for proposal template puts the brief, the questions and the scoring criteria into one document, so that every provider answers the same RFPA document sent to shortlisted agencies or sellers giving each the same facts and questions and the published scoring criteria, so responses compare.. Score the answers with a weighted scoring matrix, with the weights agreed before anyone reads a proposal, and treat ownership of the accounts as a condition rather than a weighted criterion. Once a provider is chosen, the agency scope of work template records deliverables, exclusions, access, reviews and exit, and the agency onboarding checklist covers the first agreements before anyone touches a live site. A RetainerA standing fee paid regularly to an agency or supplier for continuous work with a stable scope, as distinct from project fees, time and materials or commission. makes sense for a testing programme with a stable rhythm; a fixed fee suits a review or a sprint with a defined output.
Reports you can check
A test report is useful only if the business can verify it. Five tests apply to every read-out:
- It quotes the plan. Hypothesis, primary metric, guardrails, planned sample and reading date, from a document written before launch, with the date it was written. The A/B test launch checklist is a fair standard for what that plan contains.
- It states the result as a range. The primary metric is reported with its Confidence IntervalA range of values for an effect, such as a lift in conversion rate, that is consistent with the data at a chosen confidence level., against the smallest effect worth shipping, so that a win reads as "somewhere between" two values rather than as one headline figure.
- It reports every guardrail. Each Guardrail MetricA metric an experiment must not worsen beyond an agreed bound, watched beside the primary metric that decides the test. named in the plan appears with its result, including the ones that moved the wrong way.
- It lists every test. Winners, losers and inconclusive tests appear in the same report, with the sample each reached and the decision taken.
- It can be recounted. The counts behind each arm can be checked independently. The GA4 tutorial for A/B test analysis records an exposure event in the business's own analytics so that each arm's conversions can be counted outside the testing tool and set beside the order system or the CRM.
The experiment results report template sets out a one-page ReadoutThe meeting or short document where a study's or test's results are presented against the decision it was run for, ending in a recorded decision. in that order, with the interval ahead of the point estimate. Ask for it, or something equivalent, as the standard format for every test the provider runs. The conversion rate calculator turns any arm's counts back into a rate, which is a quick check that the figures in a report agree with the counts beneath them.
Getting winners into the site
A winning test settles a question, but the site is still unchanged. The tutorial for shipping a winning variant covers what has to happen after the read-out for the gain to last: the decision confirmed against the plan, the variant rebuilt in the site's code, device and tracking checks, an entry in the release log and a watch on the metric. Agree before the work starts who does each of those steps.
The question matters because of what happens when nobody owns the rebuild. A testing script left serving a winner adds weight to every page, can briefly display the original before swapping in the new one and takes the browser's attention away from the site's own code, and the change sits outside the codebase, out of the next developer's sight. The testing tool can carry a winner for a short time while the rebuild is scheduled, provided the scope names the date it stops.
The playbook treats the "implemented" field as the most important in its log, since winners commonly wait in the development queue while other work goes first. The share of winners that actually shipped is the measure to ask a provider for, and to report alongside the results themselves.
Judging the work once tests have run
Judge the process early and the outcomes later. In the opening weeks, check what is within the provider's control from the start: access granted in the business's accounts, the tracking verified against the business's own records, research with its evidence attached, plans written before launch and quality checks done before tests count.
Outcomes take longer. A test cannot be judged before it reaches its planned sample, and a programme cannot be judged before several tests have been read and their winners shipped. When that point arrives, look at four things together:
- The quality of the evidence behind each test. The research hub judges a programme by how many hypotheses rest on evidence and how many of those moved the metric the way they predicted.
- How results were read. Tests run to their planned samples, results declared by the person the plan named, and inconclusive results recorded rather than rerun until they win.
- What shipped. The share of winners rebuilt in the site and the time each took, from the log.
- What the business saw. Orders, revenue or qualified leads over comparable periods, read by source and device rather than as one overall figure. Reported wins that never appear in those figures need explaining, and the usual explanations are tests read early, single figures summed, novelty that faded or winners that never shipped.
Be careful with win rate as a verdict in either direction. The playbook warns that a programme winning almost every test may be choosing timid changes or declaring results early, while one that rarely wins may be testing ideas with too little evidence behind them. On a site with little traffic, the honest provider spends more time on research and fixes than on tests, and the output to judge is the backlog and the fixes shipped.
Changing or ending the arrangement
Order matters here, and the agency offboarding checklist works through its groups by risk: ownership, then assets and knowledge, then continuity decisions, then access removal, with the close-out of invoices and the handover note after it. For conversion work that means confirming the business holds administrator access to every account; collecting the test log, every plan and read-out, the research backlog and the evidence behind it; deciding whether each running test is read at its planned sample or stopped and recorded as stopped; rebuilding any winner still served by the testing script; and only then removing the outgoing provider's access, person by person. The incoming provider starts from the log and the backlog rather than from a blank page.
A sewing supplies retailer compares proposals (illustrative)
Wimbercoth Sewing Supplies chooses conversion help
Wimbercoth Sewing Supplies, an invented online retailer of sewing machines, fabric and haberdashery, takes about 9,000 orders a month from roughly 410,000 sessions. It sends the same request for proposal to three providers and scores each answer from 1 to 5 against weights agreed in advance.
| Criterion (weight) | Provider A (agency) | Provider B (agency) | Provider C (freelancer) |
|---|---|---|---|
| Accounts and test log (condition) | Fails: tests run in its own account | Passes: works in the retailer's accounts | Passes |
| Research before testing (25%) | 2: a list of changes from other clients | 5: tracking check and research sprint first | 4: expert review |
| Test sizing and reading (25%) | 2: eight tests a month, read weekly | 5: sample fixed before launch, read once | 3: not asked to test |
| Reporting (20%) | 2: lifts as single figures | 5: ranges, guardrails, every test listed | 3: written findings |
| Shipping winners (15%) | 1: winners left in its script | 4: rebuilt with the retailer's developer | 2: not covered |
| Fee basis (15%) | 1: retainer plus £600 per winning test | 4: fixed fee for the sprint, then a retainer | 4: fixed fee |
| Weighted score out of 5 | 1.70 | 4.70 | 3.25 |
Ownership rules Provider A out before any score is added, and the score merely agrees. It buys a six-week research sprint from Provider B for £14,000. The sprint finds the purchase event firing twice on the payment provider's return page, which had put the reported conversion rate at 2.6% against 2.2% in the order system, and a delivery charge first shown at the last checkout step. The tracking is fixed and the delivery charge moved to the basket as a fix without a test. Over the next quarter two tests run to plan: a fabric-finder change on category pages ends inconclusive, with an interval from minus 3% to plus 5%, and is logged; a rewritten machine-comparison table wins, with an interval of plus 2% to plus 9% on orders per visitor and no change in returns, and is rebuilt in the site's code within three weeks. Orders per visitor over the quarter, read against the same weeks a year earlier, rise by about 4%.
Common mistakes
- Scoring proposals before ownership is settled. A low bid that keeps tests in the provider's own account turns out the dearest when the relationship ends.
- Letting the provider define success. What counts as a conversion, what it is worth and which guardrails apply are business decisions.
- Starting with a testing retainer. Testing before the tracking is checked and the research is done buys activity, not answers.
- Paying per winner. The measure becomes the target, and the results stop meaning what they say.
- Accepting single-figure results. Ask for the range, the guardrails and the inconclusive tests every time.
- Leaving winners in the testing tool. Agree who rebuilds them, and by when, before the first test runs.
- Judging on reported wins alone. Set the provider's figures beside the business's own orders or leads.
- Planning the exit at the end. Agree at the start what the business keeps and in what order access is removed.
