Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Conversion Rate Optimisation › Experimentation

Hiring and Evaluating a CRO Agency

How to hire and judge a conversion optimisation agency or consultant: what the business keeps, questions to ask, warning signs, honest reports and shipping.
Guideintermediate16 min readUpdated

What a conversion optimisation provider does

A conversion agency, consultant or freelancer sells some or all of the loop the experimentation hub describes: research that finds where and why visitors leave, hypotheses written from that research, a prioritised backlog, controlled experiments planned and built, results read and recorded, and recommendations for changes that should simply ship. Some providers also design and build pages; some only research and advise; some run an A/B test programme on a retainer. The conversion research hub and the testing guides set out what good practice looks like at each stage, and they are the standard to hold any provider to, whatever it calls its method.

No provider can settle what the business is trying to achieve. What counts as a conversion, what an order or a qualified lead is worth, which harms a change must not cause and what is allowed to ship are business decisions. A provider can advise on all of them, but the answers belong to the people who will live with the consequences after the provider has gone.

This guide adds what is specific to conversion work. Shortlists, pitches, fee models, contracts, service levels and exits for any agency are covered in the guide to choosing, briefing and reviewing agencies, and whether to hire at all is the subject of the in-house versus agency comparison. The hiring guide for SEO agencies and the hiring guide for Google Ads agencies do the same job for search, and their principles carry across: the business owns what it pays for, defines the work before signing up to it, and judges the provider on results it can check for itself. Where the work reaches beyond page and funnel tests into activation, retention and acquisition economics, the guide to hiring and evaluating a growth marketing agency adds what stays with the business for the model, the definitions and the log.

What the business keeps: accounts, research and the test log

The agencies guide puts the ad and analytics accounts, customer data and paid-for creative in the business's hands. Conversion work adds several assets to that list, and each is worth settling before any proposal is compared, as general practice.

AssetWhy the business keeps itWhat goes wrong otherwise
Analytics property and tag containerEvery test and every report depends on the same countingA provider who rebuilds tracking in its own setup leaves no history behind
Testing tool accountRunning and past tests, their settings and their data stay availableLeaving the provider means losing every test and starting again
Recording, heatmap and poll accountsThe research evidence can be re-read by the next teamFindings survive only as slides, without the sessions behind them
The research backlogProblems, locations, evidence and decisions in one placeThe next provider repeats research the business has already paid for
The test logHypotheses, plans, results and decisions for every testNobody can say what was tried, at what sample, or why it shipped
The code of shipped winnersChanges live in the site's own templatesImprovements disappear when a testing script is switched off

The accounts are created in the business's name and the provider is given access, at the lowest level that does the job, with the business's own administrators kept in place. The access review tutorial for marketing tools keeps an access register that includes agency and freelancer access, so that access can be withdrawn person by person when the team or the arrangement changes.

The test log deserves particular care. The experimentation programme playbook calls the Learning Log the programme's memory: one entry per test, from the question and hypothesis through the design and result to the decision and whether it was acted on. A provider that keeps that record in its own systems is keeping the most valuable thing the business paid for.

Research brings personal data with it. The recordings and heatmaps tutorial handles a recording tool the way it handles every other tag: it loads only under the consent decision the site has made, every input is masked by default, and the tutorial points out that the GDPR, one of this page's sources, gives personal data a broad definition. A Session Recording that captures what people type is personal data nobody needed, and the guide to keeping personal data out of analytics shows where else it leaks. Who may see recordings, how long they are kept and what the contract says about them are decisions for the business's data protection lead, not for the provider or the marketing team alone.

Questions to ask before hiring

What counts is how specific each answer is. A careful provider answers with how it works on sites like yours; a careless one answers with reassurance or with results from somebody else's site.

QuestionWhat a careful answer contains
Who will do the research, the analysis and the build, and how much of their time do we get?Named people and roles, including who reads results and who writes the code for variants
How do you decide what to test?Research before tests, with each backlog entry located on a page and step and seen by more than one method
How do you size a test, and what happens when our traffic cannot support one?A baseline from our own analytics, the smallest effect worth shipping, a sample fixed before launch, and research, fixes or bolder changes where the arithmetic rules a test out
When do you read a test, and who declares the result?Once, at the planned sample, against a stopping rule written before launch, by someone other than the person who proposed the idea
How will results be reported?The plan quoted, the primary metric as a range, every guardrail, and inconclusive and losing tests listed alongside winners
How does a winner reach the site?Rebuilt in our own code by our developers or theirs, with the testing script used only as a short bridge
Whose accounts will the work run in?Ours, with access granted to the provider
How will you handle recordings and personal data?Our consent decision respected, inputs masked, access limited and agreed with our data protection lead
What happens when we part?The accounts, the backlog, the test log and the research stay with us; scripts and variants are removed in an agreed order

The sizing question separates providers more quickly than any other. The prioritising tests guide makes the effect a test can detect with the traffic available the binding limit on any backlog, and the guide to sample size and significance shows why a page with little traffic cannot detect a small change, however sound the reasoning behind it. A provider who promises a steady stream of tests to a site with few conversions each week is promising tests that cannot answer their questions. The honest answer for such a site is the one given in the guide to improving conversion without A/B tests: fix what is broken, research the rest, test bold changes or whole templates where the traffic allows, and read shipped changes against a prediction written in advance.

The reading question is the second filter. Evan Miller's essay, listed in the sources, shows that a test watched continually and stopped at the first significant-looking result will declare winners among changes that do nothing far more often than the significance level promises. The A/B testing guide turns that into procedure: the metric, the sample size and the decision rule are all set before launch, and the result is read a single time. A provider who reads results daily and calls them when they look good will report more winners than one who does not, and fewer of them will be real.

Warning signs

  • Accounts in the provider's name. A testing, recording or analytics account the business does not own means the history leaves with the provider.
  • Promised uplift before the site has been read. Nobody can say in advance which changes will help a particular site, which is the point the conversion rate optimisation pillar makes about tricks; a figure in a proposal is a sales line.
  • A fee tied to the number of winning tests. The playbook warns that win rate is a poor measure on its own, and Goodhart's law describes what happens to any measure once it becomes a target. Paid per winner, a provider is paid to call results early and to test changes too timid to lose.
  • Tests without research. A plan that starts with a list of changes rather than with where visitors leave is testing opinions, which the experimentation playbook warns rarely teaches much.
  • No written plan before launch. Without one, nobody can tell later whether the metric, the audience or the end date moved after the data arrived.
  • Few inconclusive results. The A/B testing guide notes that most results in most programmes are inconclusive. A report that shows mostly winners is reporting selectively or reading early.
  • Segments presented as wins. A test that lost overall but "won on mobile returning visitors" has found a hypothesis, not a result, as the testing pitfalls guide explains.
  • Single figures added into an annual claim. Point estimates from several tests, summed and projected over a year, describe nothing the business will see.
  • Winners left in the testing tool. A change served indefinitely by a script has not shipped.

The diagnosis for glowing agency reports beside flat sales traces how several of these signs work together in any agency relationship: the agency's own figures in the report, credit claimed for outcomes the business would have had anyway, and a fee that grows with activity.

Scoping the opening piece of work

It helps to tell four kinds of work apart, since each fits a different situation and leaves something different behind.

ScopeWhat it deliversWhen it fits
Tracking check and expert reviewVerified conversion counts and a list of issues across a journey, each with the evidence that would confirm itAny site, and always before tests are planned
Research sprintA ranked backlog with evidence from several methods for one funnel and one questionBefore a testing programme, or when results have stalled
Testing programmePlans, builds, quality checks, read-outs and the log, on a rhythmOnce the tracking is trusted, a backlog exists and the traffic can support tests
Fixes and page buildsChanges that should simply ship, built into the siteWhen research finds faults rather than open questions

Start with the first or the second. The expert review tutorial and the research sprint tutorial show what good work of each kind contains, and the landing page audit scores a single page where that is the job. A provider who wants to start testing before anyone has checked that the purchase or lead event fires once is skipping the step that most often turns a test into a confident wrong answer.

The agency request for proposal template puts the brief, the questions and the scoring criteria into one document, so that every provider answers the same RFP. Score the answers with a weighted scoring matrix, with the weights agreed before anyone reads a proposal, and treat ownership of the accounts as a condition rather than a weighted criterion. Once a provider is chosen, the agency scope of work template records deliverables, exclusions, access, reviews and exit, and the agency onboarding checklist covers the first agreements before anyone touches a live site. A Retainer makes sense for a testing programme with a stable rhythm; a fixed fee suits a review or a sprint with a defined output.

Reports you can check

A test report is useful only if the business can verify it. Five tests apply to every read-out:

  • It quotes the plan. Hypothesis, primary metric, guardrails, planned sample and reading date, from a document written before launch, with the date it was written. The A/B test launch checklist is a fair standard for what that plan contains.
  • It states the result as a range. The primary metric is reported with its Confidence Interval, against the smallest effect worth shipping, so that a win reads as "somewhere between" two values rather than as one headline figure.
  • It reports every guardrail. Each Guardrail Metric named in the plan appears with its result, including the ones that moved the wrong way.
  • It lists every test. Winners, losers and inconclusive tests appear in the same report, with the sample each reached and the decision taken.
  • It can be recounted. The counts behind each arm can be checked independently. The GA4 tutorial for A/B test analysis records an exposure event in the business's own analytics so that each arm's conversions can be counted outside the testing tool and set beside the order system or the CRM.

The experiment results report template sets out a one-page Readout in that order, with the interval ahead of the point estimate. Ask for it, or something equivalent, as the standard format for every test the provider runs. The conversion rate calculator turns any arm's counts back into a rate, which is a quick check that the figures in a report agree with the counts beneath them.

Getting winners into the site

A winning test settles a question, but the site is still unchanged. The tutorial for shipping a winning variant covers what has to happen after the read-out for the gain to last: the decision confirmed against the plan, the variant rebuilt in the site's code, device and tracking checks, an entry in the release log and a watch on the metric. Agree before the work starts who does each of those steps.

The question matters because of what happens when nobody owns the rebuild. A testing script left serving a winner adds weight to every page, can briefly display the original before swapping in the new one and takes the browser's attention away from the site's own code, and the change sits outside the codebase, out of the next developer's sight. The testing tool can carry a winner for a short time while the rebuild is scheduled, provided the scope names the date it stops.

The playbook treats the "implemented" field as the most important in its log, since winners commonly wait in the development queue while other work goes first. The share of winners that actually shipped is the measure to ask a provider for, and to report alongside the results themselves.

Judging the work once tests have run

Judge the process early and the outcomes later. In the opening weeks, check what is within the provider's control from the start: access granted in the business's accounts, the tracking verified against the business's own records, research with its evidence attached, plans written before launch and quality checks done before tests count.

Outcomes take longer. A test cannot be judged before it reaches its planned sample, and a programme cannot be judged before several tests have been read and their winners shipped. When that point arrives, look at four things together:

  • The quality of the evidence behind each test. The research hub judges a programme by how many hypotheses rest on evidence and how many of those moved the metric the way they predicted.
  • How results were read. Tests run to their planned samples, results declared by the person the plan named, and inconclusive results recorded rather than rerun until they win.
  • What shipped. The share of winners rebuilt in the site and the time each took, from the log.
  • What the business saw. Orders, revenue or qualified leads over comparable periods, read by source and device rather than as one overall figure. Reported wins that never appear in those figures need explaining, and the usual explanations are tests read early, single figures summed, novelty that faded or winners that never shipped.

Be careful with win rate as a verdict in either direction. The playbook warns that a programme winning almost every test may be choosing timid changes or declaring results early, while one that rarely wins may be testing ideas with too little evidence behind them. On a site with little traffic, the honest provider spends more time on research and fixes than on tests, and the output to judge is the backlog and the fixes shipped.

Changing or ending the arrangement

Order matters here, and the agency offboarding checklist works through its groups by risk: ownership, then assets and knowledge, then continuity decisions, then access removal, with the close-out of invoices and the handover note after it. For conversion work that means confirming the business holds administrator access to every account; collecting the test log, every plan and read-out, the research backlog and the evidence behind it; deciding whether each running test is read at its planned sample or stopped and recorded as stopped; rebuilding any winner still served by the testing script; and only then removing the outgoing provider's access, person by person. The incoming provider starts from the log and the backlog rather than from a blank page.

A sewing supplies retailer compares proposals (illustrative)

Illustrative example — not a real result

Wimbercoth Sewing Supplies chooses conversion help

Wimbercoth Sewing Supplies, an invented online retailer of sewing machines, fabric and haberdashery, takes about 9,000 orders a month from roughly 410,000 sessions. It sends the same request for proposal to three providers and scores each answer from 1 to 5 against weights agreed in advance.

Criterion (weight)Provider A (agency)Provider B (agency)Provider C (freelancer)
Accounts and test log (condition)Fails: tests run in its own accountPasses: works in the retailer's accountsPasses
Research before testing (25%)2: a list of changes from other clients5: tracking check and research sprint first4: expert review
Test sizing and reading (25%)2: eight tests a month, read weekly5: sample fixed before launch, read once3: not asked to test
Reporting (20%)2: lifts as single figures5: ranges, guardrails, every test listed3: written findings
Shipping winners (15%)1: winners left in its script4: rebuilt with the retailer's developer2: not covered
Fee basis (15%)1: retainer plus £600 per winning test4: fixed fee for the sprint, then a retainer4: fixed fee
Weighted score out of 51.704.703.25

Ownership rules Provider A out before any score is added, and the score merely agrees. It buys a six-week research sprint from Provider B for £14,000. The sprint finds the purchase event firing twice on the payment provider's return page, which had put the reported conversion rate at 2.6% against 2.2% in the order system, and a delivery charge first shown at the last checkout step. The tracking is fixed and the delivery charge moved to the basket as a fix without a test. Over the next quarter two tests run to plan: a fabric-finder change on category pages ends inconclusive, with an interval from minus 3% to plus 5%, and is logged; a rewritten machine-comparison table wins, with an interval of plus 2% to plus 9% on orders per visitor and no change in returns, and is rebuilt in the site's code within three weeks. Orders per visitor over the quarter, read against the same weeks a year earlier, rise by about 4%.

Common mistakes

  • Scoring proposals before ownership is settled. A low bid that keeps tests in the provider's own account turns out the dearest when the relationship ends.
  • Letting the provider define success. What counts as a conversion, what it is worth and which guardrails apply are business decisions.
  • Starting with a testing retainer. Testing before the tracking is checked and the research is done buys activity, not answers.
  • Paying per winner. The measure becomes the target, and the results stop meaning what they say.
  • Accepting single-figure results. Ask for the range, the guardrails and the inconclusive tests every time.
  • Leaving winners in the testing tool. Agree who rebuilds them, and by when, before the first test runs.
  • Judging on reported wins alone. Set the provider's figures beside the business's own orders or leads.
  • Planning the exit at the end. Agree at the start what the business keeps and in what order access is removed.

Frequently asked questions

Should we pay a CRO agency for each winning test?

Not as the main way of paying. A fee that rises with the number of winners rewards calling tests early, quoting the most flattering figure and choosing timid changes that are easy to win, which is the gaming Goodhart's law predicts once a measure becomes a target. Pay for the work agreed in the scope, and judge it on the quality of the research, whether tests ran to their planned samples, how many winners reached the site and what happened to orders or leads afterwards.

What should a conversion provider be asked to do first?

A check of the tracking and a short piece of research, such as an expert review or a research sprint, before any testing retainer. Tests built on events that fire twice, or on guesses, waste traffic and produce confident wrong answers. The research gives the business a ranked backlog it keeps whoever runs the tests, and it shows how the provider thinks: whether it locates problems with evidence or arrives with the list of changes it makes on every site.

How soon can we tell whether a CRO agency is working?

Judge the process from the start: access granted in the business's own accounts, the tracking checked against the business's own records, research with evidence attached and a written plan for every test before it launches. Judge results only once tests have reached their planned samples and winners have shipped, and judge them on the business's own orders or leads over comparable periods. On a site with little traffic, research and the fixes it produces stay the main output for longer, and that is a sign of honesty rather than slowness.

Terms used on this page

Tools to try

Frameworks

Where this page is used

Sources

  1. How Not To Run an A/B Test — Evan Miller
  2. Regulation (EU) 2016/679 (GDPR) — EUR-Lex

Related