Why an agency's results need testing before you appoint it
An agency chosen on the results it shows is chosen on evidence that the agency selected and presented. The hierarchy of evidence framework lists an agency's view among the typical sources of anecdote and opinion, the weakest kind of proof, and asks for stronger evidence as more is committed and as the decision becomes harder to reverse. Appointing an agency is a decision of that kind: a budget is committed, and ending the relationship later calls for a transition plan, as the selecting guide describes. That is a reason to test the results before appointing, not a reason to distrust agencies as a class. The aim is to find out how much weight each claimed result can carry.
The agency selection guide gives references one row in its stage table and warns against shortlisting on awards or reputation alone. A request for proposalA document sent to shortlisted agencies or sellers giving each the same facts and questions and the published scoring criteria, so responses compare. puts the same numbered questions, in a fixed order, to each shortlisted agency. Neither says how to read a result once an agency shows one, and that is the gap this tutorial fills. The guide to stating a contribution honestly teaches a marketer three separations: their own part from the group's, the result from the outcome that would have occurred regardless, and platform figures from the business's own records. This tutorial is the buyer's side of those same separations, with the evidence graded and a referee's own figures added.
Where a step rests on a site guide, the text says which one. Where a step goes beyond the guides, such as a single result sheet for all the shortlisted agencies, asking for the cases that did not work, or asking a referee for figures from their own records, it is general practice and is labelled that way; none of it is a standard. The page states no figure, share or period for how agencies report, how common misleading case studies are or what a good result looks like. It also leaves out contract warranties, whether an agency's figures were audited, the confidentiality of client work, legal clearance of a claim, what an agency costs, how any named platform behaves and how long a test should run.
Run the tutorial on the shortlist, not the long list. The selecting guide records the long-standing encouragement from the Institute of Practitioners in Advertising to run proportionate processes with a limited number of pitching agencies, and notes that each agency which pitches puts in significant time. It is written for the head of marketing, owner or marketing manager who is choosing between agencies and needs to know how far each agency's shown results can be trusted.
Before you start
- A shortlist and a decision to make. The agencies still in the process after the long list, and the appointment they are being compared for. As general practice, this tutorial works on what the agencies show, alongside the meetings and the references stage.
- The work, the objective and the budget range written down. The results an agency shows only matter if they bear on the work it would do here, so the work and what success looks like need to exist on paper before the sheets arrive.
- The business's own records. Orders, enquiries or pipeline from the systems the business keeps, not from a platform, so the business can describe itself plainly on the sheet and later read the agency's work against its own history.
- Named scorers, including the in-house colleague who will be the agency's working contact. The RFP template asks for the scorers to be named, and both the selecting guide and the RFP template ask for someone from the in-house team to be among them.
- Someone who can read figures. An analyst, or whoever looks after reporting, for the steps on the numbers and on the referee's figures.
- One place for the sheets, the notes from each call and the scores. Each score has to cite its evidence, and the evidence has to be somewhere a second scorer can find it.
The steps
Fix the weights, the gates and what the evidence must show before any sheet arrives
Decide how the agencies will be judged before any of them has shown a result, and date what is decided, as the weighted scoring matrix framework sets out. Write down, with a date, three things.
The criteria and their weights for the evidence. The rest of the appointment is scored on other criteria, such as those in the RFP template's filled example; this tutorial scores only what an agency shows about its results. As general practice, the criteria for that part are the strength of the evidence behind each claimed result, how cleanly the agency's part is separated from the client's and the market's, how far the result carries over to this business, who produced the work and whether they are on the account, and how far a referee's own figures agree with the sheet. Weight them by what this decision needs, and tie each heavy weight to something the appointment must achieve.
The gates. A gate is a requirement an agency either meets or does not, checked before any score is given, and an agency that does not meet one is out. As general practice, two gates suit this part. The first is that the work will be done in accounts that stay in the business's name, which is the ownership point the selecting guide makes about ad accounts and analytics properties. The second is that the business can test the agency's reports against its own orders or records, which the selecting guide describes for several of the channel hiring guides.
What the evidence must show. Before seeing any sheet, write what you would accept as the level of evidence for a result, scaled as the hierarchy framework's rule of proportion asks. Then write the anchors for each criterion in terms of this evidence: what a weak, a middling and a strong result sheet would show. Name the scorers in the same record.
Ask every agency for the same result sheet
Prepare one result sheet and ask each shortlisted agency to fill it in, with the same instructions and the same return date. The agency RFP template explains why a fixed format matters: answers given in the client's format can be compared, and answers in the agency's own format cannot. If the shortlist is being sent a request for proposal, the sheet goes in as one more section; if not, it goes out alone. If an agency sends a credentials deck anyway, only the sheet is scored.
The sheet has one row for each result the agency wishes to claim, with the same fields each time, laid out in the table below. Ask the agency to choose the results it thinks bear closely on the work on offer and to say why, since the choice is itself information. The fields are:
- The claim in one sentence, in the agency's own words, so the later grade is of what was said and not of what was meant.
- The measure and where the figure came from. The KPIA metric tied directly to an objective. the result is reported on, and whether the figure comes from a platform's report, from the client's own records or from a test.
- The comparison. The period or group the result is measured against, and the BaselineAn estimate of what a metric would have done without a given change, campaign or promotion, against which the effect is measured. that comparison starts from.
- What else changed in the period, on the client's side and in the market.
- The agency's part and the client's part, in plain verbs.
- The client's situation: what it sold, to whom, in which season and with what already in place.
- Who at the agency produced the result.
- A referee who can confirm the figures from the client's own records.
- The cases not shown: comparable work that did not give this result.
Leave a field blank rather than fill it with a guess. A blank is recorded as not shown, and the scoring step treats it as the lowest description for the part of a criterion that field feeds. The sheet is general practice, built from the questions the later steps ask, and the table after the steps shows how each field is used.
Separate the agency's part from the client's and the market's
A claimed result is a movement in a number, and several people and forces moved it. The contribution guide notes that marketing results are produced by a group of people at once and move for reasons outside marketing's control, among them price changes, the season, stock levels, rivals' moves and events in the news. A case study that credits the whole movement to the agency has not made those separations, so make them on the sheet.
The agency's part against the client's. Ask the agency to list everyone who touched the result, the client's own decisions included. Then ask for the agency's own part in verbs that can be checked, as the contribution guide's verb test does for a claimant; the buyer can apply the same test in reverse when reading a sheet.
The result against the market. Ask the agency to state what else changed in that period and what it did to rule those changes out. The guide to explaining a metric's movement sets out how to test an explanation on data other than the data that suggested it, and an agency that has done the work can show it. Where the agency cannot name what else changed, the sheet records that, and the business can add what it knows of the market in those weeks.
Write beside each claim a line saying what the agency's part appears to have been once the client's part and the market's are set aside. It is a judgement and is labelled as one, and it feeds the score for how cleanly the agency's part is separated.
Grade the evidence behind each claim
Take each claimed result to the table in the hierarchy of evidence framework and write down the row its evidence belongs in. The framework's own test is a question: could the evidence look the same if the claim were untrue? As general practice, ask the agency to say how the comparison was made, in plain terms. Was any group of people, places or periods held apart as a point of comparison? How was it chosen? What range did the result come with? An agency that ran an IncrementalityConversions caused by marketing. test can say what was withheld from whom. One that did not can say so, and the answer places the result in a lower row without discrediting it.
Three points keep the grading fair. First, the grade is of the evidence, not of the agency. A before-and-after readingA measure read before and after a change with no comparison group, so the season and everything else that moved in those weeks is credited to the change. with a stated baseline and the other changes listed sits in its own row of the framework's table, and it can still answer the question well enough for a small decision. Second, compare the grade with the level you wrote down in the first step. A result graded below the level the decision needs is not thrown out; the gap is recorded and carried to the last step. Third, keep any range that comes with a result, because a midpoint quoted alone overstates what is known.
Write the grade on the sheet, beside the claim, since the framework asks that the strength of the evidence be labelled next to the result. The incrementality testing guide covers how the tests at the top of the ranking are designed and read, including their size and duration, and this tutorial does not.
Check the numbers can bear the claim
A graded result can still rest on numbers too loosely compared or too thin to carry the claim. Run three checks on each row, each from an analytics guide that owns the method.
The comparison. The guide to comparing periods sets rules for a fair comparison; read the sheet's comparison against them. A rise measured across a season that held the client's seasonal peak, with no comparison period, is read against the calendar before it is read against the agency.
The count. The guide to reading small numbers wants the count shown beside each rate and a range put around a figure built on few events. Ask for the count behind each change on the sheet. The range around a figure depends on the count behind it, so a change built on a handful of orders is read with a wide range and a change built on a large count with a narrow one.
The source. Ask whether each figure rests on a platform's AttributionAssigning conversion credit to touchpoints., which credits conversions to that platform's own ads, or on the client's own order records. The guide to platform and analytics attribution explains that platform figures added together can claim the same order more than once, and the diagnosis of the gap between an agency's reporting and the sales ledger shows how to set an agency's figure beside the business's orders for a closed month. A platform's figure is a starting point and the business's own record is the test. As general practice, a sheet whose only source is a platform report is graded in the attributed figures row of the hierarchy table, and the later step on referees sets a referee's records beside it.
Write on each row whether the numbers bear the whole claim, part of it or none of it, and why. The reasons are what the second scorer will read.
Ask about the cases you are not shown
A sheet of case studies can be made of survivors. The survivorship bias framework names the missing cases as the ones that left, failed or closed, and the data library's benchmarks section makes the point for agency-compiled figures: the sample is the agency's own client base, and the clients that remain in it are those whose results are good. The agency chose which cases to include, so the cases that went badly may not be among them.
As general practice, put the other side of each result to the agency in writing:
- Which comparable clients did it work with in the same period, and how did their results compare with the one shown?
- Which clients left in that period, and what reasons did they give?
- Which work of this kind was stopped, changed or did not produce the effect, and what was learned?
- How were the cases on the sheet chosen?
This builds on the question the RFP template's filled example puts about a client whose results plateaued, and it gives you a way to read the answer. Look for specifics: what was tried, what the comparison showed and what changed afterwards. An answer that names no case, or a refusal, is recorded as not shown.
As general practice, the same caution applies to the referees an agency offers. The later step on referees is what tests what they say.
Test whether the result carries over to your business
A well-supported result can still answer a different question, so grade the fit on its own, as the hierarchy framework asks.
As general practice, set the client's situation on the sheet beside the business's own, on points that may bear on whether the same work would have the same effect. They can include what was sold and how people buy it, the audience, the season the result fell in, whether the account was new or long-running, the channels the work used and the budget it ran on relative to the client's size, and what was already in place when the agency arrived, such as tracking that feeds orders back, a team able to act on advice and stock to sell. A tested result is tied to its spend level, because the incrementality guide describes a test as speaking only for the channel, the spend level and the period it covered.
For each difference, ask whether it would help the result here, hurt it or leave it unchanged, and record the answer. A difference that cannot be judged from the sheet becomes a condition to check in the first quarter. Do not use another business's average as the yardstick: the data library argues for comparing a business with its own history, and this tutorial gives no view on what a good result looks like in any sector or channel. The question is only whether this result, on this evidence, would follow here.
Ask who produced each claimed result and whether they are on your account
This step asks one thing that the meetings and the written questions do not: who at the agency produced each claimed result, and whether those people are on the business's account. The selecting guide's chemistry meetings row covers meeting those who would carry out the work, and the RFP template asks which people would staff the account and what share of their time it would receive. Those answers are collected there and are not repeated here.
As general practice, record on each row of the sheet the roles or names of the people who produced the result. Then lay that list beside the team the agency proposes for the account. For each result, ask whether anyone who produced it is on this account. A result produced by people who are not proposed for the account is evidence about the agency's other people, and the sheet should say so; as general practice, it is scored under who produced the work and whether they are on the account, and not under the strength of the evidence.
Ask each referee for figures from their own records and compare them with the agency's sheet
The SEO agency hiring guide and the media agency hiring guide each carry a question for referees; keep those questions for the calls. This step adds a different one, and it is general practice: ask the referee for the figures from their own records for the specific result on the sheet, and compare them with what the agency wrote.
- Ask the agency for a referee tied to the specific result, someone at the client who can read the client's own records for it, not a general contact.
- Ask for the referee's figure first. Name the measure and the period from the sheet and ask what the client's own records show, from the order system, the enquiry log or the sales pipeline, before saying what the sheet says.
- Then read out the sheet's figure and compare the two: is it the same measure over the same period, counted the same way, for instance with refunds and repeat customers either in or out of both?
- Write the difference beside the sheet's figure and do not argue it on the call. Record one of four outcomes: the figures agree, they differ and the difference is explained, they differ and it is not, or the referee could not give figures.
A referee who cannot give figures is recorded as not shown, which is a statement about the evidence and not about the agency's work. A referee's agreement strengthens the sheet's figure for that one result without telling you how the agency's other clients fared, which is what the earlier step on missing cases asks about.
Score each agency's evidence against those weights, citing the evidence for every score
Apply the weights and gates fixed in the first step. Apply the gates first and drop any agency that fails one. For the agencies that pass, score and compare as the matrix framework sets out. The scorers include the in-house colleague who will be the agency's working contact, as the selecting guide and the RFP template ask.
Cite the evidence for every score: the sheet row, the grade, the referee's figures or the note from the call, and mark a guess as a guess. A blank on the sheet scores at the lowest description for the part of a criterion it feeds, and is written up as not shown. Run the matrix framework's sensitivity check on a second set of weights, and settle a narrow lead with a further reference call or a further written question, not by adding criteria until one agency edges ahead.
The evidence score is one input to the appointment, alongside criteria such as those in the RFP template's filled example.
Turn what stays unproven into first-quarter checks
Some of what the sheets show will remain unproven when you appoint: a result graded below the level the decision needed, a condition for carry-over that could not be confirmed, a referee's figure that differed or could not be given. Appointing the agency does not prove any of them. List each one, and as general practice turn each into a check written into the plan for the first quarter: the measure, the business's own source, the comparison, the date it will be read and who reads it.
Record the baseline before the agency changes anything, as the agency onboarding checklist asks, so that the checks have something to be read against. Once the agency is working, the renewal tutorial and the diagnosis of an agency's reporting against sales read its results against the business's own measures, and the checks written now are what those later readings start from. Keep the sheets, the grades, the scores and the checks together with the record of the decision, so that whoever reviews the agency later can see what was claimed, what was tested and what was left to be proved.
The result sheet at a glance
Keep each agency's rows in one document, one row for each claimed result. Treat the columns as a layout that suits this tutorial rather than a standard, and change them to fit the shortlist.
| Field on the sheet | What it holds | Who fills it | What it feeds |
|---|---|---|---|
| The claim | One sentence in the agency's own words | The agency | The grade and the score |
| Measure and source | The KPI, and whether the figure is from a platform report, the client's records or a test | The agency, then the referee | The check on the numbers |
| Comparison and what else changed | The period or group compared, the baseline, and the changes on the client's side and in the market | The agency, with what the business knows added | The agency's part and the check on the numbers |
| The agency's part and the client's | All who touched the result, and the agency's own part in verbs that can be checked | The agency | The score for how cleanly the part is separated |
| Strength of the evidence | The row of the hierarchy table the evidence belongs in, and the range if there is one | The buyer | The grade and the first-quarter checks |
| Cases not shown | Comparable clients, clients who left, work that was stopped, and how the cases were chosen | The agency | The score for the evidence and the missing-case record |
| The client's situation | What was sold, to whom, in which season and with what already in place | The agency | The score for how far the result carries over |
| Who produced it | Roles or names, and whether they are on this account | The agency, compared by the buyer with the proposed team | The score for who did the work |
| The referee's figures | The measure and the period from the referee's own records, and the difference from the sheet | The referee, recorded by the buyer | The cross-check and the first-quarter checks |
| Score and citation | The score against the written description, with the evidence cited | The scorers | The decision and the record kept |
A scythe maker weighs agencies' results (illustrative)
Drumelcote Scythes weighs three agencies' results
Drumelcote Scythes, an invented maker of hand scythes and garden sickles, sells through garden centres and through its own online shop. It is choosing between 3 agencies for paid search and paid social. Before any sheet is opened, the owner, the marketing manager and the shop manager, who will be the agency's working contact, fix the weights and 2 gates and date them. The gates are accounts held in Drumelcote's name and reports that can be checked against the shop's own orders. The weights, out of 100, are strength of the evidence 30, the agency's part separated from the client's and the market's 20, fit with Drumelcote 15, who produced the work and whether they are on the account 15, and the referee's figures agreeing with the sheet 20. Each agency fills in the same sheet.
The first agency quotes a return on ad spend taken from its own platform reports, with no order figures beside it, and cannot say who worked on the account. Asked in writing whether its reports can be set beside the shop's orders, it replies that they cannot, so it fails the second gate and is not scored.
The second agency shows a rise over a spring that included its client's seasonal peak, from a base of a few orders a week, with no comparison period and no list of other changes. It names a team lead who is proposed for the account but not the people who produced the result. Its referee cannot give figures from the client's own records.
The third agency shows a smaller change against the same weeks a year earlier, matched to the client's own order system, with 2 other changes in the period listed: a price change and a new product range. It names the 2 people who did the work, and 1 of them will work on Drumelcote's account. Its referee gives order figures from the client's own records that agree with the sheet's, and says what went wrong in the first months.
The second and third agencies pass both gates, and each score is entered with the sheet row, grade or call note that supports it. A blank scores at the lowest description only for the part of a criterion that rests on it, so a criterion shown in part scores above the lowest.
| Criterion | Weight | Second agency (score out of 5) | Third agency (score out of 5) |
|---|---|---|---|
| Strength of the evidence | 30 | 2 | 3 |
| The agency's part separated | 20 | 2 | 4 |
| Fit with Drumelcote | 15 | 3 | 4 |
| Who produced the work, and on the account | 15 | 2 | 4 |
| The referee's figures agree with the sheet | 20 | 1 | 5 |
| Weighted total, out of 500 | 195 | 390 |
The 3 scorers score separately and compare. With the weights on the first and last criteria swapped, the totals are 185 and 410, so the ranking holds. The third agency scores highest on the evidence and, with the team, approach and fee also weighed, is appointed. Its result sits below a randomised test or holdout in the hierarchy table, so 2 checks go into the first quarter for what could not be proved: whether the improvement it showed for search appears in Drumelcote's own shop orders, and whether the improvement still shows once any price or range changes of Drumelcote's own are listed beside it, each with its measure, its source in the shop's orders, its comparison, a date and a named reader. Drumelcote records its baseline from the shop's orders before the agency changes anything.
The agencies and partners hub lists the other common agency symptoms and says where to start on each.
Common mistakes
- Accepting a credentials deck in place of a sheet. Each agency then answers a different question in a different shape, and nothing can be set side by side. Send one sheet and score only that.
- Fixing the weights after the sheets arrive. Weights set once a favourite has emerged describe the favourite. Date them before the first sheet is opened.
- Reading a case study as the client's account. It is the agency's account until a referee's own records say the same. Label it as the agency's account until then.
- Taking a platform's figure as the business's result. Platform figures count what the platform can connect to its own ads. Look for the client's own orders or records.
- Reading a rise without a comparison. A season, a promotion or a price change can move the same number. Ask what it was compared with and what else changed.
- Treating a small count as a trend. A change built on a handful of orders carries a wide range. Ask for the count beside each change.
- Skipping the cases not shown. A sheet can show only the work that went well. Ask about the clients who left and the work that was stopped.
- Crediting a result to people who are not on the account. The result is then evidence about other people at the agency. Tie each result to who produced it and compare that with the team proposed.
- Taking a referee's praise in place of figures. Ask for the measure and the period from the referee's own records, then compare.
- Demanding the strongest level of evidence for each claim. Ask in proportion to what is committed and how hard the decision is to undo, and write that level down first.
- Leaving unproven results unproven after appointing. A gap noted at selection and left unchecked remains a gap. Write each into the first quarter with a measure, a source and a date.
