Why one good answer proves nothing
AI output varies. The same prompt run twice gives different wording, and sometimes different facts, because most tools add controlled randomness when generating text, as how language models generate text explains. Output also varies with the input: a prompt that handles a simple product perfectly may fail on one with a complicated specification, a missing field or a regulated claim. A demo or a first try samples one point from that range, usually a flattering one, because people try the obvious case first and vendors choose their best examples.
The decision a team actually faces is about the range: across the tasks this prompt or tool will meet, how often is the output usable, how often does it need heavy editing, and how often does it fail in a way that could reach a customer? That is a measurement question. The NIST AI Risk Management Framework names "measure" as one of its four functions, alongside govern, map and manage, and for a marketing team measuring mostly means answering that question with evidence from its own work before relying on a tool. The accuracy and review workflows guide checks outputs one at a time once they exist; testing decides beforehand whether the approach is good enough to be worth reviewing.
The method is the same discipline as an A/B TestControlled comparison of two variants., applied offline: a fixed set of inputs, a predefined way of scoring, a comparison that removes bias, and a decision rule set before the results are in. It fits the experimentation loop: hypothesis, test, read, decide, and run again.
Building a test set from real work
A test set is a collection of real inputs for the task, each with notes on what a good output must contain. Draw it from the work the tool will actually do, not from examples invented for the test.
- Typical cases. The ordinary, frequent inputs: the standard product, the usual email, the common question.
- Hard cases. Long or messy inputs, conflicting sources, specialist terms, products with exceptions, anything that has caused problems before.
- Edge cases. Inputs that should make the tool decline or flag a gap: a missing price, a question outside the documents, a request for a claim the business cannot make.
- Sensitive cases. Where a mistake would matter most: regulated categories, prices, legal wording, named people.
For each case, write the reference notes: the facts that must appear, the facts that must not, and any constraints. Where an approved human-written output exists, keep it as a reference too. Remove personal data from cases unless the tool is approved for it, following the disclosure and data privacy guide.
Size the set to the decision. A quick check on a low-risk prompt can use a small set; a decision to automate a customer-facing task needs a larger and more varied one. The idea of an adequate Sample SizeThe number of users or sessions an experiment needs to detect a given effect reliably. applies loosely: too few cases and the result depends on which ones happened to be chosen. Keep the set fixed once testing starts, and store it alongside the prompt; a team prompt library keeps test cases in each entry for exactly this reason.
Writing a scoring rubric
A rubric turns "this one feels better" into scores that two people would broadly agree on. Write it before running any outputs, so it reflects what the task needs rather than what the first output happened to do well.
Choose a small number of criteria that matter for the task, define each, and describe what each score means. A simple scale with clear descriptions is more reliable than a fine one with vague ones. Separate criteria that are pass or fail, such as a factual error or a banned claim, from criteria that are matters of degree, such as tone. A single serious failure should not be averaged away by good scores elsewhere.
| Criterion | Type | Unusable | Needs editing | Usable as is |
|---|---|---|---|---|
| Factual accuracy against the source | Pass or fail | Any fact not in the source, or contradicting it | Not applicable: any error fails | Every fact traceable to the source |
| Required content | Graded | Key facts or sections missing | Minor omissions | Everything the reference notes require |
| Audience fit | Graded | Wrong level or wrong reader | Right reader, uneven level | Reads as written for the named audience |
| Voice and style | Graded | Clearly off-brand | Needs a pass for tone | Matches the voice guidelines |
| Constraints | Pass or fail | Breaks a hard rule: length, banned claim, legal line | Not applicable: any breach fails | All constraints met |
| Handling of gaps | Pass or fail | Invents to fill a gap | Not applicable | Flags missing information |
Test the rubric before using it. Two reviewers score the same handful of outputs independently; where they disagree, the description is unclear and gets rewritten. The rubric then becomes the reviewers' standard in live use as well.
Running the comparison blind
Most tests compare options: the current prompt against a revised one, two models, a tool against the team's existing process. People favour the option they built, the brand they recognise and the one they expect to win, so remove that information before scoring.
Run every test case through every option, several times each where outputs vary between runs, and keep every output, not just the best. Then strip anything that identifies the source, shuffle the outputs and give them to reviewers with the rubric and the reference notes. Where a human-written version exists, include it in the mix; it is the baseline the AI options have to beat or match, and reviewers are often surprised by where it lands.
Keep the conditions equal. The same inputs, the same context blocks and the same instructions apart from the thing being tested. If the comparison is between models, use the same prompt; if it is between prompts, use the same model. Change one thing at a time, or the result cannot say what made the difference.
Log every failure with its case, option and a short description: invented a feature, ignored the length limit, missed the exclusion, wrong audience. The failure log is often more useful than the scores, because it shows what kind of review the output will need and where the prompt or the documents need work.
A lighting retailer tests a revised description prompt
Glimmerholt Lighting, an invented online lighting retailer, wants to replace its product description prompt. It builds a test set of 24 real products: 12 typical, 6 with complex specifications, 3 with missing fields and 3 with safety-rated bathroom fittings. It runs the current prompt and the revised one 3 times per product, giving 144 outputs, and adds the 24 existing human-written descriptions. Two reviewers score all 168 blind against a 5-criterion rubric. The revised prompt produces 61 of 72 outputs usable as is or with light editing, against 44 of 72 for the current prompt. The failure log matters more: the current prompt invented an IP rating in 5 of the 9 bathroom-fitting outputs; the revised one, which includes the instruction to write "not in source", did so once. Because the pass bar set in advance was zero invented safety ratings, the team adopts the revised prompt for standard products only and keeps bathroom fittings with a human writer.
Reading the results and setting a pass bar
Agree the pass bar before looking at the results, in the same way an A/B test fixes its success measure in advance. A pass bar has two parts: a minimum share of outputs that are usable as is or with light editing, and a list of failures that are unacceptable at any rate, such as invented prices, safety claims, legal wording or anything that would breach the claims rules. The first is a matter of efficiency; the second is a matter of risk.
When reading results, look past the averages:
- Check the failure log first. One unacceptable failure type can rule out an option whatever its average.
- Look at hard and edge cases separately. An option that wins on typical cases and fails on edge cases may still be the right choice, but only with a rule that sends edge cases elsewhere.
- Count review effort. An option whose outputs need heavy editing may save less time than it seems; include the editing in the comparison.
- Treat small differences as ties. If changing a few scores would reverse the result, the test has not separated the options. Add cases or decide on other grounds.
- Compare with the human baseline. The question is whether the AI option is good enough for the task, not whether it is perfect.
Record the decision with the evidence: the test set, the rubric, the scores, the failure log and the pass bar. The share of outputs passing review first time then becomes a useful KPIA metric tied directly to an objective. to track in live use, to check that the tool behaves in production as it did in the test. The academy exercise on measuring AI's impact for a flooring retailer practises building a test set and scoring rubric for a product-page prompt, with the pass bar agreed before any output is scored.
Testing output quality is one input to choosing a tool, not the whole decision. Data handling, contract terms, cost and fit with the stack are covered by the evaluating AI tools guide, the martech tool selection guide, the martech tool evaluation template and the evaluating an AI tool lesson, which all assume a quality test like this one has been done.
Re-testing when the model changes
A passed test describes one combination of prompt, context blocks, model and tool settings at one point in time. Change any of them and the result may no longer hold. Vendors update the models behind their products, sometimes with little notice, and a prompt that passed last quarter can behave differently after an update.
Rerun the stored test set when any of these happens: the prompt or a shared context block is edited; the vendor announces a model change or the team switches models; tool settings such as length or creativity controls are changed; the source documents the tool draws on are restructured; or reviewers notice a change in the kind of errors they are catching. Compare the new scores and failures with the recorded baseline, apply the same pass bar, and log the result. If the test set is stored and the rubric is fixed, a rerun is a routine task rather than a project. When output has already drifted and reviewers are rewriting drafts that once passed, the diagnosis for AI output quality getting worse dates the decline and tests each change that might explain it. For a customer service assistant, the AI customer service launch checklist asks for this test, and a plan to rerun it, before the assistant answers its first customer. For a workflow step that composes, classifies or acts without a person reading each output, the AI automation go-live checklist makes the rerun a check before any prompt, model, vendor or source change goes live.
Once outputs are live, testing gives way to two other kinds of evidence. Review logs show whether the failure rate in practice matches the test. And where the outputs are customer-facing copy, a live test with real traffic, as the A/B testing guide describes, shows which of the passing options performs better with the audience. Offline testing decides what is good enough to publish; live testing decides what works.
Common mistakes
- Deciding on a demo. One good answer, often on the vendor's own example, says nothing about the range.
- Testing on invented inputs. Made-up cases are cleaner than real work and flatter every option.
- Writing the rubric after seeing outputs. The criteria then reward whatever the first output did well.
- Averaging away serious failures. An invented price is not offset by good tone; keep pass or fail criteria separate.
- Scoring with the labels showing. Reviewers favour the option they expect to win.
- Changing several things at once. A new model with a new prompt cannot say which made the difference.
- Running each case once. Outputs vary between runs; a single run can mislead either way.
- Never re-testing. A model update can change behaviour silently; stored tests catch it before customers do.
