Definition
In AI-assisted marketing, a test set is a collection of real inputs for one task, each paired with reference notes on what a good output must contain. The guide to testing prompts and models defines it in those terms and insists on the word real: the cases are drawn from the work the tool will actually do, not from examples invented to show it off.
The set is what makes AI testing comparable. Two prompts, two models or a tool and the team's existing process can only be compared fairly if they face the same inputs. Without a fixed set, every comparison is between different examples, and the option that happened to get the easier ones wins.
A test set in this sense is not an experiment arm or a group of ads in a creative test. It never meets a customer. It is run offline, scored by reviewers, and used to decide which outputs are good enough to publish before any live test begins.
Building a test set and a rubric
The testing guide sorts cases into four kinds, and a useful set has some of each:
- Typical cases, the ordinary and frequent inputs the tool will mostly see.
- Hard cases, such as long or messy inputs, conflicting sources and products with exceptions.
- Edge cases that should make the tool decline or flag a gap, like a missing price or a question the documents do not answer.
- Sensitive cases, where a mistake would matter most: prices, regulated categories, legal wording.
Each case carries its reference notes: the facts that must appear, the facts that must not and any constraints, with an approved human-written output kept alongside where one exists. Personal data is removed unless the tool is approved for it. The scoring rubric is written before anything is run, so it reflects what the task needs rather than what the first output did well, and pass-or-fail criteria such as an invented fact are kept apart from matters of degree such as tone.
The same idea scales down. The tutorial on turning a brand voice guide into a pack for AI uses a small set of real briefs, a scoring sheet and baseline drafts to check whether a model follows the voice.
In practice: comparing options and rerunning after changes
The first job of a test set is a decision: which prompt, which model, whether to use the tool at all. Every case is run through every option, outputs are stripped of anything that identifies their source and scored blind, and failures are logged by case. The second job comes later. A passed test describes one combination of prompt, context, model and settings at one point in time, and vendors update models behind their products. The guide to AI for automation calls the stored version a regression test set, a fixed collection of inputs with known good outputs re-run whenever the prompt, model or vendor changes, and describes it as the cheapest control on its list and the one most teams skip.
Customer-facing uses raise the stakes. The AI customer service launch checklist asks for the test set, rubric, pass bar and results from real past questions before the assistant answers anyone, plus a plan to rerun the set. The comparison of automation and AI agents applies the same method before an agent goes into use, with several runs per case because outputs vary between runs.
A hearing centre chain's reply assistant passes the demo and fails the test set
Kailtott Hearing Centres, with 9 clinics, trialled an AI assistant to draft replies to customer emails. In the vendor's demo it answered 5 sample questions flawlessly. The marketing lead built a test set of 60 real emails from the previous quarter, 18 of them about hearing aid repairs, each with notes on the correct turnaround and what must never be promised. Scored blind against the rubric, the assistant passed 47 of 60, but 9 of the 13 failures were repair emails promising a return within 5 working days, when the clinics' policy depended on the manufacturer. The prompt was given the repair policy as a source and an instruction to say when a date could not be confirmed. A rerun of all 60 cases passed 58, and the stored set was rerun after the vendor's next model update.
Related terms
A test set is how a team judges a PromptThe instruction and material given to an AI language model, which works like a brief: the model knows language, not the audience or product, unless told.: a revision is kept only if it scores better on the same cases. It is equally how one Language ModelAn AI system trained on large amounts of text that writes by predicting which piece of text is likely to come next. is compared with another, since fluency in a demo says nothing about accuracy on the team's own inputs. When failures are invented facts, the usual fix is GroundingMaking a language model answer from specified material, such as documents retrieved for the question, rather than from what it absorbed in training., giving the model the source material to answer from, and the test set then shows whether that worked. Fine-TuningTraining an existing language model further on a set of examples so that a style, format or task pattern is built into the model itself. builds a style or format into a model through further training on examples; a test set built from the same work is how the result is checked against prompting alone.
Common confusions
- A good demo is a test. A demo shows the tool can do the job once, on inputs someone chose. A test set shows how often it does the job on the team's own inputs, including the awkward ones.
- Testing on a test set replaces a live test. The two answer different questions. The test set decides which outputs are good enough to publish; a live test with real traffic decides which of those performs better with the audience.
- A passed test lasts. It holds for one combination of prompt, model and settings. When any of them changes, the set is rerun and the result compared with the recorded baseline.
- The set can change between options. Keep it fixed once testing starts, or the comparison stops being fair.
