What is different about evaluating AI tools
Conventional marketing tools are largely deterministic: given the same input, an email platform sends the same email and an analytics tool reports the same number. Evaluation can check whether the features exist and work. AI tools are different in four ways that change the evaluation.
First, output quality is variable and task-specific. The same tool can write excellent product descriptions and unreliable analysis, and the vendor's examples say little about your tasks. Second, the data you send may be retained, reviewed or used to train models, depending on the plan and the terms, so the data questions are sharper than for most tools. Third, the model behind the tool can change without your involvement, so what you approved may not be what you are using in a few months. Fourth, AI features often arrive inside tools you already use, so evaluation is needed even when nothing new is being bought.
The NIST AI Risk Management Framework gives a useful structure for these questions: map the context and the intended use, measure how the system actually performs and what could go wrong, manage the risks you accept, and govern the whole with clear accountability. A marketing team does not need the full framework to evaluate a tool, but the sequence is a good check that nothing was skipped. The four functions of the NIST AI Risk Management Framework are set out together, at a small team's scale, on their own page.
Trial on your own tasks
The trial is the heart of an AI tool evaluation, and it is the part most often replaced by a demo. Design it before you talk to vendors, so that every candidate is judged on the same work. The testing prompts and models guide sets out the output-testing method in detail; this section applies it to a purchase decision.
Choose a few representative tasks from the uses the tool is meant for, drawn from the team's real work: the product descriptions, ad variants, report summaries or customer replies you actually produce. For each, prepare the same inputs and sources for every candidate, and define what an accepted output looks like, using your brand standard and the failure modes in the accuracy and review workflows guide. Include a few hard cases: an ambiguous brief, a source that contradicts itself, a request the tool should decline.
Have the people who will use and review the tool run the trial, blind to which output came from which tool where possible. Record, per output, whether it was accepted, the time taken including review, and the defects found by type. Compare against the current way of doing the task, not only against other tools; a tool that wins the comparison but does not beat current practice is not worth adopting. Use only non-personal or properly minimised data in the trial until the data terms have been reviewed.
Data, training and retention terms
The disclosure and data privacy guide lists the terms that matter for any AI tool: training use, retention, sub-processors, region, vendor access and ownership of outputs. In evaluation, get answers to each in writing, for the specific plan you would buy, since consumer and business plans of the same product often differ.
Where the tool will process personal data on your behalf, such as customer records, transcripts or your First-Party DataData a business collects directly from its own customers, visitors and users. more broadly, the vendor acts as a processor under the GDPR. The regulation expects controllers to use only processors that give sufficient guarantees of appropriate protection, and to have a contract that sets out, among other things, that the processor acts on documented instructions, keeps data confidential, uses sub-processors only with authorisation, helps the controller meet its obligations, and deletes or returns the data at the end of the service. A vendor that cannot offer such terms is not a candidate for personal data. Your legal and data protection advisers review the actual terms; the evaluation's job is to collect them and flag gaps.
Pay particular attention to whether training opt-outs are contractual or just a setting, whether retention applies to prompts, uploads and outputs alike, and whether the AI feature inside a tool you already use is covered by your existing agreement or by new terms. Where staff are already pasting customer records into consumer plans because no approved tool exists, the page on staff pasting customer data into AI tools sets the missing approved tool beside the other gaps behind that habit.
Admin, access and audit controls
A tool used by a team needs controls a single user never thinks about. Ask to see them, not just to hear about them.
- Accounts and access. Single sign-on, role-based permissions, and the ability to remove a leaver's access and keep their work.
- Workspace settings. Whether admins can enforce settings, such as disabling training on inputs, turning off features, limiting file uploads or restricting connections to other tools, for everyone.
- Logs. Whether admins can see who used the tool, when and for what, and export logs for review or an incident investigation.
- Data controls. Whether retention can be set, conversations deleted on request and data exported.
- Connections. What the tool can read or act on in other systems, such as your CDPUnified first-party customer profiles., CRM, drive or ad accounts, and whether those permissions can be narrowed.
Controls matter more as the tool moves up the risk tiers in the AI governance hub. An internal drafting assistant needs basic access control; a tool that replies to customers or acts on records needs logs, limits and an owner who can switch it off.
Model changes and vendor stability
An AI tool's behaviour depends on a model the vendor can change. A new model version can improve output, change its style, alter how it handles your prompts or introduce new failure modes, sometimes without much notice. Ask how model changes are announced, whether you can pin a version or delay an upgrade, what notice is given before older versions are retired, and whether release notes describe behavioural changes.
Plan for change whatever the answers. Keep the trial tasks and their accepted outputs as a small regression set, and re-run it when the vendor announces a change or when users report that output feels different. Record the model version alongside prompts in the version history the governance hub recommends, so a shift in output can be traced to a shift in the model.
Vendor stability is the slower version of the same risk. The AI tools market changes quickly, and products are repositioned, repriced, merged or withdrawn. Ask what happens to your data and your configured workflows if the product is withdrawn or the vendor is acquired, and weigh how much of your work would depend on this particular product.
Integration and exit
Integration decides whether a tool becomes part of the work or a separate tab people copy and paste into. Check how it connects to the systems where your inputs live and your outputs go, what data those connections move and in which direction, and whether the connection respects the minimisation and consent rules your data already carries. The marketing tool onboarding SOP covers setting up an approved tool; the evaluation should confirm the integration it will need is possible.
Exit is the question nobody asks at purchase. Before you buy, find out how you would leave: whether prompts, templates, libraries, conversation histories and outputs can be exported in a usable format, whether custom configurations or tuned assistants can be recreated elsewhere, and how the vendor confirms deletion of your data at the end. A tool that is easy to adopt and impossible to leave is a long-term commitment disguised as a trial.
Scoring and the pilot decision
Score each candidate on the criteria above, with weights agreed before the trial. The table adapts the general scoring sheet in the martech template for AI-specific criteria; add your own requirements around it. How to agree the weights, what evidence each score needs and how to check that the result survives a change of weights are covered in the weighted scoring matrix framework for comparing options.
| Criterion | What good looks like | Evidence |
|---|---|---|
| Output quality on your tasks | High acceptance rate with few serious defects, better than current practice | Trial records reviewed blind |
| Time per accepted output | Lower than current practice once review is counted | Trial timings |
| Data terms | No training on inputs by contract; retention, region and sub-processors acceptable | Written terms for the specific plan |
| Processor terms | A contract meeting data protection requirements where personal data is involved | Legal and data protection review |
| Admin and audit controls | Enforced settings, access control, exportable logs | Admin console seen in the trial |
| Model change handling | Notice of changes, version pinning or delay, release notes | Vendor's written answers |
| Integration | Connects to the systems the use needs with narrow permissions | Technical check during the trial |
| Exit | Exportable work and confirmed deletion | Vendor's written answers |
| Cost at expected use | Within budget at realistic volumes | Pricing for the specific plan |
The decision is rarely a simple yes. Common outcomes are approval for named uses and tiers only, approval on condition that specific terms are added, a time-limited pilot with defined measures and an end date, or rejection with reasons recorded. Whatever the decision, enter the tool in the AI register with its approved uses, plan, owner and review date, and set one KPIA metric tied directly to an objective. for the pilot, normally time per accepted output with a defect rate alongside. The AI use register template gives that entry a column for the approval date, the evaluation and the vendor's data terms.
Gildenmoor Language School chooses a drafting assistant
Gildenmoor Language School, an invented provider of evening language courses, evaluates 3 drafting assistants for course pages and enrolment emails. The trial uses 8 tasks, including 2 hard cases: a course page with contradictory timetable notes and a request to promise exam results. Two tutors and the marketing manager review 72 outputs blind. Tool A has the highest acceptance rate, 19 of 24, but its business plan's terms allow vendor staff to review inputs for quality purposes, so it is approved only for non-personal data. Tool B accepts 16 of 24, refuses the exam-results promise and offers contractual no-training terms with regional storage, but its logs cannot be exported. Tool C accepts 11 of 24. The school runs a 10-week pilot of Tool A for course pages only, with a defect rate target of under 1 serious defect per 10 pages, and asks Tool B's vendor about log export before reconsidering it for enrolment emails.
Common mistakes
- Judging from the demo. Vendor examples say little about your tasks; trial on your own work.
- Trialling with real customer data before reading the terms. Use non-personal or minimised data until the terms are reviewed.
- Comparing tools only with each other. The benchmark is current practice, review time included.
- Reading the consumer terms for a business purchase, or the reverse. Terms differ by plan.
- Accepting a settings toggle as a data commitment. Training opt-outs belong in the contract.
- Ignoring model changes. The tool you approved may behave differently next quarter; keep a regression set.
- Forgetting AI features in existing tools. They need the same evaluation even when nothing new is bought.
- Never asking how to leave. Export and deletion are easier to negotiate before signing than after.
