First, trace one wrong message back through trigger, data, prompt, model, checks and send
This page starts where the guide to handling an AI mistake that reached customers reaches its fourth move. That guide's first three, contain the source, correct what customers saw and tell the people who need to know, are assumed done or under way, and are not repeated here. The trace is the move that decides whether the incident repeats, and the incident guide gives it a shape, treating the output as the end of a chain and walking it one link at a time, from the input data and the source material through the prompt or template, the model and its settings, the automation logic, the review step and the send. In most incidents one cause creates the error and another lets it through, and repairing only the first leaves a gap the next error can pass through.
Pick one wrong message and reconstruct its run. What triggered it, and was the trigger the event the workflow was built for? Which record fields reached the prompt, and were they right at the moment of sending? Which source material did the model see, and was it the current version? What did the prompt allow the model to write, and what did it allow it to change? Which automated checks ran on the output, and which were missing? What permitted the send: a rule, an agent's own decision or a person? Open the logs the automation guide asks every acting step to keep, the input the model saw and the reason it gave, and read the run in that order. If the logs do not exist, that is the first finding.
Then carry the message back with the five whys. The email offered a discount because the offer line was composed by the model; the model composed it because the template left the offer line open; the line was open because the brief treated the offer as copy rather than as a rule; nobody noticed because no sample of sent messages was being read; nothing bounded the damage because the send ran without a volume alarm. The fix sits at the last why for each branch, and the guardrail that should have stood there is what the diagnosis table names. The NIST AI Risk Management Framework, as the automation guide applies it, encourages the habit this trace depends on: treat every model as a component with known failure modes rather than as a clever colleague, map where it is used, measure how it behaves and manage what the measure shows. The applications section sets automation beside the other uses of AI and the review cost each carries; this page assumes the message has already gone out.
If the batch reached people who had opted out, keep that branch separate. The tutorial on fixing a mistake in a sent email treats mail sent to people who had opted out as a consent matter before it is anything else, and the ICO's direct marketing guidance, as the site's guides state it, explains the UK rules on electronic marketing, including offering a way to opt out in every message. That branch ends at the suppression rail, not at the prompt, and belongs with whoever handles data protection.
Diagnosis table
Diagnosis
| Symptom | Check | Likely cause | Fix |
|---|---|---|---|
| The message carries a discount, price, claim, legal line or footer that was never approved | Which parts of the template were fixed and which the model was allowed to write | Offers, prices, claims or legal copy left to the model instead of locked as rules | Lock those blocks as rules and confine the model to approved variants or a tight brief |
| Messages went out, or records changed, with no person or rule in between | What permitted the send or the change: a rule, an approval step or the agent's own decision | An agent allowed to send or change records where a fixed workflow with an approval step was needed | Move sending and record changes to rules with an approval step; grant the agent permissions per action |
| A whole segment received the bad batch inside an hour, and nobody was alerted | The send volume for the hour against the usual rate, and whether any limit or alarm existed | No rate limit, dry run or volume alarm, so a bad hour reached a whole segment | Add a rate limit that pauses for review, a dry-run mode and a volume alarm to the owner |
| The tone or wording has been drifting for weeks and the complaint is the first anyone heard | When a person last read a sample of sent messages, and what the sample found | No weekly sample of sent messages read by a person, so drift went unnoticed | Set a sampling plan with a cadence, a reader, a failure log and a pause threshold |
| The message states a promotion, policy or product fact that was true once | Which document the model composed from, its version and its owner | The model composing from stale or wrong source material | Keep one current source per fact, archive the rest and tie updates to price and policy changes |
| The problem began on the day the prompt, model or vendor changed | The change log for the workflow and the date the regression set was last run | A prompt, model or vendor change made without rerunning the regression set | Rerun the stored test set against the recorded baseline before any change goes live |
Fix, verify and prevent each cause
Offers, prices, claims or legal copy left to the model
The AI for marketing automation guide draws the line plainly: offers, prices, legal copy, claims about the product and the unsubscribe footer are fixed, and the model chooses between approved variants, or writes within a tight brief, only for the lines where tone and relevance matter and an error is survivable. A template that leaves the offer line open invites the model to compose one, and a model composes fluently whether or not the offer exists. The same applies to a merge fieldA placeholder in an email or message template replaced with a value from each contact's record, such as a first name, with a fallback when it is empty.: the guide to personalising email content gives every field a fallback written for its sentence, and a field the model is allowed to rewrite is no longer a field.
Fix: rebuild the template so that offers, prices, claims, legal lines, footers and merge fields are rules the model cannot touch, and give the model a short list of approved opening lines or a brief with the facts it may use and nothing else.
Verify: generate the message for a seed contact with the model instructed to be as creative as it can be, and confirm the locked blocks come out identical every time while only the permitted lines vary.
Prevent: run every generated element through the automated checks the automation guide lists before send, banned terms, claim patterns, length, links that resolve and placeholders left unfilled, and treat any template that opens a locked block to the model as a reviewable change.
An agent allowed to send or change records where a fixed workflow was needed
The comparison of rule-based automation and AI agents answers the question most incidents of this kind turn on: anything that sends, spends or changes records belongs to rules, and an agent is used only where the steps cannot be written down in advance and a person approves the result before it reaches a customer. An agent that was given the whole task, qualify, write, send and update, has been given permissions the comparison would refuse, and its mistakes are varied rather than systematic, which is why the same run can send a good message to one customer and a wrong one to the next.
Fix: separate drafting from acting, as the automation guide puts it: the agent may read, classify and draft, and a rule or a person sends, changes the stage, applies the offer or deletes anything, with permissions granted per action rather than per agent.
Verify: run the workflow with sending switched off and read what the agent proposes for a varied set of records, several runs each, before any action is switched back on; then confirm every action it keeps is logged with the input it saw and the reason it gave.
Prevent: list every agent in the automation inventory with its permissions, as the governance guide asks, name an owner who reads its logs and can pause it, and treat each new task it is given as a new deployment with its own test and its own permissions.
No rate limit, dry run or volume alarm
A workflow that can send to everyone it matches will do so in the hour the fault appears. The automation governance guide names the volume alarm as the place most quiet failures become visible first: a workflow that sends far more or far fewer messages than usual in a day should notify its owner, and a scheduled report comparing the day's counts with the recent average does the job where the platform offers nothing. The automation guide adds the two controls that turn a bad hour into a bad minute for an acting step: a rate limit, so a workflow can take only a bounded number of actions before pausing for review, and a dry-run mode that shows what it would have done.
Fix: add a rate limit that pauses the workflow for review once a bounded number of sends has gone out, a dry-run mode the owner can switch on after any change, and a volume alarm to the workflow's owner.
Verify: raise the trigger for more seed contacts than the limit allows and confirm the workflow stops and asks, and that the alarm reaches a person who will act on it the same day.
Prevent: make the rate limit, the dry run and the alarm part of the inventory row for every workflow with an AI step, and check them in the governance guide's weekly read of volume alarms; the frequency cap and the global suppression check every send should pass stay as rails on every workflow.
No weekly sample of sent messages read by a person
A model that composes at send time produces messages nobody on the team has seen, and the automation guide's answer is a sample of sent messages read by a person each week, because that is the only way to catch a model whose tone has drifted. The accuracy and review workflows guide sets the customer-facing and automated tier's review as exactly this: guardrails in the system, a sampled review on a schedule, an incident route and an owner who reads the logs, with the sampling plan deciding the sample size, the cadence and who reads it, logging what the sample found by failure mode, and setting a threshold at which the use pauses for a full review. Drift that ran for weeks is the signature of a sample nobody read.
Fix: write the sampling plan down, name the reader, start the failure log, and set the pause threshold before the workflow is switched back on.
Verify: read the first week's sample against the brand's voice guidelines and the locked facts, and confirm the log records each defect by failure mode with the prompt and model version beside it.
Prevent: keep the human reviewThe step where a named person checks AI-assisted work for accuracy, sources, originality, privacy and disclosure before it is published or sent. at that tier permanent rather than a launch-week habit, and read the failure log monthly for the rise in defects that signals a change nobody announced.
The model composing from stale or wrong source material
A message can be wrong because the model did its job on the wrong input. The accuracy guide calls these inherited errors, an old spec or a wrong field reproduced with confidence, and the tutorial on preparing documents for an AI assistant shows how they start: a library holding old and new versions of the same promotion, a document with no owner and no review date, a policy that changed while the article did not, and a superseded passage that matches the template's wording better than the current one. A model asked to compose from that library will quote whichever version it is given.
Fix: list every document the workflow's model can reach, pick one current source per fact with its owner, archive every other version outside the library rather than marking it superseded, and give each remaining document an owner and a last-reviewed date.
Verify: generate the message for a seed contact and trace every fact in it to the current source; a citation shows what the model was given, not that the sentence is supported, so open the passage.
Prevent: tie document updates to the processes that change prices, promotions and policies, so the owning document is updated and the workflow's test questions rerun before the change goes live, and keep a change log of the library so a later wrong message can be traced to the day it started.
A prompt, model or vendor change made without rerunning the regression set
The automation guide calls the regression test set, a fixed collection of inputs with known good outputs rerun whenever the prompt, model or vendor changes, the cheapest control on its list and the one most teams skip. The guide to testing prompts and models explains why it is needed: vendors change the models behind their tools, sometimes quietly, so a prompt that once passed can start failing. The guide's trigger list covers an edited prompt or shared context block, a model change announced or chosen, changed tool settings, restructured source documents, and reviewers noticing a new kind of error. A workflow whose wrong messages began on the day of such a change has told you its cause.
Fix: rerun the stored test setA collection of real inputs for an AI task, each noting what a good output must contain, run to compare prompts or models and rerun when they change. against the recorded baseline, apply the pass bar agreed in advance, including the failures that are unacceptable at any rate such as an invented price or offer, and roll back or repair the prompt until it passes.
Verify: compare the new failure log with the baseline's and confirm the failure type that produced the incident no longer appears; then switch the workflow back on in dry-run mode for its first runs.
Prevent: put the rerun inside change controlThe routine a proposed change to a live system follows: written down first, reviewed, tested, signed off and recorded before it reaches customers or data., so no prompt, model, vendor or settings change reaches a live workflow without a logged rerun and an owner's sign-off, and store the test set beside the prompt so a rerun is a routine task rather than a project.
Prevention: locked blocks, permission per action, a rate limit and a regression set before any change
Every cause above is a guardrail the automation guide describes and the workflow lacked, and the prevention routine is to put all of them in front of every workflow with an AI step rather than only the one that failed. The incident guide's last move, log it and change the system, and the governance guide's last incident step, check whether the same cause exists elsewhere in the inventory, are the same instruction: walk the inventory and ask of each workflow which of the six guardrails it carries. The AI automation go-live checklist puts those guardrails into one gate to tick before the paused workflow is switched back on and before any new AI step goes live.
Start with the template. Offers, prices, claims, legal lines, footers and merge fields are rules; the model chooses between approved variants or writes within a brief that lists the facts it may use. Every generated element passes the automated checks before send. A template that opens a locked block to the model is a reviewable change with a second person's sign-off, in the same way the governance guide treats a change to the suppression list or the frequency cap.
Then the permissions. Sending, spending and record changes are rules with an approval step; an agent drafts, classifies and proposes, and holds permissions per action, logged with the input it saw and the reason it gave. Each agent has an inventory row listing its permissions, an owner who reads its logs weekly and can pause it, and a rule that each new task is a new deployment. The comparison's four questions, whether the task sends, spends or changes records, whether the steps can be written down, whether a person can check the result before it matters, and whether the team can test, monitor and own it, are asked again whenever the task grows.
Then the limits. A rate limit that pauses for review, a dry-run mode for the first runs after any change, a volume alarm to the owner, the frequency cap and the global suppression check before every send. The alarm, the cap and the suppression check are the governance guide's rails, applied to every workflow rather than left to each designer; the rate limit and the dry run are the automation guide's controls for any step that acts. Read the volume alarms weekly in minutes, and treat an unusual hour as something to open the same day.
Then the evidence. A sampling plan per automated use, with its reader, cadence, failure log and pause threshold, and a stored test set per prompt, rerun under change control whenever the prompt, model, vendor, settings or source documents change, with the result logged and signed off. Pick a KPIA metric tied directly to an objective. at each of the three levels the automation guide measures: the share of outputs approved unedited or passing the sample at the component level, incidents and their severity at the operational level, and the flow's own outcome against a holdout at the workflow level, because a flow reporting no incidents and almost no review time is usually one nobody watches. Keep the documents the model composes from under the same discipline, with one current source per fact, an owner and a review date, and a line in every price, promotion and policy change that updates the document and reruns the tests before the change goes live. When the same gaps show up in an assistant answering customers, the diagnosis for an AI chatbot giving wrong answers traces its own chain. When the model's output is ad creative a platform has turned down rather than a message a customer has received, the diagnosis for rejected AI-generated ad creative applies. The AI troubleshooting section lists the neighbouring symptoms. The tutorial on writing a sampling plan for an automated AI use walks through a sampling plan for one use, from naming the reader to setting the pause rule.
A holiday park operator finds the two gates its retention flow lacked (illustrative)
Molquessa Holiday Parks
Molquessa Holiday Parks, an invented operator of 12 holiday parks, runs a retention workflow that sends a check-in email to past guests whose usual booking window is approaching. The model composes the opening line and a suggested-stay paragraph at send time inside a template. One evening the workflow sends 4,180 emails in a little over an hour. Guests reply within minutes: 1,930 of the emails offer 20% off the next stay, which the parks never agreed, and 212 address the guest by another guest's first name.
The team pauses the workflow, sends a correction the next morning and honours the discount for anyone who books within the week. Then it traces one message. The offer line was composed by the model from a promotions document in the shared folder that still described a spring campaign, alongside the current one; the greeting came from a merge field the template allowed the model to rewrite, and in 212 runs the model wrote a name from the suggested-stay paragraph's example instead; and the send ran at full speed with no rate limit and no volume alarm, 3 days after the vendor updated the model behind the tool, a change nobody had tested.
The fixes follow the trace. The offer block and the footer become fixed rules; the greeting becomes a merge field with a fallback the model cannot touch; the model is restricted to choosing one of 6 approved opening lines and composing the suggested-stay paragraph from a brief that names the parks and nothing else. The spring promotions document is archived outside the folder and the current one gets an owner and a review date. A rate limit pauses the workflow for review after 300 sends in an hour, a volume alarm goes to the workflow's owner, and 25 sent messages are read every Monday with each defect logged by type. The team stores a regression set of 40 guest records with known good outputs and reruns it after any prompt or model change; the first rerun, after the next vendor update 7 weeks later, catches 2 outputs in which the model tried to add a discount to the suggested stay, before the workflow is switched back on.
Common mistakes
- Fixing the prompt and switching the workflow back on. The prompt created the error; the missing guardrail let it through, and it is still missing.
- Leaving the offer line, the legal line or a merge field open to the model. Anything the model can rewrite, it will.
- Giving an agent the whole task. Sending and record changes are rules with an approval step; the agent drafts.
- Running a send-time model at full speed with no limit. A bounded number of actions before a pause turns a bad hour into a bad minute.
- Treating the launch-week sample as the review. The sample is permanent, with a reader, a log and a pause threshold.
- Trusting a document because it is in the folder. One current source per fact, with an owner and a date; the rest archived outside.
- Letting a vendor's model update go live untested. The stored test set, rerun under change control, is how the change is noticed before customers notice it.
- Tracing only the workflow that failed. The same guardrail is usually missing elsewhere in the inventory.
