First, date the change and list everything that changed around it
Start from the review log rather than from impressions. The accuracy and review workflows guide has reviewers record, per item, the model, the version of the prompt and of the source pack, the defects found by failure mode and the time taken, and it treats defects climbing after a prompt or model change as the cue to roll back. Read that log for the point where defects per item or editing time began to climb, and for which kinds of defect grew: features that are not in the source, a voice that has gone flat, missing sections, the wrong order. Where no log exists, the dates on reviewer comments and the version history of the drafts give a rougher answer.
Then list everything that changed in the weeks around that date, one line each: the model behind the tool and any change the vendor announced; the promptThe instruction and material given to an AI language model, which works like a brief: the model knows language, not the audience or product, unless told. template and every shared context block it uses; the standing instructions, if the tool is an assistant; the documents it draws on; the tool's settings, such as length or creativity controls; and the briefs and input fields going in. Several lines usually change at once, which is why argument about the cause goes nowhere until the versions are tested.
Where a stored test setA collection of real inputs for an AI task, each noting what a good output must contain, run to compare prompts or models and rerun when they change. exists, rerun it against each version, changing one thing at a time, as the guide to testing prompts and models requires of any comparison. The NIST AI Risk Management Framework, as the site's framework page applies it, frames the work in two of its four functions: measure how the system actually behaves, then manage by changing the setup on that evidence.
This page covers gradual drift with no single error. One invented fact belongs to the diagnosis for AI content that contains invented facts, a customer given a wrong answer to the diagnosis for an AI assistant giving wrong answers, a wrong automated message to the diagnosis for AI automation that sent the wrong messages, and drafts that were generic from the start to the diagnosis for AI content that is not performing.
Diagnosis table
Diagnosis
| Symptom | Check | Likely cause | Fix |
|---|---|---|---|
| The decline began around a vendor update, and nobody knows which model produced which output | The vendor's change notices and the tool's model setting, against the date in the review log | The vendor changed the model and nothing records the version | Record the model version beside each prompt, rerun the regression set, pin or adjust |
| Reviewers noticed only when they started rewriting, and nobody can say how good the output used to be | Whether stored test cases and recorded scores exist for the task | No stored test set or baseline | Build a test set from accepted outputs, score the current setup and record the baseline |
| Output changed for everyone using the same template at once | The template's and each context block's version history, owner and last test | A shared prompt or context block edited without a version, a test or an owner | Restore or re-release the block with a version note, an owner and a passing test |
| An assistant follows some rules and ignores others, differently from week to week | The instructions read top to bottom for overlapping or conflicting rules and stale facts | Standing instructions patched rule by rule until they contradict each other | Consolidate the instructions into one coherent version with a test per rule |
| Answers or drafts mix old and new wording, or draw on the wrong section | The documents the tool reads: duplicates, recent reorganisation, owners and review dates | Source documents restructured, duplicated or out of date | One current document per fact; move older versions out; rerun the questions |
| The same template works for some people and not others | Recent inputs against the inputs in the test cases: audience, material, constraints | Briefs and inputs thinner than when the prompt was tested | Make the required inputs named fields with sources, and check them before running |
Fix, verify and prevent each cause
The vendor changed the model behind the tool
The guide to evaluating AI tools makes the point plainly: a tool's behaviour rests on a model the vendor is free to swap, sometimes without much notice, and a new version can change style, alter how prompts are handled or introduce new failure modes. Its advice is to ask how changes are announced, whether a version can be pinned or an upgrade delayed, and what notice comes before older versions are retired; to keep the trial tasks as a small regression set; and to record the model version beside prompts so that a change in output can be matched to a change of model. A Language ModelAn AI system trained on large amounts of text that writes by predicting which piece of text is likely to come next. swapped under a working prompt is a new combination that has never been tested.
Fix: record the current model version beside every prompt in use, rerun the regression set, and either pin the earlier version where the vendor allows it or adjust the prompt and retest until it passes.
Verify: the rerun scores return to the recorded pass bar, and the failure types that appeared with the change are gone from the log.
Prevent: write the model version into every review log entry, and ask about change notice and pinning at each renewal.
No stored test set or baseline
Drift that is noticed only when reviewers start rewriting is the signature of a team with nothing to compare against. The testing guide's method is a test set built from real work, typical, hard, edge and sensitive cases, each with notes on what good output must contain; a rubric written before any output is scored; a pass bar set before scoring; and a failure log kept beside the scores. The recorded result is the baselineAn estimate of what a metric would have done without a given change, campaign or promotion, against which the effect is measured..
Fix: build the set now from outputs that passed review with light edits before the decline, with their original inputs, and score the current setup against them.
Verify: two reviewers mark the same few outputs separately and broadly agree; where they do not, the rubric is unclear and gets rewritten before the baseline is recorded.
Prevent: store the set beside the prompt and rerun it on the guide's triggers: an edited prompt or shared block, a model change, changed tool settings, restructured source documents, or reviewers noticing a change in the errors they catch.
A shared prompt or context block edited without a version, a test or an owner
The tutorial on building a prompt library gives every entry a version, a change logA dated record of changes made to tracking, accounts, reports, lists or rules, with who made each and why, so later movements in the figures can be traced., test cases and an owner, and gives shared blocks such as the voice, the audience or the claims list owners of their own, because a single edit to a block flows into every template built on it. That is also how one well-meant edit degrades every draft at once. The tutorial adds that personal copies drift from the shared version.
Fix: compare the current template and each block it uses with the last version that passed, restore or rewrite the block, and re-release it with a version note, a named owner and a passing test.
Verify: the template's test cases pass at the rate recorded before the decline, on several runs each, because output varies between runs.
Prevent: route block edits through the block's owner, rerun the test cases before any change is shared, and note which entry and version produced each reviewed output.
An assistant's standing instructions patched rule by rule
Instructions grow by patches: a complaint arrives, a rule is added; a test fails, a line is reworded. The tutorial on writing instructions for an AI assistant warns that tightening one rule can slacken another, which is why it reruns the whole test set after every change, and keeps facts out of the instructions because they go stale where no document owner sees them. The brand voice guide for AI adds that instructions that try to anticipate everything end up in conflict, leaving the model to obey whichever line it gives most weight.
Fix: read the instructions top to bottom, mark overlapping and conflicting rules, move facts into owned documents, and write one consolidated version with a test conversation for every rule.
Verify: the full test set passes on the consolidated version, including the conversations that tempt each rule.
Prevent: number and date each version of the instructions, name its owner, keep the history, note the test result behind each edit and rerun the full set whenever instructions, documents or model move.
The documents a grounded tool draws on restructured, duplicated or out of date
A tool that uses groundingMaking a language model answer from specified material, such as documents retrieved for the question, rather than from what it absorbed in training. writes from the passages it retrieves, and the tutorial on preparing documents for an AI assistant explains how a library degrades it: a superseded version that matches the wording can surface before the current page, a reorganisation splits a rule from its exception, and a document with no owner quietly goes stale. The output drifts because its material did.
Fix: list what the tool can reach, name a single current document for each fact, with its owner and last-reviewed date, move older versions out of the library instead of labelling them superseded, and rewrite split sections so each stands alone.
Verify: rerun the test questions and trace each answer or draft to the passage it drew on; a citation proves only which passage the model received, not that the sentence is backed by it.
Prevent: tie document updates to price, product and policy changes, and log library changes so a later decline can be dated to the change that caused it.
Briefs and inputs thinner than when the prompt was tested
A prompt is tested with good inputs and then used with whatever arrives. The guide to prompt patterns for marketing treats a prompt as a brief that has to carry an audience, a single task, material, constraints, a format and a quality bar, and notes that most weak output comes from prompts with those gaps, which the model fills with the most typical content. The library tutorial adds that a good template given an outdated specification writes confident copy that is out of date.
Fix: compare recent inputs with the inputs in the test cases, and make each required input a named field with its source written beside it.
Verify: rerun the same template with full inputs for a few recent jobs; if the output recovers, the inputs were the cause.
Prevent: check inputs against the field list before running, and add a thin-input case to the test set so the gap shows in a rerun.
Prevention: a stored test set, versions beside every prompt and a rerun on each change
Every cause above is a change nobody measured. Prevention makes every change measurable and every output traceable to the setup that produced it. Start with a stored test set for each recurring task and each assistant, built from real work, scored with a rubric written in advance and recorded as a baseline, with a pass bar that lists the failures unacceptable at any rate. Keep it beside the prompt or instructions it tests, so a rerun takes minutes of preparation rather than a project.
Next, record versions where they can be read together. Each reviewed output should be traceable to the template version, the context block versions, the instruction version, the model version and the state of the source library, which is what the accuracy guide's review log and the library's change logs make possible. When output shifts, the question "what changed?" is then answered by reading, not by memory.
Then rerun on every change. The testing guide's trigger list is the routine: a prompt or shared block edited, a model changed by the vendor or the team, tool settings changed, source documents restructured, or reviewers noticing a new kind of error. Treat the rerun as part of making the change, not as a check afterwards, and do not release a change that fails the pass bar. Voice belongs in the same routine: the brand voice guide reruns its test briefs after every change to the voice pack, since fixing one channel can break the next.
Finally, read the review log on a fixed rhythm. The main measure the accuracy guide names is defects per reviewed item, split by tier, and a climb straight after a prompt or model change is the moment to roll back while the change is still fresh. A slow rise with no logged change is the cue to look at the inputs and the documents. The AI troubleshooting section lists the neighbouring symptoms.
A letting agency finds why its listing drafts got worse (illustrative)
Ommerdax Lettings
Ommerdax Lettings, an invented letting agency with 4 branches, drafts property descriptions from a shared prompt. In the spring a description took about 5 minutes of editing; by early autumn it takes about 20, with features that are not on the listing sheet, the house style gone and rooms in the wrong order. The prompt library shows no change to the template.
The content lead dates the rise to the first fortnight of July from the review log and lists what changed around it: a negotiator in another branch edited the shared context block in late June, adding local selling points and dropping the room-order rule; the vendor moved the drafting tool to a newer model the same month; and a change to the listing sheet export had dropped the floor-plan field. With no stored test set, nobody can say which change mattered.
The agency builds a test set from 24 spring descriptions that passed with light edits, with their listing sheets, and runs each version 3 times. The current setup gives usable descriptions for 8 of the 24 listings on all 3 runs. Restoring the context block raises that to 15 of 24; adding the floor-plan field back raises it to 21 of 24. The model change accounts for a slightly longer average length, fixed with a length line. The block is re-released as version 4 with the content lead as owner, the model version is recorded beside the prompt, the floor-plan field is restored, and the set is rerun after every change from then on.
Common mistakes
- Blaming the model first. It is one of several things that change; test each version before deciding.
- Changing several things at once. Swap the model and the prompt together and the result cannot show which one mattered.
- Fixing by adding more rules. Patched instructions start to contradict each other; consolidate instead.
- Leaving old documents marked as superseded. A superseded page that matches the wording can still be used.
- Editing a shared block without a test. One edit reaches every template that uses it.
- Testing with ideal inputs only. The prompt then meets thinner briefs in daily use and fails quietly.
