Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

AI in Marketing › Troubleshooting

AI Output Quality Getting Worse

Why prompts and assistants that used to pass review now need heavy editing: a model change, no test set, edited prompts, patched instructions and stale sources.
Troubleshootingintermediate12 min readUpdated

First, date the change and list everything that changed around it

Start from the review log rather than from impressions. The accuracy and review workflows guide has reviewers record, per item, the model, the version of the prompt and of the source pack, the defects found by failure mode and the time taken, and it treats defects climbing after a prompt or model change as the cue to roll back. Read that log for the point where defects per item or editing time began to climb, and for which kinds of defect grew: features that are not in the source, a voice that has gone flat, missing sections, the wrong order. Where no log exists, the dates on reviewer comments and the version history of the drafts give a rougher answer.

Then list everything that changed in the weeks around that date, one line each: the model behind the tool and any change the vendor announced; the prompt template and every shared context block it uses; the standing instructions, if the tool is an assistant; the documents it draws on; the tool's settings, such as length or creativity controls; and the briefs and input fields going in. Several lines usually change at once, which is why argument about the cause goes nowhere until the versions are tested.

Where a stored test set exists, rerun it against each version, changing one thing at a time, as the guide to testing prompts and models requires of any comparison. The NIST AI Risk Management Framework, as the site's framework page applies it, frames the work in two of its four functions: measure how the system actually behaves, then manage by changing the setup on that evidence.

This page covers gradual drift with no single error. One invented fact belongs to the diagnosis for AI content that contains invented facts, a customer given a wrong answer to the diagnosis for an AI assistant giving wrong answers, a wrong automated message to the diagnosis for AI automation that sent the wrong messages, and drafts that were generic from the start to the diagnosis for AI content that is not performing.

Diagnosis table

Diagnosis

SymptomCheckLikely causeFix
The decline began around a vendor update, and nobody knows which model produced which outputThe vendor's change notices and the tool's model setting, against the date in the review logThe vendor changed the model and nothing records the versionRecord the model version beside each prompt, rerun the regression set, pin or adjust
Reviewers noticed only when they started rewriting, and nobody can say how good the output used to beWhether stored test cases and recorded scores exist for the taskNo stored test set or baselineBuild a test set from accepted outputs, score the current setup and record the baseline
Output changed for everyone using the same template at onceThe template's and each context block's version history, owner and last testA shared prompt or context block edited without a version, a test or an ownerRestore or re-release the block with a version note, an owner and a passing test
An assistant follows some rules and ignores others, differently from week to weekThe instructions read top to bottom for overlapping or conflicting rules and stale factsStanding instructions patched rule by rule until they contradict each otherConsolidate the instructions into one coherent version with a test per rule
Answers or drafts mix old and new wording, or draw on the wrong sectionThe documents the tool reads: duplicates, recent reorganisation, owners and review datesSource documents restructured, duplicated or out of dateOne current document per fact; move older versions out; rerun the questions
The same template works for some people and not othersRecent inputs against the inputs in the test cases: audience, material, constraintsBriefs and inputs thinner than when the prompt was testedMake the required inputs named fields with sources, and check them before running

Fix, verify and prevent each cause

The vendor changed the model behind the tool

The guide to evaluating AI tools makes the point plainly: a tool's behaviour rests on a model the vendor is free to swap, sometimes without much notice, and a new version can change style, alter how prompts are handled or introduce new failure modes. Its advice is to ask how changes are announced, whether a version can be pinned or an upgrade delayed, and what notice comes before older versions are retired; to keep the trial tasks as a small regression set; and to record the model version beside prompts so that a change in output can be matched to a change of model. A Language Model swapped under a working prompt is a new combination that has never been tested.

Fix: record the current model version beside every prompt in use, rerun the regression set, and either pin the earlier version where the vendor allows it or adjust the prompt and retest until it passes.

Verify: the rerun scores return to the recorded pass bar, and the failure types that appeared with the change are gone from the log.

Prevent: write the model version into every review log entry, and ask about change notice and pinning at each renewal.

No stored test set or baseline

Drift that is noticed only when reviewers start rewriting is the signature of a team with nothing to compare against. The testing guide's method is a test set built from real work, typical, hard, edge and sensitive cases, each with notes on what good output must contain; a rubric written before any output is scored; a pass bar set before scoring; and a failure log kept beside the scores. The recorded result is the baseline.

Fix: build the set now from outputs that passed review with light edits before the decline, with their original inputs, and score the current setup against them.

Verify: two reviewers mark the same few outputs separately and broadly agree; where they do not, the rubric is unclear and gets rewritten before the baseline is recorded.

Prevent: store the set beside the prompt and rerun it on the guide's triggers: an edited prompt or shared block, a model change, changed tool settings, restructured source documents, or reviewers noticing a change in the errors they catch.

A shared prompt or context block edited without a version, a test or an owner

The tutorial on building a prompt library gives every entry a version, a change log, test cases and an owner, and gives shared blocks such as the voice, the audience or the claims list owners of their own, because a single edit to a block flows into every template built on it. That is also how one well-meant edit degrades every draft at once. The tutorial adds that personal copies drift from the shared version.

Fix: compare the current template and each block it uses with the last version that passed, restore or rewrite the block, and re-release it with a version note, a named owner and a passing test.

Verify: the template's test cases pass at the rate recorded before the decline, on several runs each, because output varies between runs.

Prevent: route block edits through the block's owner, rerun the test cases before any change is shared, and note which entry and version produced each reviewed output.

An assistant's standing instructions patched rule by rule

Instructions grow by patches: a complaint arrives, a rule is added; a test fails, a line is reworded. The tutorial on writing instructions for an AI assistant warns that tightening one rule can slacken another, which is why it reruns the whole test set after every change, and keeps facts out of the instructions because they go stale where no document owner sees them. The brand voice guide for AI adds that instructions that try to anticipate everything end up in conflict, leaving the model to obey whichever line it gives most weight.

Fix: read the instructions top to bottom, mark overlapping and conflicting rules, move facts into owned documents, and write one consolidated version with a test conversation for every rule.

Verify: the full test set passes on the consolidated version, including the conversations that tempt each rule.

Prevent: number and date each version of the instructions, name its owner, keep the history, note the test result behind each edit and rerun the full set whenever instructions, documents or model move.

The documents a grounded tool draws on restructured, duplicated or out of date

A tool that uses grounding writes from the passages it retrieves, and the tutorial on preparing documents for an AI assistant explains how a library degrades it: a superseded version that matches the wording can surface before the current page, a reorganisation splits a rule from its exception, and a document with no owner quietly goes stale. The output drifts because its material did.

Fix: list what the tool can reach, name a single current document for each fact, with its owner and last-reviewed date, move older versions out of the library instead of labelling them superseded, and rewrite split sections so each stands alone.

Verify: rerun the test questions and trace each answer or draft to the passage it drew on; a citation proves only which passage the model received, not that the sentence is backed by it.

Prevent: tie document updates to price, product and policy changes, and log library changes so a later decline can be dated to the change that caused it.

Briefs and inputs thinner than when the prompt was tested

A prompt is tested with good inputs and then used with whatever arrives. The guide to prompt patterns for marketing treats a prompt as a brief that has to carry an audience, a single task, material, constraints, a format and a quality bar, and notes that most weak output comes from prompts with those gaps, which the model fills with the most typical content. The library tutorial adds that a good template given an outdated specification writes confident copy that is out of date.

Fix: compare recent inputs with the inputs in the test cases, and make each required input a named field with its source written beside it.

Verify: rerun the same template with full inputs for a few recent jobs; if the output recovers, the inputs were the cause.

Prevent: check inputs against the field list before running, and add a thin-input case to the test set so the gap shows in a rerun.

Prevention: a stored test set, versions beside every prompt and a rerun on each change

Every cause above is a change nobody measured. Prevention makes every change measurable and every output traceable to the setup that produced it. Start with a stored test set for each recurring task and each assistant, built from real work, scored with a rubric written in advance and recorded as a baseline, with a pass bar that lists the failures unacceptable at any rate. Keep it beside the prompt or instructions it tests, so a rerun takes minutes of preparation rather than a project.

Next, record versions where they can be read together. Each reviewed output should be traceable to the template version, the context block versions, the instruction version, the model version and the state of the source library, which is what the accuracy guide's review log and the library's change logs make possible. When output shifts, the question "what changed?" is then answered by reading, not by memory.

Then rerun on every change. The testing guide's trigger list is the routine: a prompt or shared block edited, a model changed by the vendor or the team, tool settings changed, source documents restructured, or reviewers noticing a new kind of error. Treat the rerun as part of making the change, not as a check afterwards, and do not release a change that fails the pass bar. Voice belongs in the same routine: the brand voice guide reruns its test briefs after every change to the voice pack, since fixing one channel can break the next.

Finally, read the review log on a fixed rhythm. The main measure the accuracy guide names is defects per reviewed item, split by tier, and a climb straight after a prompt or model change is the moment to roll back while the change is still fresh. A slow rise with no logged change is the cue to look at the inputs and the documents. The AI troubleshooting section lists the neighbouring symptoms.

A letting agency finds why its listing drafts got worse (illustrative)

Illustrative example — not a real result

Ommerdax Lettings

Ommerdax Lettings, an invented letting agency with 4 branches, drafts property descriptions from a shared prompt. In the spring a description took about 5 minutes of editing; by early autumn it takes about 20, with features that are not on the listing sheet, the house style gone and rooms in the wrong order. The prompt library shows no change to the template.

The content lead dates the rise to the first fortnight of July from the review log and lists what changed around it: a negotiator in another branch edited the shared context block in late June, adding local selling points and dropping the room-order rule; the vendor moved the drafting tool to a newer model the same month; and a change to the listing sheet export had dropped the floor-plan field. With no stored test set, nobody can say which change mattered.

The agency builds a test set from 24 spring descriptions that passed with light edits, with their listing sheets, and runs each version 3 times. The current setup gives usable descriptions for 8 of the 24 listings on all 3 runs. Restoring the context block raises that to 15 of 24; adding the floor-plan field back raises it to 21 of 24. The model change accounts for a slightly longer average length, fixed with a length line. The block is re-released as version 4 with the content lead as owner, the model version is recorded beside the prompt, the floor-plan field is restored, and the set is rerun after every change from then on.

Common mistakes

  • Blaming the model first. It is one of several things that change; test each version before deciding.
  • Changing several things at once. Swap the model and the prompt together and the result cannot show which one mattered.
  • Fixing by adding more rules. Patched instructions start to contradict each other; consolidate instead.
  • Leaving old documents marked as superseded. A superseded page that matches the wording can still be used.
  • Editing a shared block without a test. One edit reaches every template that uses it.
  • Testing with ideal inputs only. The prompt then meets thinner briefs in daily use and fails quietly.

Frequently asked questions

Is the model simply getting worse, and should we switch tools?

A vendor can change the model behind a tool, and that is one possible cause, but the same gradual decline also comes from an edited prompt, patched instructions, stale documents and thinner briefs. Switching tools without finding out which of those changed usually carries the real cause across to the new tool. Rerun a stored test set against each version of the setup first. If the model change is what made the difference, a new tool is judged on the same test set and against the same pass bar as the old one.

We never kept a test set. Can we still work out what changed?

Partly. Build one now from outputs that passed review with light edits before the decline began, together with the inputs that produced them, and score the current setup against those accepted outputs. You may not be able to run the old model again, but you can rerun the current one with an earlier version of the prompt or context block, with the fuller inputs, and with the documents as they were, changing one thing at a time. What you build becomes the baseline for every later change.

Could the heavier editing just mean reviewers have become stricter?

It could, and the review log usually tells you. If the same kinds of defect appear at the same rate and only the editing time has grown, look at the reviewers' standard and the scoring rubric, and check that two reviewers score the same outputs alike. If new kinds of defect appear from a particular date, or old ones become frequent, something in the setup changed, and the list of what changed around that date is where to look.

Terms used on this page

Frameworks

Where this page is used

Sources

  1. AI Risk Management Framework — NIST

Related