Why a launch-week read is not a plan
The accuracy and review workflows guide makes a short claim about automated uses: the sampling plan is the review. The AI automation go-live checklist turns it into a line to tick that asks for a named reader, a rhythm, a log and a rule for when the use stops, and the diagnosis of an automation that sent the wrong messages gives writing the plan down as the fix when no one reads a sample. None of the three says how to write one, and without a plan the sampling reviewA person reading a random sample of AI answers or generated copy on a fixed rhythm, plus every case that ended badly, and routing findings to the owners. has no fixed reader, draw or stopping point: the sample can drift to whatever the reader finds interesting, each finding can be fixed in the one item that was drawn, and nobody can say at what point the use should be switched off.
A written plan answers those questions in advance. It names the reader, defines a failure, says how the sample is drawn and on what schedule, says where a finding goes and what happens to everything that shares its cause, and sets the point at which the use pauses. It is short and it belongs to one use. A business with several automated uses writes one plan for each and records each on the use's row of the register.
The plan is not a test. A test setA collection of real inputs for an AI task, each noting what a good output must contain, run to compare prompts or models and rerun when they change. is chosen in advance, while the sample is drawn from what the live use has produced. It is not an experiment either, and it does not produce a score for a person or a tool. And it does not replace a check before use where one is possible: output a person reads before it is published or sent is governed by the content review checklist, and a statement that could harm someone is either locked as fixed text or read in full.
Before you start
- A row for the use in the register. The AI use register template records the tier, the owner, what the use can do and the review route. The plan is what the review route points to, so write the row first if it does not exist, and say in it whether the use drafts, publishes after review or acts on its own.
- A pass bar with its list of unacceptable failures. The guide to testing prompts and models asks for one before any output is scored: a minimum for what counts as usable, plus a list of failures that are not tolerated whatever their rate. The pause rule in step nine is built on that list, so a use that went live untested gets its test set and pass bar first.
- A reference for every item. Something the reader can open to see what the item should have said: the source documents, the locked blocks, the approved variants, the data row it was written from. Without one, the reader has nothing to set a statement against. The tutorial on preparing a document library builds the reference for an assistant.
- An owner who can pause the use. The owner holds an inventory rowThe single list of every active automated workflow, kept outside the platform, with its purpose, owner, trigger, exits, dependencies, measure and review dates. for the use and can pause it, as the go-live checklist asks of the person who ticks it, with the row kept as the automation governance guide describes.
- A place for the log. A sheet or a ticket queue, kept beside the register row. It needs no tool of its own.
- A reader with time. The reading has to be a protected task in someone's week, not something done between other jobs.
The steps
Name the use and decide what counts as an item
Write the use in one sentence, saying what it produces and where the output goes. Then name the unit the reader will open: a sent message with the data it was composed from, a label with the text it labelled, an assistant conversation from the first message to the last, a generated description with the attribute row behind it, an agent's run with its log. The unit has to be something that can be found again, so each one carries an identifier, a date and the versions of the prompt, the settings and the sources behind it. Then sort the output into the groups that differ in the way that matters for errors: the template, the category, the channel, the language, the contact reason, the segment. The groups are what the sample is spread over in step five, and a use that treats all its output as one pile is read as if it were one.
Write down what counts as a failure
Take the failure modes from the accuracy guide and from the failure column of the automation guide's guardrail table, and keep the ones that can occur in this use: a fabricated or outdated fact, an unsupported claim, drift from the brief, an error inherited from the input, an omission, an inconsistency between items, an off-brand tone, a wrong label, a link that does not resolve. Write each in words a reader can check against a single item, such as "names a product that is not in the source it was given", not "inaccurate". Then give each a class: unacceptable at any rate, which is the pass bar's list; to be fixed at its cause; or to be noted. The testing guide's examples of the first class are invented prices, safety claims and legal wording, so keep anything that touches those out of the other two classes. Use the same names in the log every time, so that counts from different weeks can be added up and compared.
Name what each item is checked against
Match every failure to the reference that shows it. A sent message is compared with the locked blocks, the approved variants and the facts it was allowed to use; a generated description with its attribute row; an assistant's answer with the passage it cited and with whether that passage was correct and current; a label with the written definition of the label; an analysis with the query and its result. Where the reader finds a statement the reference does not support, it counts as a failure whether or not it happens to be true. As general practice, if a use has no reference, say so in the plan and make building one the first task, because a sample read against nothing can find fluent text and call it fine.
Choose a reader who did not set the use up
The review guide keeps an operator from reviewing the output they produced once a use is above the internal tier, and the same reasoning applies here: the person who wrote the prompt reads what they meant, not what was sent. As general practice, name a reader who knows the subject well enough to see a plausible error, name a second who covers absences and joins calibration, and agree the time in the diary. In calibration both read the same items, mark them separately, compare and settle borderline cases in writing, as the quality review of support conversations does for human agents. The reading is the reader's work, done by named people on named dates and not left to whoever has a spare hour.
Draw the sample in groups and add the cases that went wrong
Draw at random within each group, and make sure every group is represented, because a hand-picked sample follows what the reader already worries about. Weight the draw towards what is new or changed: a new template, recently edited instructions, the records with the least data behind them, where invention concentrates, as the guide to AI product descriptions advises for a catalogue. On top of the random draw, add every case that visibly went wrong: a conversation passed to a person, one that drew a complaint, one that earned a poor rating and, as general practice, any other item a customer questioned. Report these cases separately from the random ones, so the failures chosen for being failures are not mistaken for a rate. Output where an error could harm someone is read in full. For the size, no Sample SizeThe number of users or sessions an experiment needs to detect a given effect reliably. needs calculating: the routine exists to find causes, so the number is the amount the reader can read carefully, kept steady between routine reads and, as general practice, increased after a change or an incident. The guide to AI in customer service lays out the weekly conversation sample and the extra cases added to it.
Set the rhythm and the extra reads
The rhythm follows how fast the use produces output and how far a wrong item travels before anyone sees it. The site's guides name a weekly read for sent messages and assistant conversations, and that is a reasonable starting point as general practice: a use that reaches a large audience quickly is read sooner than one that updates a page slowly, and a new use is read more closely than an established one. Then write the triggers for an extra read, taken from the testing guide's list of reasons to rerun a test set: a rewritten prompt or edited shared block, a model change made by the vendor or the team, a change in settings, a restructured source, a change in the kind of errors reviewers have started to see. As general practice, the first outputs after any such change are read before the routine read comes round. Put each date in the reader's diary, because a read that relies on someone remembering can be skipped in a busy week.
Log each finding by failure mode, with the versions behind it
Keep one log per use. For each item read, record the failure modes found, using the names from step two, and the decision. The log also holds the fields the accuracy guide asks for: the tier, which model and prompt produced the item, which version of the sources it drew on, who read it and how long the reading took. Add the identifier of the item, the cause once it is known and the owner it went to. As general practice, a read with no findings is logged too, with what was read and against what, because a log of only bad news cannot show that the plan is being followed. The log is read monthly, as the accuracy guide's monthly hour and the monthly read in the AI governance operations playbook describe, and the plan says who joins that read.
Trace each finding to its cause and fix everything that shares it
A finding is evidence, and the fix goes to its cause. The troubleshooting section sorts causes into five layers, source, prompt, model, review and permission, and each layer has an owner who can change it, so the AI troubleshooting hub is the place to look when the layer is not obvious. Keep asking why until you reach a cause someone has the authority to change; the five whys is the method. Then find every other item that shares the cause and put it right, not only the one that was drawn, whether by regenerating or correcting it. As general practice, add the failing case to the stored test set if it is not there yet, and rerun the set against the pass bar before the fix goes live. Also as general practice, close the finding in the log only when later reads show the failure gone.
Set the pause rule and say what pausing does
Write two kinds of rule. The first is a single event: as general practice, any failure on the unacceptable list pauses the use at once. An item that reached a customer wrongly with a failure to be fixed at its cause is handled as an incident under the guide to handling an AI mistake, which starts by containing the source; as general practice, where that source is one document the owner can correct, correcting it, with the faulty version kept for the trace, is the containing move, and the plan says in advance whether the use also pauses while that is done. The second is drift. Agree before the work starts that the owner must act once a failure mode moves beyond a range, which is the idea of management by exception, and size that range from what a miss would cost and from the use's own history, in the way a control chart takes its limits from a metric's own history, treating early limits as provisional and recomputing them as reads accumulate. The rule is agreed before any result is read, as the bound on a guardrail metricA metric an experiment must not worsen beyond an agreed bound, watched beside the primary metric that decides the test. is in a test. Put no figure in the plan until the first reads have given you one. Then write what pausing does for this use, as general practice: a composing step is switched off and a fixed template sent instead, a workflow moves to dry run, an assistant is limited to handing over, labels route to a person, drafts return to approval of every item. Name who may pause without asking, who is told and how the response follows contain, correct, tell, trace and log, and say, as general practice, that a restart waits for the cause fixed, the test set rerun against the pass bar, a read of the first outputs after restart and the owner's sign-off on the register row.
Decide how the plan loosens and tightens, and where it is kept
A new use begins with a person approving each output, and the guide to AI in automation lays out how routine groups then move to sampling while serious consequences stay on approval and some review stays in place. Write down how this use will follow that path. As general practice, the plan tightens after a change or an incident, and it loosens group by group, only once the reads stay free of the failures that matter. It does not loosen for the unacceptable list. Point the review route cell of the register row to the plan, tick the line on the go-live checklist, and set the date on which the plan itself is reviewed. The NIST AI Risk Management Framework describes four functions, govern, map, measure and manage; this plan is the measuring and managing for one use.
Report causes and counts, not a score
Report what the reads found by failure mode and cause, with the counts behind every rate, and read small counts as the reading small numbers guide advises: small counts of items are a poor basis for ranking. Do not make the number of defects found a target for the reader or the owner. Goodhart's law describes what can follow, a measure that tends to lose its value once it is rewarded, and a reader who is judged on finding few defects may find few. As general practice, pair the count with a measure the reader does not control, such as repeat contacts, complaints or unsubscribes, and read the two together.
The plan on a page: a table of fields
The plan fits on one page once the steps are done. The table lists its fields so the page can be checked for gaps; the register row points to it, and the page itself holds the detail.
| Field | What it records | Step |
|---|---|---|
| Use and unit | One sentence for the use, and what the reader opens | Name the use |
| Groups | The splits that matter for errors, such as template, category, channel, language | Name the use |
| Failure list and classes | Each failure in checkable words, marked unacceptable, fix at the cause or note | Write down what counts as a failure |
| Reference | What each item is compared with, per failure | Name what each item is checked against |
| Reader, second reader, cover | Named people and diary time; the builder is not the reader | Choose a reader |
| Sample rule | Random within groups, weight to what is new or changed, full read where harm is possible, every flagged case apart | Draw the sample |
| Rhythm and extra reads | The routine date and the changes that trigger a read | Set the rhythm |
| Log fields | Item, versions, failure modes, cause, owner, decision, time | Log each finding |
| Cause owners | Who fixes source, prompt, model, review and permission problems | Trace each finding |
| Pause rule | Single events and drift, from the use's own history | Set the pause rule |
| What pausing does, and the restart | The fallback, who is told, what must be true to restart | Set the pause rule |
| Loosening and tightening | When groups move between approval and sampling | Decide how the plan loosens |
| Review date | When the plan itself is read again | Decide how the plan loosens |
| Report | Failure modes and causes with the counts behind them, and no score for a person or a tool | Report causes and counts |
How the plan changes with the use: a table by kind of use
The same fields hold across the uses below; the unit, the reference and the fallback differ. The table sets out some shapes, as general practice, so that a reader writing a plan can see which entries to change.
| Use | One item is | Checked against | Groups to cover | Added to every draw | What pausing does |
|---|---|---|---|---|---|
| Copy composed at send time | A sent message with the data behind it | Locked blocks, approved variants, the facts the step may use | Template, segment, language | Messages that drew a complaint, and a heavier draw for a new template | The composing step is off and a fixed template goes out |
| Classification and routing | A label with its input text | The written definition of each label | Each label, and the "unknown" route | Labels that suppress or escalate | Everything routes to a person |
| Assistant answering customers | A whole conversation | The cited passage, and whether it was correct and current | Contact reason, channel | Handovers, complaints, poor ratings | The assistant only hands over |
| Generated catalogue copy | A description with its attribute row | The row, the banned list | Category, template, thin records | Categories where an error could harm someone | Regeneration stops and the live copy is checked |
| Agent that acts on records | A run with its log | The permitted actions and the record before and after | Action type | Every action that changed or deleted a record | Actions are limited to drafting |
A group of bagel shops reads its pre-order emails (illustrative)
Kavrul Bagels writes the plan for its pre-order emails
Kavrul Bagels, an invented group of 6 bagel shops, sends pre-order reminder emails that include a paragraph about the day's specials, composed by a model from the shop's menu sheet inside a fixed template. The allergen line and the prices are locked blocks filled from the sheet, and the model composes only the specials paragraph. The operations lead, who did not write the prompt, is named as reader, with a second reader for absences; the owner of the register row can pause the step. The unit is one sent email with its menu row; the groups are the shop and the template. Failures are written in checkable words. "Touches the allergen line or a price" and "states an allergen detail anywhere outside the locked block" are marked unacceptable; "names a bagel the shop is not baking" and "tone off the style guide" are to be fixed at their causes.
The reader pulls 12 emails a week, 2 from each shop at random, plus every email whose customer replied with a complaint or a question about ingredients, and reads all of them against the sheet and the shop manager's word on what that shop is baking. In week 3 the sample finds 3 emails from one shop, 2 drawn at random and 1 that drew a complaint, that name a bagel the shop had stopped baking the previous week. The cause is the source: the sheet for that shop was not updated. The failure is not on the unacceptable list and the source is one sheet the owner can correct, so the owner corrects the sheet, keeping the faulty version for the trace, and the plan lets the step keep running. The reader finds every other email composed from that sheet and the team sends a short correction to those customers. The finding goes into the test set and the set is rerun against the pass bar. In week 5 the vendor changes the model; the first 20 emails after the change are read before the routine date, the stored test set is rerun, and 1 email is flagged for tone, logged against the new model version and handled as an incident, with its cause traced; the first 4 weeks had also logged 1 tone flag, so the drift rule below is not triggered.
The pause rule says that any email that touches the allergen line or a price, or states an allergen detail outside the locked block, pauses the step at once and sends the fixed template, and that the owner acts if a failure mode appears in more reads in one week than it did in any of the first 4 weeks. In week 9 an email's specials paragraph states an allergen detail taken from an older sheet; the step is paused, the customers who received it are told, the sheet the step drew on is corrected, the fixed template runs for 3 days, the test set is rerun, the first emails after the restart are read, and the owner signs off the restart on the register row. The reader's monthly report shows defects by mode and cause, not a count per shop.
Common mistakes
- Reading in launch week only. The reading carries on after launch, with its reader, its log and its pause rule, and a use whose log is empty may be one nobody is reading.
- Letting the builder be the reader. A builder checks the output against what they intended, so the reader has to be someone else.
- A hand-picked sample. It drifts towards the reader's worries; draw at random within groups and add the cases that went wrong.
- Fixing the item that was drawn. Every item that shares its cause carries the same error.
- A pause rule with no owner. A threshold nobody may act on is an alarm nobody answers.
- Inventing a threshold before the first reads. Write the rule now and the number from the reads, then recompute it as history builds.
- Loosening the plan for the unacceptable list. A failure that is unacceptable at any rate keeps its lock or full read and its pause rule, whatever the sample shows.
- Reading against nothing. Without a reference the reader has no way to check whether a fluent item is supported.
- Making defects found a target. The reader may find fewer, and the plan can stop showing what the use is doing.
- Treating a small sample as a score. Small counts of items are a poor basis for ranking people or tools; read them for causes.
For the review and sign-off structure this plan sits inside, start from the AI governance hub.
