Why this matters
The intermediate AI in marketing level chose tools on evidence and designed pilots with a decision rule. This exercise starts after the choice, with the question that decides whether an assistant is right in the details customers act on: where does the business's knowledge live? The comparison of retrieval and fine-tuning answers it. Retrieval leaves the model unchanged and places the relevant passages from the brand's documents in front of it for each question, so facts stay current and answers can cite their source. Fine-tuning trains a model further on the brand's examples, which shapes style and format but carries facts only as an impression. The tutorial on preparing a document library then makes the point that decides most of the work: an assistant that answers from documents knows nothing the documents do not say, and most of its wrong answers start in the library it reads from.
The retailer and its two proposals
Nivquade Cycles is an invented online retailer of bicycles, parts and accessories. It has a help centre, product pages with specifications, a few policy pages and a shared folder that someone connected during a trial. Two vendors have proposed a product question assistant. The first retrieves from the help centre and product pages and cites the page it used. The second offers "a model trained on your brand" built from past product copy and customer emails, and its demo sounds more like the retailer than the first vendor's does. The owner likes the voice. The service lead has run both on a test set of real questions from chat logs, with names removed.
Nivquade Cycles: what the assistant can reach, and what the test runs returned
| Source connected to the assistant | Documents | What the list shows |
|---|---|---|
| Help centre articles | 48 | 3 versions of the returns policy, 2 of the delivery policy |
| Product pages | 310 | Frame sizes in a table with internal codes; 22 pages for discontinued models still live |
| Policy pages | 7 | Guarantee terms with the exceptions in a footnote |
| Shared folder "assistant trial" | 14 | A margin-by-brand sheet, an order export with customer names, a supplier price list and 2 draft blog posts |
Test set: 40 real questions.
| Result | Retrieval vendor | "Trained on your brand" vendor |
|---|---|---|
| Right, with a source the lead could open | 29 | 0 (no source to open) |
| Right in substance | 29 | 25 |
| Said it did not know and offered a person | 5 | 0 |
| Wrong on a size, a price, compatibility or a policy | 6 | 15, of which 9 concerned discontinued models |
| Sounded like the retailer, in the lead's judgement | 21 | 36 |
The second vendor confirms that removing a discontinued model from its answers means another round of training.
Facts in documents, form in instructions
The comparison's decision rule is a sort. List what the assistant must know and separate facts from form. Sizes, compatibility, prices, delivery, returns, the guarantee and the approved claims are facts: they change, customers act on them and a reviewer may ask where an answer came from, so they belong in retrieval, where GroundingMaking a language model answer from specified material, such as documents retrieved for the question, rather than from what it absorbed in training. keeps each answer tied to a passage the retailer owns and can update. Voice and format are form: handle them with prompts, approved examples and the standing instructions first, and consider Fine-TuningTraining an existing language model further on a set of examples so that a style, format or task pattern is built into the model itself. only for a measured, stable gap in form that remains after those have been tried, with enough approved examples to train on and the capacity to re-tune and re-test when the base model changes. Most tuned set-ups still use retrieval for facts.
The library then decides the quality of the retrieval. The tutorial's routine is the order of work: list what is connected, take out what must not be quoted, keep one current Source of TruthThe system whose figure is treated as authoritative for a given number, such as billing for revenue or the CRM for pipeline, with other tools reconciled to it. per fact, name an owner and a review date for each document, rewrite sections to stand alone, put tables into sentences, use the customers' words, trace test answers to passages and tie updates to the changes that move the facts.
Task
- Sort what the assistant must know into facts and form, choose the mechanism for each, and write the questions the retailer must put to the second vendor before "trained on your brand" can mean anything in a contract.
- Start from the connected sources, not the intended ones. Decide what leaves the library and why.
- Group the remaining documents by the fact they describe, pick one current source per fact with the person who owns it, archive the rest outside the library and record an owner and a last-reviewed date for every document that stays.
- Rewrite the passages the test run shows cannot travel alone: the size table, the guarantee footnote and the product customers call by a name that appears in no document.
- Trace each of the retrieval vendor's wrong answers to its cause, fix the document rather than the instructions, and say what the re-run should show.
- Add the line to the price and policy change process that keeps the library current, and write the change-log entry for this clean-up.
Worked answer
Facts and form. Everything in the retailer's questions that can be wrong is a fact: a frame size, whether a part fits a model, a price, a delivery time, a returns condition. Those go to retrieval over owned documents. The voice the owner liked is form, and the comparison says to handle it with a voice guide, approved examples and instructions before anything is trained. The second vendor's accuracy on discontinued models is the comparison's warning in practice: a tuned model holds an impression of its examples, and removing information from it means re-tuning. The decision is retrieval, with a few approved voice examples in the instructions and a re-test later.
The vendor questions. The comparison lists them: which mechanism sits behind the promise and for which features; where the documents and examples are stored, for how long and who at the vendor can see them; whether the retailer's material trains models offered to anyone else, and whether that can be switched off; how a changed price or policy reaches answers and how quickly; how a document or example is removed; what happens when the base model changes; whether the vendor will run the retailer's own test set and report the results; and what the retailer can take away at the end. The vendor terms on inputs and training belong in the agreement, as the disclosure and data privacy guide sets out.
What the assistant can reach. The trial folder leaves the library whole: the margin sheet and supplier prices are internal-only, the order export is personal data that must be removed at source following the data minimisation tutorial, and the draft posts are not approved facts. Where a trial document held a public fact, that fact is written into a customer-facing article and the document stays out. The discontinued product pages are archived outside the library too: a page that matches a customer's wording can be retrieved ahead of the current one, so "superseded" labels are not enough.
One source per fact, with an owner. The returns and delivery policies each keep one version, chosen with the person who sets the policy; the other versions are archived outside the library and the register records where they went. Each remaining article and product page gets the owner of the fact it describes and the date it was last checked against the live facts, shown on the document where the system allows, so Data FreshnessWhether a table, feed or report holds data as recent as it should; a freshness check fails when the latest date is older than the source's expected arrival. is visible to anyone reading it.
Passages that stand alone. The size table gains sentences that state the rule for each size in words, with the everyday size name beside the internal code. The guarantee's footnoted exceptions move into the same paragraph as the rule they qualify. The folding model customers ask about by a nickname gets that name in the heading and first line of its page, spelled the same way everywhere.
Nivquade Cycles: tracing the retrieval vendor's 6 wrong answers
| Cause found by opening the cited passage | Answers | Fix |
|---|---|---|
| The oldest returns policy matched the question's wording | 2 | 1 current version; 2 archived outside the library |
| Frame size read from the table alone | 2 | A sentence per size with its everyday name |
| Guarantee exception sat in a footnote | 1 | Exception moved into the rule's paragraph |
| "The folding one" appears in no document | 1 | Nickname added to the page heading and first line |
Re-run: 38 of the 40 answers trace to the current passage. The remaining 2 are questions no document covers, where the assistant says it does not know and offers a person. The team writes the 2 missing articles and adds the questions to the set.
The trace. Each wrong answer is read back to the passage the assistant was given, because a citation shows what the model was handed, not that the answer is supported. Every cause is a library fault the earlier steps describe, so the fix is to the document, never to the instructions. Scoring and pass bars for the re-run follow the guide to testing prompts and models.
Keeping it current. A line goes into every process that changes a fact customers ask about: update the owning document, reset its review date and rerun the test questions that touch it before the change goes live. The change-log entry records what was archived, what was rewritten and why, so a later wrong answer can be traced to the day it started. The instructions the assistant follows, with their silence rule, hand-over list and disclosure line, are a separate job in the tutorial on writing standing instructions, and the AI customer service launch checklist ticks off both before customers see the assistant. The mechanism this whole exercise rests on is in how grounding and retrieval work.
Common mistakes
- Choosing the vendor whose demo sounds right. Voice is form; the test set showed where the facts were wrong.
- Fine-tuning to teach facts. Discontinued models, prices and sizes drift into plausible errors that re-tuning alone can remove.
- Cleaning the documents you meant to connect. The trial folder was the surprise, and it held personal data.
- Marking old versions instead of removing them. The oldest returns policy was still retrieved first.
- Fixing the instructions when the document is wrong. Every traced cause was a passage, not a rule.
- Treating the clean-up as finished. Without the line in the price-change process, the library drifts back within a season.
The checkpoint that follows in this module tests the same judgements.
