What this checkpoint covers
The questions apply the comparison of retrieval and fine-tuning, the tutorial on preparing a document library and the tutorial on writing standing instructions. Every scenario is set at Vorlinsea Holidays, an invented cottage letting agency adding a guest assistant to its website.
Vorlinsea Holidays: the connected sources and the first test run
| Source connected | Documents | What the listing shows |
|---|---|---|
| Help centre | 36 | 2 versions of the cancellation terms |
| Cottage pages | 120 | Seasonal prices in a table under internal season codes |
| Shared folder "owner documents" | 18 | Owner contracts with cleaning fees and home addresses, connected in the trial |
First run: 21 of 30 guest questions right with a cited passage; 6 wrong on a price, a cancellation rule or a cottage feature; 3 answered where it should have handed over.
Task
- Write a decision and a reason for every question before reading any explanation.
- Mark each answer right, partly right or wrong, noting the guide behind any miss.
- Six or seven right: move on. Fewer: reread the sections behind the misses.
Questions
- A vendor's demo "trained on your brand" sounds like the agency. Prices, pet rules and cancellation terms change yearly. Where should each kind of knowledge live, and what can the demo not show?
- The proposal reads "your own AI, trained on your content". What must be established before the phrase goes in a contract, and which questions settle it?
- For a summer holiday week the assistant quotes the low-season price and a refund deadline the current cancellation terms dropped. Both answers cite passages. What do you open, expect and fix?
- The assistant quotes a cleaning fee that appears on no public page, and the lead is certain it reads only the help centre. What do you check first?
- The marketing lead wants the prompt library's "guest enquiry reply" entry used as the assistant's instructions. What is missing?
- The draft instructions say "use good judgement about when to pass a guest to a person". Rewrite that rule.
- Two tests fail: a guest who mentioned her father in hospital while asking about late check-in got only the times, and a hot tub question about the mill cottage was answered from another cottage's page. Which fix goes where?
Answers explained
- Facts in retrieval, voice in the instructions. Prices, rules and terms change and guests act on them, so they belong in retrieved, cited documents, with GroundingMaking a language model answer from specified material, such as documents retrieved for the question, rather than from what it absorbed in training. tying answers to owned passages. Voice is form: examples first, Fine-TuningTraining an existing language model further on a set of examples so that a style, format or task pattern is built into the model itself. only for a measured gap. A demo cannot show accuracy; a test set can, as the comparison says.
- Which mechanism, then the terms that follow. The phrase can mean retrieval, a tuned model, stored instructions or a mix, and the mechanism decides the risks. The comparison's questions settle it: mechanism, storage and access, training for others, how changes and removals reach answers, base model changes, the agency's own test set, and exit.
- Open each cited passage, expect library faults, fix the documents. A citation shows what the model was handed, not that the answer is supported. The document library tutorial names both faults: old terms matching the guest's wording ahead of the current ones, and prices only in a table under season codes. Archive the old terms outside the library, not marked superseded, and add sentences giving each season's dates and prices in guests' words.
- The connected sources, not the intended ones. Whatever the assistant can reach, it can say. The listing shows owner contracts connected in the trial: the fee came from there, and the home addresses are personal data that leave at source, following the data minimisation tutorial. Public facts move into a guest article; the contracts stay out.
- Scope, sources and the silence rule, hand-over topics, disclosure, the never list, a test for each. A PromptThe instruction and material given to an AI language model, which works like a brief: the model knows language, not the audience or product, unless told. in the library is run by a colleague who reads the output first; standing instructions are followed with nobody reading the reply, so they carry that colleague's decisions. The entry gives a task and a tone; the standing instructions tutorial gives the rest.
- Topics and signals, written plainly. Complaints about a stay, anything about money back, exceptions to the terms, medical needs, legal, safety and regulatory matters, and guests in vulnerable circumstances. Beside them the signals: the guest asks for a person, the conversation goes in circles, or the guest is plainly upset.
- Distress: instructions. Hot tub: documents. Then the whole set runs again. The vulnerable circumstances rule was a label the assistant could not apply, so it becomes plain descriptions of the signs, with an example. The mill cottage's passage never named its cottage, so the page names it in the heading and first line and becomes the Source of TruthThe system whose figure is treated as authoritative for a given number, such as billing for revenue or the CRM for pipeline, with other tools reconciled to it. for that cottage. The test is never changed to fit the answer.
Where to go next
A miss on the third, fourth or seventh question means the library is still treated as the model's problem: redo the library steps of the bicycle retailer grounding exercise. A miss on the fifth or sixth means the instructions are still a prompt. The intermediate AI in marketing level chose the tools; the next module handles an answer that reached customers wrong.
