Brief
Krovelby Learning is an invented provider of online professional courses in accounting, project management and data skills. Its marketing team is a head of marketing, two content marketers, a paid media manager, a community and support lead and a junior marketer. The intermediate adoption plan was written and followed, so chosen use cases, ad guardrails, review tiers, a data rule and one adopted drafting tool are in place.
Since then AI has grown in pieces rather than by plan. A course-support assistant on the website answers learners' questions from a document library that nobody owns and that a vendor connected at set-up. The content team drafts course pages, and the outcome claims on them, with the drafting tool. The paid media agency has switched on the AI enhancements in both ad accounts. The creative agency used a generated presenter in an advert, and learners wrote in to ask whether she taught the course. The managing director has read last quarter's incident notes and wants one operating model that says who owns each part, how mistakes are handled, what the agencies may do, how the team's skills grow and how anyone will know whether AI is helping. She has asked for owners and dates, not principles.
Inputs and constraints
Krovelby Learning: the assistant's library and its test run
| Source connected to the assistant | Documents | What the list shows |
|---|---|---|
| Course pages | 94 | Prices and start dates in a sidebar block; 11 pages for retired courses still live |
| FAQ articles | 31 | 3 versions of the refund policy |
| Tutor biographies | 40 | 6 tutors who have left |
| Policy pages | 6 | Accreditation terms with the exceptions in a footnote |
| Shared folder "support bot" | 41 | Old price lists, draft syllabuses, a learner export with names and employers |
Test set of 50 real learner questions: 34 answered correctly with a source the lead could open; 9 wrong, of which 6 gave a price or start date from an old sheet and 3 described a superseded refund policy; 7 answered "I don't know" where a current document existed. The standing instructions were written by the vendor and have no hand-over list.
Krovelby Learning: last quarter's incident notes
| Incident | Found by | What happened next | Status |
|---|---|---|---|
| The assistant quoted an early-bird price that had ended 9 days earlier to 23 learners | A complaint on day 10 | Price honoured for the 23; nothing changed in the library | Closed |
| A course page claimed "92% of graduates promoted within a year" | The paid media manager, before an ad used it | No evidence found; line cut from the page; still in 2 nurture emails | Open |
| The creative agency's advert used a generated presenter; 4 learners asked whether she taught the course | The support inbox | Agency called it standard practice; advert still running | Open |
| The paid media agency's automatically created assets named a certification the course does not confer | The monthly paid media review | 1 asset paused; the setting is still on across 3 campaigns | Open |
Krovelby Learning: agencies, self-reported hours and two quarters of output
The paid media agency is on a monthly retainer with admin access to both ad accounts and no AI clause. The creative agency works project by project on a scope of work that says nothing about generated elements. Both agreements renew at the start of next quarter. Each content marketer reports saving 10 hours a week; the support lead reports that the assistant "handles 60% of questions". No baseline was taken for any task before the tools arrived.
| Measure | Two quarters ago | Last quarter |
|---|---|---|
| Course pages drafted / accepted | 140 / 128 | 260 / 190 |
| Pages sent back at least once | 12% | 34% |
| Corrections after publication | 2 | 9 |
| Head of marketing's review hours a week | 4 | 14 |
| Assistant conversations | 4,100 | 6,800 |
| Conversations handed to a person | 38% | 29% |
| Complaints about a wrong answer | 3 | 14 |
| Paid media spend / enrolments the platforms attribute | £48,000 / 410 | £51,000 / 430 |
| Enrolments recorded in the course platform | 395 | 402 |
Constraints you must work within:
- Any claim about course outcomes, pass rates or careers needs evidence on file before it is published, in the approved wording and nowhere else.
- Learners are told when they are talking to an assistant, and no generated person may be presented as a tutor, learner or expert.
- Learner records are personal data and go only into tools whose terms the data protection lead has approved; treat the model's data rules as general direction to confirm with that lead, not legal advice.
- No new tools this year; the model governs what is already in use.
- The course platform is the system of record for enrolments; platform-attributed figures are not.
- The model is one document with appendices, and every part has a named owner and a date on which it is judged.
Task
- Decide where facts and voice live, and who owns them. Sort facts from form with the comparison of retrieval and fine-tuning, so prices, dates and policies stay in retrieval, where GroundingMaking a language model answer from specified material, such as documents retrieved for the question, rather than from what it absorbed in training. ties each answer to an owned passage. Clean the library with the routine in the tutorial on preparing a document library: one current source per fact, an owner and review date on each document, and the change that keeps it current. Say what the standing instructions must add, using the tutorial on writing instructions for an AI assistant, and name an owner for each, as the bicycle retailer grounding exercise does.
- Write the incident procedure and run last quarter through it. Set contain, correct, tell, trace and log as steps with an owner for each, from the guide to handling an AI mistake. Run each incident in the notes through it as the opticians chain incident exercise does, down to the creation and escape causes and the one thing each changes in the system. Record them in the incidents column of the AI use register, and set the review that the tutorial on reviewing a marketing error after the fix describes.
- Set the claims and endorsement rules. Require a SubstantiationThe evidence a business holds, before a claim is published, that the claim is true; advertising rules in most markets expect claims to be capable of it. file entry before any outcome claim runs, built as the tutorial on building a claims library builds it, and make the wording check part of every review with the AI-assisted content review checklist. Apply the guide to AI claims and consumer protection to the graduates claim, the nurture emails and the generated presenter: what is withdrawn, what is labelled and who checks what from now on.
- Extend the policy to both agencies. For each agency settle the points the guide to setting AI rules for agencies lists, from permitted uses and the business's data to the ad account AI settings, records and exit. Write them into the agency scope of work template before the renewals, build the settings register from the tutorial on reviewing AI enhancements in ad accounts, and add a register row for each agency use.
- Plan the team's skills from a baseline. Follow the guide to building AI skills in a marketing team: a baseline session on real tasks, practice on the team's own work, review drills with seeded errors, champions with protected time, a library built as the prompt library tutorial describes and the tasks the junior keeps manual. Record each person on the marketing skills matrix and name the measure the programme is judged by.
- Design the measurement for each team. For content, support and paid media, follow the guide to measuring AI's impact on team output and the flooring retailer measurement exercise: a BaselineAn estimate of what a metric would have done without a given change, campaign or promotion, against which the effect is measured. from records, the accepted unit and its whole cost, defect, rework and reviewer load measures, an outcome from a KPI tree and the strongest comparison each team can run. Say what the self-reported hours and the hand-over share each hide. Add a stored test set and pass bar for the course-page prompt and the assistant's question set, from the guide to testing prompts and models, with re-test triggers.
- Set the review rhythm and the owners. Make the register the spine of the model, one row per use with its owner, tier and incidents. Fold the incident, agency and claims rules into the AI Use PolicyA team's written rules for AI: approved tools, what each use may do, what data may go in, what is reviewed, when customers are told and who owns problems. with the review triggers the tutorial on writing an AI use policy sets, assign each part of the model on a RACI matrix, put decisions that change a use in the marketing decision log, and fix the quarterly review with the managing director at which the measures are read and the model is amended.
- Write the document. One operating model: a page per part, the owner and judgement date on each, and the appendices (library register, incident log, substantiation file, agency schedules, skills matrix, measurement design and test sets) behind it.
Evaluation criteria
| Criterion | What a strong answer shows | Common gap |
|---|---|---|
| Grounding and ownership | Puts facts in a retrieved, owned library with review dates and keeps voice in instructions and examples, with a named owner for each | Accepts a model trained on the brand for facts, or leaves the library without an owner |
| Incident procedure | Sets contain, correct, tell, trace and log as steps with owners, and names what each quarter's incidents changed in the system | An apology template with no trace of causes |
| Claims and endorsements | Requires a substantiation file before any outcome claim is published and labels synthetic people so no learner mistakes one for a tutor | Treats AI output as exempt from the rules that apply to any claim |
| Agency rules | Settles permitted uses, the business's data in agency tools, review, disclosure, ad account AI settings, records and exit for both agencies | A policy that stops at the team's own staff |
| Skills plan | Starts from a baseline, trains on the team's own tasks, practises review as well as prompting and protects junior judgement | A vendor demo booked as training |
| Measurement | Uses a baseline, time per accepted output, rework and outcome measures with a comparison that holds up, reported honestly | Hours saved as reported by the people using the tools |
A strong submission also says which of last quarter's incidents would have been caught by the model it proposes, and at which step, and what evidence at the first quarterly review would change it.
Submission checklist
When you have finished, check the model against the evaluation table and against the adoption plan from the intermediate AI in marketing level that it assumes.
