Why this matters
The intermediate AI in marketing level piloted tools before adoption. Once a tool is in daily use the question becomes whether it is helping, and the usual answer, hours saved as estimated by the people using it, is the one the guide to measuring AI's impact on team output ranks lowest: it forgets the checking and compares against an imagined manual version. The guide starts instead from a BaselineAn estimate of what a metric would have done without a given change, campaign or promotion, against which the effect is measured. taken from records.
Underneath the team measure sits a prompt measure. The guide to testing prompts and models builds a Test SetA collection of real inputs for an AI task, each noting what a good output must contain, run to compare prompts or models and rerun when they change. from real work, writes the rubric first and sets a pass bar in advance. A team that has never tested its daily prompt cannot say whether a defect rate belongs to the tool or to the prompt.
The retailer and its hours-saved claim
Zorvantik Flooring is an invented retailer of engineered wood, laminate and vinyl flooring. Its content team drafts product pages from supplier specification sheets and advice pages on fitting, underlay and care, has used a drafting assistant for two quarters, and the content lead reviews every page. The finance director will not pay for more seats on the strength of the team's survey or the vendor's slide, and wants a measurement she could defend.
Zorvantik Flooring: the two claims and what the task tracker holds
| Source | What it says | What it counts |
|---|---|---|
| Team survey (4 people) | Each saves 7 to 8 hours a week, "about a day" | Remembered drafting time; no review or rework |
| Vendor slide | Pages a month up from 48 to 84, "75% more output" | Drafts generated, not pages accepted; the quarter of a range launch |
Baseline, the 3 months before the assistant, from the task tracker (every page was eventually accepted):
| Task | Accepted | Writing hours per page | Review hours per page | Sent back | Defects per page in review | Corrected after publication |
|---|---|---|---|---|---|---|
| Product pages | 62 | 2.8 | 0.6 | 15% | 0.4 | 1 |
| Advice pages | 18 | 5.5 | 1.2 | 22% | 0.3 | 0 |
The 3 months after a 3-week learning period:
| Task | Drafted | Accepted | Writing hours per page | Review hours per page | Sent back | Defects per page in review | Corrected after publication |
|---|---|---|---|---|---|---|---|
| Product pages | 118 | 97 | 1.0 | 1.2 | 31% | 1.3, of which 0.6 specification errors | 4 |
| Advice pages | 26 | 21 | 3.9 | 2.1 | 38% | 0.9 | 2 |
Rework averages 0.9 hours per page sent back in both periods. Sample-order rate from product pages: 3.1% before, 2.9% after, across a range launch and a price promotion. The product-page prompt has never been tested. The team works 13-week quarters at 7.5 hours a day.
Baselines, accepted output and comparisons
Work in the measuring guide's order: tasks and units first, the baseline from records, then the whole cost of an accepted unit paired with acceptance, defects by type and reviewer load. For the outcome, follow product pages up a KPI tree to the sample orders they exist to win. Fix the comparison, measures, period and decision rule before reading a figure.
Task
- Say what the survey figure and the vendor's slide each count, and why neither answers the finance director's question.
- Set the baseline from the tracker and calculate hours per accepted page, acceptance rate and reviewer load for both tasks in both periods.
- Name the outcome measure for product pages and say why the before-and-after sample-order figures cannot be read as the assistant's effect.
- Build a test set for the product-page prompt from real products, write the rubric before any run and set a pass bar with the content lead and the finance director.
- Choose a comparison for each task, with its period, measures and decision rule written first, and name the weaker design where the stronger one is impossible.
- Write the report, with the decision on seats and what the numbers cannot show.
Worked answer
The two claims. The survey counts the moment the draft appeared, not the hour the lead spent checking it: the measuring guide's example of Vanity MetricsMetrics that look impressive but do not inform a decision or track a business outcome., easy to grow and loosely connected to results. The vendor's slide counts drafts rather than accepted pages, in a quarter when a range launch raised demand for pages anyway. Neither includes review, rework, defects or what the pages did once published.
The baseline and the whole cost. The baseline comes from the tracker, not memory. Each accepted page is costed as writing plus review plus rework, with the learning weeks set aside.
Zorvantik Flooring: hours per accepted page with review and rework counted
| Measure | Product pages before | Product pages after | Advice pages before | Advice pages after |
|---|---|---|---|---|
| Writing hours | 62 × 2.8 = 173.6 | 118 × 1.0 = 118 | 18 × 5.5 = 99 | 26 × 3.9 = 101.4 |
| Review hours | 62 × 0.6 = 37.2 | 118 × 1.2 = 141.6 | 18 × 1.2 = 21.6 | 26 × 2.1 = 54.6 |
| Rework hours | 9 × 0.9 = 8.1 | 37 × 0.9 = 33.3 | 4 × 0.9 = 3.6 | 10 × 0.9 = 9 |
| Hours per accepted page | 218.9 ÷ 62 = 3.5 | 292.9 ÷ 97 = 3.0 | 124.2 ÷ 18 = 6.9 | 165 ÷ 21 = 7.9 |
| Acceptance rate | 62 of 62 | 97 of 118 = 82% | 18 of 18 | 21 of 26 = 81% |
| Reviewer load a week | 2.9 hours | 10.9 hours | 1.7 hours | 4.2 hours |
Saving on product pages: 0.5 hours per accepted page, which is 97 × 0.5 ÷ 13 = 3.7 hours a week across the whole team, against a claimed 4 × 7.5 = 30. Advice pages cost 1 hour more per accepted page than before. The content lead's review load has risen from 4.6 hours a week to 15.1.
Production time fell, review time rose and the saving shrank to a few hours a week while the cost moved to the lead. The defect columns say why: specification errors, the type that matters on a flooring page because customers buy on them.
The outcome. The sample-order rate sits one level up the tree and is already tracked. Its fall straddles a range launch and a price promotion, so the before-and-after figure cannot be read as the assistant's effect; only a comparison inside the same period can separate them.
The test set and the pass bar. The product-page prompt gets the testing guide's method.
Zorvantik Flooring: the product-page test set and the pass bar
Test set of 30 real products: 16 typical ranges; 7 hard cases (herringbone patterns, ranges rated for underfloor heating, boards sold in 2 widths); 4 with a missing field (no slip rating on the supplier's sheet); 3 sensitive cases (a fire rating, a sheet that says "water-resistant" where the brief says "waterproof", a guarantee with exclusions). Each case carries the facts that must appear, the facts that must not, and the existing human-written page as a reference.
Rubric, written before any run: accuracy against the specification sheet (pass or fail), required content (graded), audience fit (graded), voice (graded), constraints such as length and banned claims (pass or fail), handling of gaps (pass or fail: "not stated on the sheet" passes, an invented value fails).
Pass bar, agreed with the content lead and the finance director before the run: at least 24 of 30 cases usable as is or with light editing, and no invented fire rating, slip rating, waterproof claim or guarantee term in any output. Each case is run 3 times, giving 90 outputs, scored blind by 2 reviewers with the human pages shuffled in.
The comparison. Product pages are frequent enough to split: each new product is assigned at random to AI-assisted or manual drafting, by the same writers under the same review, and the arms are compared on hours per accepted page, defects by type, rework and sample-order rate, with the rate reported as a guardrail and a small difference treated as a tie, as the testing guide advises. Advice pages arrive too rarely to split and no comparable team could serve as a Holdout GroupA randomly chosen part of an audience kept from exposure to measure what exposure caused., so they are compared against the baseline trend and the report says the design is weaker. The decision rule, written first: keep the assistant for a task if hours per accepted page fall and defects per page return to baseline; drop it otherwise.
Zorvantik Flooring: 8 weeks of split product pages and the report's figures
| Arm | Pages accepted | Hours per accepted page | Defects per page | Specification defects per page | Sample-order rate |
|---|---|---|---|---|---|
| AI-assisted, weeks 1 to 4 | 14 | 2.9 | 1.1 | 0.6 | 3.0% |
| AI-assisted, weeks 5 to 8, with the specification source pack | 15 | 2.6 | 0.4 | 0.1 | 3.1% |
| Manual, all 8 weeks | 27 | 3.4 | 0.4 | 0.1 | 3.2% |
The test run found 5 invented ratings in 90 outputs, all on the missing-field and sensitive cases, so the prompt failed the bar. The team added a source pack of the supplier's sheet to the prompt and reran the stored set: 26 of 30 usable and no invented rating. The pack went live at the start of week 5. Advice pages against trend: 7.6 hours per accepted page, no better than the baseline.
The honest report. The assistant lowers the cost of an accepted product page by a modest margin, and defects return to baseline only after the source pack, so the saving depends on a process change and the report credits the pack, not the tool alone, for that part. The sample-order rate is reported as not yet separable, and the lead's review load stays in the report as a cost. Advice pages show no saving and the assistant is dropped for them. On seats, none for now: the saving is a few hours a week across the team, not a day each, and the question reopens after a second period of split data and a re-run of the stored test set. The report ends with what the numbers cannot show: whether the junior writer is still learning to write a page from a sheet, which the guide to building AI skills in a marketing team keeps on the manager's list, and whether the lead is tiring.
Common mistakes
- Reporting the survey. It is a memory of drafting, not a measurement of an accepted page.
- Counting drafts. The vendor's figure rose with discarded pages and with a range launch.
- Leaving review out. Review time tripled; the claimed day a week rests on leaving it out.
- Reading before-and-after as an effect. A launch and a promotion moved the sample-order rate in the same weeks.
- Setting the pass bar after the run. With the invented ratings already in view, the bar would have bent to fit.
- Crediting the tool for the source pack. Without the pack the assistant's specification errors stayed above the manual pages'; the pack is a condition of the saving.
