Why this matters
Creative questions arrive vague: "does UGC work for us?" Run as two ads in one ad set and read on clicks, the test gives a confident answer that means nothing. The creative testing guide explains what a test must isolate, and the ad creative hook test design shows a controlled plan for one variable. Here the question is harder to isolate, because customer and studio videos differ in more than one way.
The brand and its current ads
Wagtail Fresh is an invented subscription brand delivering fresh-cooked dog food, with Meta prospecting optimised for first orders.
Wagtail Fresh: current prospecting and assets
Prospecting over the last 28 days: one broad ad set, £300 a day, £8,400 spent, 280 first orders, a cost per first order (CPACost of each conversion.) of £30 against the founder's ceiling of £38. Click-through rate (CTRShare of impressions that result in a click.): 0.9%. Post-click conversion rate to first order: 6.1%.
| Asset | Type | Status | Opening |
|---|---|---|---|
| Studio A | Studio video, bowl being filled | Running, takes most spend | Logo, then bowl |
| Studio B | Studio video, ingredients laid out | Running | Ingredients, no caption |
| Studio C | Studio video, box unboxed | Running | Box on doorstep |
| Customer 1–6 | Phone videos from subscribers, each showing their dog eating, filmed for a competition | Not yet run; written licence for paid use on Meta for 12 months | Varies |
Constraints: prospecting stays at £300 a day; the customer clips can be cut and captioned but not reshot; offer, landing page and price are fixed for two months.
Isolating one variable
Customer and studio videos differ in who is on screen, how it is filmed, what is said, how it opens and how long it runs. If all of those vary, a win says which pile of ads did better, not why. If the answer will shape what the brand commissions next, the variable is production style: the same insight, offer, length, captions and end card, filmed by customers or in the studio. That still bundles person and setting together, and the plan should say so.
Hold the hook constant with the same caption over the first frame in both arms; the hooks and user-generated content guide covers writing it to work without sound. Use several ads per arm, so the result is not one dog's charisma, and run the arms through the A/B test tool so the audiences do not overlap.
Task
Write the plan before reading the worked answer.
- State the question in one sentence and the hypothesis in the format of the experiment hypothesis template.
- Define the two arms: which assets go in each, what is held identical and what still differs.
- Choose the method and say what happens to current prospecting during the test.
- Choose the primary metric, the guardrails and the diagnostics.
- Set the budget, the minimum result count per arm and the duration, and write the stopping rule.
- Say what the brand does if either style wins, if the result is flat, and if customer videos win on clicks only.
Worked answer
Question. For cold audiences, does customer-filmed footage of the same message produce first orders more cheaply than studio footage?
Hypothesis. Because customer footage reads as a peer rather than an advert, it will earn more stops from dog owners with the switching problem, and lower the cost per first order, with post-click conversion rate holding.
Wagtail Fresh: the test plan
| Element | Choice |
|---|---|
| Arm A (control) | Studio A, B and C, re-cut to 20 seconds, caption on the first frame "Switching foods shouldn't take a fortnight", same end card |
| Arm B (challenger) | Customer 1, 2 and 4, cut to 20 seconds, same caption, same end card |
| Held identical | Offer, landing page, length, opening caption, end card, optimisation event, placements, attribution setting |
| Still differs | Footage, setting and person on screen: the production style |
| Method | A/B test tool, two arms, randomised non-overlapping audiences |
| Current prospecting | Paused; Arm A is effectively the current creative, so risk is low |
| Budget | £150 a day per arm |
| Primary metric | Cost per first order |
| Guardrails | Post-click conversion rate; 7-day frequency per arm |
| Diagnostics | Click-through rate; share of impressions that become a play of the opening seconds |
| Minimum before reading | 80 first orders per arm, chosen by the brand before launch |
| Duration | 3 whole weeks: £3,150 per arm at about £30 an order buys around 105 orders, above the minimum |
| Stopping rule | No reading before day 21 and 80 orders per arm; no edits; stop early only for a fault |
| Later check | Share of each arm's first orders that renew to a second box, read once renewal data exists |
Why the metric. Cost per first order matches the business outcome; clicks only say the opening earned attention. The renewal check matters because the styles may attract different customers, and a first order that never renews is not cheap.
Why the sizing. The minimum count is set before launch so nobody reads the test early; the sample size and significance guide is the place to firm it up. Whole weeks cover weekday and weekend behaviour, and the duration stays short enough that frequency and fatigue do not become a third variable.
Decisions. If customer videos win, brief a steady supply of licensed customer and creator content, then test hooks within that style. If studio wins, keep it and test hooks there. If the result is flat, style is not the bottleneck: choose the cheaper one and test the offer or hook next. If customer videos win on click-through but not on cost per first order, the footage promises something the page does not deliver; check message match first.
Common mistakes
- Putting both styles in one ad set. Delivery backs one early and the other never gets a fair exposure.
- Letting the hook vary with the style. Two variables, no way to tell which won.
- One ad per arm. The result describes a single clip, not a production style.
- Reading on click-through. Attention is a diagnostic; the decision rests on first orders.
Next, test yourself with the creative testing and fatigue checkpoint, or return to the intermediate Meta Ads level. The same A/B TestControlled comparison of two variants. design serves any later audience or offer question.
