Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Checklists

Metric Definition Change Checklist

A checklist for changing how a marketing metric is defined: owner and sign-off, restated history, every dashboard and test that reads it, targets and notice.
Checklistintermediate13 min readUpdated

When to run this checklist

Run it once a change to a reported metric has been decided and while no report or dashboard yet uses the new version: fees added to cost per lead, refunds and cancellations taken out of revenue, a different date a conversion is counted on, a new attribution rule or channel grouping, a new source of truth for a figure, a qualified lead or a lifecycle stage redefined, or a north star or other headline measure replaced. Run it again for every later change to the same metric. The comparing periods guide counts definition, attribution and channel grouping changes among the breaks in the data, and the tutorial on automating lifecycle stage changes makes the same point about stages: every report that reads them moves on the day the new rules go live.

The person who ticks it is the metric's owner, with the analyst who maintains the definitions and someone from each team that reads or is judged on the number: finance for a measure it also reports, sales for a shared lead definition. The method sits in a handful of pages. The metrics layer guide writes the definition and its versioning rules; dashboard governance keeps the inventory that finds every dashboard a change touches; the measurement change log tutorial records it; and the measurement governance playbook names who may sign off a re-baseline. Two guides work through the whole change for a particular kind of metric: the guide to changing a north star metric re-bases everything that hangs from a headline measure, and the tutorial on moving from MQLs to account qualification changes a definition two teams share.

Neighbouring lists take the cases this one does not. A change to the conversions an ad platform optimises for, their values, their counting or the attribution windows is gated by the conversion goal change checklist, because those settings change what the platform learns from as well as what it reports. Each dashboard that reads the changed metric goes through the dashboard launch QA checklist before it is shared again. A redefined qualified lead changes routing as well as reporting, and the lead routing QA checklist tests the routes. A new tag or event that changes what is collected is proved by tracking verification first. And a number that was simply wrong is not a new definition at all: it is an error, which the tutorial on correcting a number after a report has gone out puts right in the open.

Before you start

  • The current definition, from the metrics layer or a dictionary laid out like the KPI dictionary, with its owner, version and status.
  • The proposed definition written in the same fields: formula, grain, source tables, baked-in filters, attribution rule, time rule, owner and known differences.
  • The reason for the change and who asked for it, in writing.
  • The dashboard inventory, with every dashboard's metrics listed by their keys.
  • The targets, key results and agreements that quote the metric, including any shared with sales, agencies or partners.
  • The running tests, each with the primary metric and decision rule written before it started.
  • The raw rows a restatement would need, and how far back they go.
  • The measurement change log and the next scheduled report.

The checklist

The groups follow the order of the work: make sure it really is a definition change, settle who agrees it, decide what happens to history, find everything that reads the metric, move the targets, tests and models that hang from it, then record and announce the change. A line in the first two groups that cannot be ticked halts the change, the targets group is settled before the switch date, and the last group runs past it and closes at the quarterly review.

0/27

Is it a definition change

Owners and agreement

History and the break

Everything that reads the metric

Targets, tests and models

Record, announce and review

Where each check is explained

The widget keeps the lines short so they can be ticked; the reasoning lives in the guides. Each check links to the page that explains it.

CheckExplained in
Deliberate change, not an errorCorrecting a number after a report has gone out
Tracking, calendar, mix and volume ruled outExplaining a change in a marketing metric
Old and new definitions side by sideMetrics layer and metric definitions
New version or new metricMetrics layer and metric definitions
Owner who approvesMetrics layer and metric definitions
Not decided by the judged team aloneGoodhart's law in marketing
Agreed with finance where it reports the measureFinance does not trust marketing numbers
Marketing lead signs a headline re-baseMeasurement governance playbook
Earlier periods restatedComparing periods in marketing reports
Old version deprecated with an end dateMetrics layer and metric definitions
Overlap period keptMeasurement governance playbook
End date of a side-by-side run agreedMoving from MQLs to account qualification
Before-and-after publishedMetrics layer and metric definitions
Every affected dashboard listedDashboard governance
Version shown on each dashboardMetrics layer and metric definitions
Data tests and reconciliations rewrittenWriting data tests for marketing tables
Switch date marked on chartsComparing periods in marketing reports
Agreements and service levels rewrittenMoving from MQLs to account qualification
Targets and key results re-basedChanging a north star metric
Guardrail beside any new targetGoodhart's law in marketing
Running tests left on their own metricChanging a north star metric
Model and forecast rows rebuiltChanging a north star metric
Change loggedKeeping a measurement change log
Note in the next reportMonthly reporting pack SOP
Every reader toldChanging a north star metric
Decision recorded with a check dateMarketing decision log template
Dashboards confirmed at the quarterly reviewDashboard governance

If an item fails

The first group decides whether this list applies at all. If the number was calculated, joined or filtered wrongly, stop and follow the correction tutorial, which restates the figure as an error and tells the people who acted on it. If the request comes from a number that moved and nobody has yet explained why, run the tracking, calendar and volume checks and the mix and rate split in the guide to explaining a change in a marketing metric first: a tag that stopped firing, a short month or a KPI built on a handful of conversions can make a sound definition look broken.

The agreement group is a hard stop. A change with no named owner becomes whatever the last person to edit a dashboard decided, and a change approved only by the team whose target reads the number is the pattern the Goodhart's law framework warns about: a measure people are rewarded for moving can be moved without anything improving. The qualification tutorial writes that rule into its agreement with sales, so any change to a shared criterion goes through a recorded decision that both teams sign. Where finance also reports the measure, the diagnosis of finance not trusting marketing numbers gives the prevention: definitions move only when both teams agree, and each change carries its date and a note in the following report. A headline metric waits for the marketing lead, as the governance playbook's re-baseline gate requires.

The history group decides how the change is read. Where the raw rows do not allow a restatement, the comparing periods guide offers two honest alternatives: compare only within one side of the break, or switch to a measure the change did not touch. A side-by-side run needs an end date agreed at the start; the qualification tutorial fixes one at the outset so the run closes with a decision, and points out that the change can be undone only for as long as the old stage and its reports are kept running, which is the point of a parallel run.

The readers group can go ahead with a note, provided every item is listed with an owner. The metrics layer guide treats a dashboard that calculates a defined metric for itself as a defect, and the dashboard governance guide removes its certification until it reads from the layer again; a dashboard already flagged for retirement can start its sunset instead of being rebuilt. A reconciliation test left on the old rule will fail every day, and the data tests tutorial warns that a check which fails daily for that reason trains people to stop reading its alerts, the important ones included. The academy checkpoint on redefining metrics and replacing a north star tests these groups through scenarios at a tile retailer, from choosing a new version or a new name to the forgotten dashboards and agency fee that read the old count, and a team offering to write the definition it is judged on.

The targets group has to be settled before the switch date, or a team appears to improve or decline on a day when nothing changed. Where the metric carries key results, the north star guide reports re:Work's guide to OKRs, as the north star and OKRs framework cites it, asking for key results that are measurable and can be checked; by that framework's own rule, each rewritten key result states its data source and the reading it starts from, its Baseline. Pair a new target with a guardrail metric and rework the branches beneath the measure with the KPI tree framework. Where a target moves to a different measure, the qualification tutorial applies Kaushik's measurement model, which derives every measure from the business objective, to set marketing's target on accepted accounts rather than a count of handed-over accounts. Tests already running keep the rule written before they started, because switching a live test to the new definition lets the team pick whichever reading flatters the result. The academy exercise on re-basing paid media when a craft kit subscription changes its north star practises these groups for a north star change that also re-bases bidding, with history restated back to a billing migration, every dashboard found by its key and a trial sign-up target replaced by one paired with a guardrail.

In the last group, a change that is not logged, noted and announced is a silent re-baseline, which the governance playbook warns makes every comparison across the break wrong. Write the change log entry, carry the note into the reports as the reporting cadence playbook does, and write the decision log entry with a date to check the change held.

An outdoor gear retailer redefines revenue (illustrative)

Illustrative example — not a real result

Revenue after returns: eleven dashboards checked, one test left alone

Kibbertane Outfitters, an invented online retailer of walking and camping gear, has reported revenue in its marketing dashboards as the value of orders placed, while finance's monthly accounts take out refunds and cancelled orders against the date of the original order. After a quarter in which marketing reported £412,000 and finance £371,000, the finance controller asks for one definition. The marketing lead owns the metric and runs this list with the analyst, the controller and the head of ecommerce.

The first group settles the kind of change. Nothing was calculated wrongly: marketing's figure followed its own written rule, and the order system shows the same refunds, so this is a new version, not a correction or a tracking fault. Written side by side, the old and new entries differ in two fields: the baked-in filter now removes refunds and cancellations, and the time rule counts each refund against the original order's date, so every month stays provisional until the returns period on its orders has closed. So that the board pack and the paid social agency's monthly report, which quoted the old figure, can carry a footnote instead of being reissued, it stays available for one quarter under the name "order value before returns", marked deprecated with its end date.

The marketing lead signs the change as its owner and, since revenue is a headline metric, as the re-base sign-off, and the controller signs for finance; the ecommerce team, whose targets read revenue, is consulted but does not approve it. The analyst restates 24 months from the order system's refund records and publishes a before-and-after: the last quarter moves from £412,000 to £371,000, and the year-on-year trend barely moves because returns ran at a similar share in both years.

The inventory catches what the team would have missed. Filtered by the revenue key, it lists 11 dashboards: 9 read the modelled table and switch with the version, but 2, built by the agency, compute revenue in their own calculated fields and would have gone on showing the old figure. Both lose certification until they are rebuilt on the layer, and all 11 pass dashboard launch QA again before they are shared. The reconciliation test against the shop platform's gross sales report would have failed every day; it now compares like with like. The ecommerce team's revenue targets and the agency's revenue target are re-based on the restated history, a free delivery threshold test already running finishes on the metric written before it began, and the forecast's revenue row is rebuilt from the restated months.

The change enters the measurement change log for 1 October with the reason and the analyst's name, and the October report carries a note beside every revenue figure. The decision log entry sets a check for the January close: do finance and marketing quote the same revenue for the quarter? They do, and the old name is retired.

Common mistakes

  • Editing the definition in place. A silent change moves every historical number, and the next reconciliation meeting spends its time finding out why.
  • Updating the dashboards someone remembers. The forgotten ones keep the previous figure; the inventory, filtered by the metric key, lists every dashboard the change touches.
  • Calling an error a new version. A wrong number is corrected and restated openly; versioning it hides the mistake from the people who acted on it.
  • Letting the team judged on the number redefine it. The target can be met without anything improving.
  • Switching live tests to the new definition. A test is called on the decision rule it began with, never on a later one.
  • Re-basing the chart and nothing else. Targets, key results, agreements, models and reports that hang from the metric move with it.
  • Restating without saying so. A restated history that nobody announced looks like a report that changed its mind; publish the before-and-after and mark the date.

The checklists library holds the lists that sit beside this one, from dashboard launch QA to conversion goal changes.

Frequently asked questions

Which changes count as a definition change?

Any deliberate change to how a reported number is calculated or what it counts: a cost added to or taken out of spend, refunds or cancellations removed from revenue, a different date the row is counted on, a new attribution rule or channel grouping, a filter tightened, a lead, customer or active user redefined, or one headline measure replaced by another. If the change alters what the number means, it is a definition change. A figure computed wrongly is an error, which is corrected and restated openly rather than versioned, and a tag or event that changes what is collected is a tracking change, verified on its own before any definition leans on it.

Do we always have to restate history?

Restate where the raw rows allow it, because a definition change breaks any comparison that runs across the switch. Where they do not, there are two honest alternatives: compare periods that all fall on one side of the switch and accept a shorter history, or switch to a count the change did not touch. Whichever is chosen, the switch date is marked on every chart that crosses it and the choice is written in the change log, so the step is not mistaken for a change in results.

Who should approve a change to a metric's definition?

The metric's named owner approves it, but never alone where the owner's team is judged on the number, because a team allowed to redefine its own target can meet it without anything improving. Finance agrees any measure it also reports, the team on the other side of a shared definition such as a qualified lead signs the change, and a headline measure needs the marketing lead's sign-off before history is re-based.

Terms used on this page

Frameworks

Where this page is used

Part of these learning paths

Also related

Sources

  1. Digital Marketing and Measurement Model — Avinash Kaushik
  2. Guide: Set goals with OKRs — Google re:Work

Related