Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

Email Marketing › Email Automation

Email Metrics and Testing

How to measure and test email: deliverability and engagement metrics, what open rates can tell you, revenue per email, and holdouts for flows.
Guideintermediate12 min readUpdated

Metrics by layer

Email measurement has four layers, and each answers a different question. The delivery layer asks whether the message reached the mailbox. The engagement layer asks whether the recipient did anything with it. The outcome layer asks whether that activity became the action the email was sent for. The health layer asks whether the programme is damaging the list it depends on. A report that mixes the layers, putting opens beside revenue beside bounces, hides which one moved. In the delivery layer, a hard bounce is the clearest signal, because it marks an address that will never accept mail.

LayerMetricWhat it answersWhere it comes fromWatch for
DeliveryDelivered, bounced (hard and soft), deferredDid the message reach a mailbox at allSending platform logsHard bounces on new sources; deferrals from one provider
DeliveryInbox placement, spam placementDid it reach the inbox or the junk folderSeed tests, mailbox provider postmaster toolingPlacement differing by provider or by stream
EngagementOpens (with the caveats below), clicks, click-through (CTR)Did the recipient notice and act within the emailTracking pixel and tracked linksPre-fetched opens; one link drawing all clicks
EngagementClick-to-openOf those who read, how many clickedDerivedInflated denominator from machine opens
OutcomeConversion rate (CVR), orders, leads, revenue per email, revenue per recipientDid the email produce the action it was sent forAnalytics with campaign parameters; order dataAttribution window; several emails claiming one order
OutcomeIncremental conversions from holdoutsWhat the email caused, not merely touchedHoldout designToo small a holdout; contamination
HealthUnsubscribes, spam complaints, list growth net of churnIs the programme wearing out the listPlatform; mailbox provider feedbackComplaints rising on one send type
HealthAuthentication pass rates, reputation signalsIs the sending identity trustedPostmaster tooling, DMARC reportsFailures after a DNS or platform change

Pick one or two metrics per layer as the key performance indicators (KPI) and treat the rest as diagnostics. The KPI trees guide shows how to connect them so that an outcome metric decomposes into the engagement and delivery metrics that drive it, which is the structure a report should follow. Delivery problems are diagnosed in the deliverability guide; this page is about reading the layers above it.

Open rates and privacy changes

An open is recorded when a tiny tracking image in the email is loaded. For years that was an imperfect but usable proxy for attention. It is now less than that, because some mail clients fetch every image on the recipient's behalf when the message arrives, whether or not a person ever looks at it, and some strip tracking images altogether. The result is an open figure that is partly machine activity, varies by the mix of mail clients on the list, and cannot be compared across lists or across time without knowing that mix.

Opens are not useless. Within one list, a sharp fall in opens for one send against the sends around it still signals a subject line that failed or a placement problem. Opens across a whole list over months still track the broad shape of engagement. What opens can no longer do is act as a goal, a test metric or a benchmark. A subject line test judged on opens is judged partly on which variant happened to go to more pre-fetching clients; a flow whose exit is an open fires on machines; a target open rate set against an industry figure compares two unknowns.

The practical response is to demote opens. Use clicks as the primary engagement metric and the outcome layer as the goal. The same caution applies to click-to-open rate, whose denominator is the inflated open count. Where opens are still reported, separate the mail clients known to pre-fetch where the platform can, and label the figure as approximate. Re-engagement flows that used to trigger on a lack of opens should trigger on a lack of clicks or visits, or risk keeping dormant addresses on the list and removing engaged people whose client blocks images. Google's sender guidelines, listed in sources, are a reminder that the signals mailbox providers actually weigh are complaints, authentication and whether recipients engage, not the sender's own open figure.

The open rate versus click rate comparison sets out what each figure can still be trusted for and how to rebuild a scorecard around clicks.

Clicks, conversions and revenue per email

A click is a person choosing to leave the email for the page it offered, which makes it the first honest signal of interest. Click-through is clicks divided by delivered messages; the newsletter design and copy guide explains why one call to action per email keeps that figure readable. Look at clicks by link as well as in total, because the distribution says what people wanted, and look at unique clickers rather than total clicks, since one enthusiast clicking everything is not a trend.

Conversion is the action the email was sent for, measured where the action happens: an order in the shop, a form in the analytics property, a booking in the calendar. Email conversions are attributed through campaign parameters on the links and the analytics attribution model, and each choice shapes the figure: a short window undercounts considered purchases, a long one lets a newsletter claim orders that other channels earned. Agree the window and the model with whoever owns analytics, write them into the report, and keep them stable, because a change in the rule looks exactly like a change in performance. The conversion rate calculator turns clicks or deliveries and conversions into the rate once the counting rule is settled.

Revenue per email sent, or per recipient, is the figure that lets sends of different sizes be compared and that connects the programme to the business. It is attributed revenue divided by messages delivered, under the same attribution rule, and it should be read with the attribution caveat printed beside it. For flows, read it per entrant; for campaigns, per send. The companion figure is revenue per thousand sent against the cost per thousand sent, which is where the programme's economics become visible, including the cost of incentives the emails carried. None of these outcome figures say what the email caused; they say what it touched. For that, the next section.

Holdouts for automated flows

An automated flow reaches people at moments of intent: just signed up, just left a cart, just bought. Many of those people would have acted anyway. A flow's own report counts every order it touched, which flatters it in proportion to how well it targets, and a business that budgets on that report will overvalue the flow and underinvest in the things that fill it. The question that matters is the Incrementality question: what happened because of the flow that would not have happened without it.

A holdout answers it. A random share of the people who qualify for the flow are entered but receive nothing, and their outcome over the same window is compared with the people who received the sequence. The difference, per entrant, is the flow's contribution. The design is simple; the discipline is in the details. The split has to be random and made at entry. The holdout has to be large enough that the difference seen is larger than the noise, which for a small flow may mean holding out for months. The window has to be the same for both groups. The holdout group has to be excluded from the flow but not from everything else, or the test measures the whole programme. And the result has to be read as a rate per entrant, not as a total, so that it can be applied to future volumes.

Holdouts cost revenue in the short run, which is why they are skipped. The usual compromise is a small permanent holdout on the flows that carry the most spend or the most incentive, read quarterly, and a one-off larger holdout when a flow is being redesigned or its budget questioned. The incrementality testing guide covers holdout and geographic designs in depth, including how to size a cell and how to read lift; the email-specific point is that flows, unlike campaigns, run continuously, so a holdout can run alongside them indefinitely at low cost once it is set up. Made at sign-up rather than at a flow's entry, the same split values the subscriber rather than the flow, which the guide to what an email subscriber is worth works through on contribution margin.

Illustrative example — not a real result

A meal-kit subscription reads its post-purchase flow honestly

A meal-kit subscription ran a four-email post-purchase flow aimed at a second order and reported that 38% of recipients ordered again within 30 days, which the team presented as the flow's result. A holdout was added: 10% of new customers, chosen at random at entry, received no post-purchase emails for 30 days. Over a quarter, 14,200 customers entered the flow and 1,580 were held out. Second orders came from 5,396 flow recipients (38%) and 537 of the holdout (34%). The flow's contribution was therefore about 4 percentage points, or 40 extra second orders per thousand customers, worth around £22,000 in the quarter at the average second-order value of £55. That was still clearly positive, but a fraction of what the flow's own report implied, and it changed the decision on a proposed incentive in email three: at a cost of £6 per redemption, it would have needed to add more second orders than the whole flow was adding.

A/B testing with enough volume

An A/B Test in email is a controlled comparison: two versions of one element, sent to randomly split groups at the same time, judged on one pre-agreed metric. Subject lines, preheaders, the call to action, send time and the first email of a flow are the usual candidates. The test is only as good as three conditions. The groups are random and alike. The metric is decided before the send, and it is the metric closest to the goal that the test can affect, which means clicks for a subject line test and conversions for a content test, and rarely opens for the reasons above. And the difference seen is larger than the difference chance alone would produce at that volume.

The third condition is where most email tests fail. With a few hundred recipients per variant, a difference of a few clicks is chance, and declaring a winner from it produces a programme that lurches from one lucky result to the next. The required volume depends on the baseline rate and the size of difference worth detecting: a rare outcome such as purchase needs far more recipients than a common one such as a click, and a small improvement needs far more than a large one. Small lists can still test, by running the same comparison across several sends and pooling the results, by testing larger changes that would show at the available volume, or by testing on the engagement layer where the counts are higher.

The calculator below takes the recipients and conversions for two variants and reports whether the difference clears the conventional confidence threshold, and by how much. Use it after the send to read a result, and before the send to see what difference a given volume could detect at all. A result that does not clear the threshold is not a loss; it is a test that needs more volume or a bigger change.

A/B test calculator

Run one test at a time per send, write down the hypothesis and the metric before launch, let the test finish rather than stopping when one variant pulls ahead, and keep a log of results including the ones that showed nothing. Over a year the log is the programme's real knowledge: which kinds of subject line this list responds to, which calls to action, which send times, and which questions turned out not to matter.

One question a single-send comparison cannot answer is how often to send, because frequency acts on the same people over weeks; the email frequency test sets out the subscriber-level cadence design and why it is read on revenue per subscriber rather than per email.

Reporting cadence

A report should match the rhythm of the decisions it supports. Daily: delivery and health only, as alerts rather than a report, so that a bounce spike, a complaint spike or an authentication failure is seen the day it happens. Weekly: the campaign sends of the week on engagement and outcome, compared with the same send type's recent history, and the flows on volume and conversion, to catch a broken trigger or a template that stopped rendering. Monthly: the full set of layers for campaigns and each flow, revenue per email against cost, list growth net of churn, and the log of tests concluded. Quarterly: holdout results for the flows that have them, the attribution rule reviewed with analytics, the incentive policy reviewed against margin, and the key performance indicators themselves questioned.

Three habits keep the reports honest. Print the attribution window and model on every outcome figure. Compare each send with its own type's history rather than with an industry figure. And separate flows from campaigns in every view, because a flow's steady trickle and a campaign's weekly burst behave differently and a blended total hides both. The automation hub sets out what governance each flow needs beyond the numbers, and the measurement plan behind these reports, from goals to the tracking that feeds them, follows the structure in the measurement plan guide.

Flow holdouts are one part of a longer discipline, and the experimentation loop for retention explains how to read them at fixed cohort ages and separate novelty from lasting change.

For a step-by-step way to compute click-to-delivered rate against your list's own median, use the CTR calculator set up for email campaigns.

The email channel page for online sales builds these holdouts into a sales plan, with revenue per recipient and the discounted share of email revenue as its main measures.

When the numbers suggest the problem is wider than one flow, the email programme audit scores testing and reporting next to list health, deliverability and segmentation.

When the open rate falls in a single step, the open rate drop diagnosis checks clicks and the provider split to decide whether measurement, placement or audience moved.

When unique clickers hold and email-credited orders fall, the diagnosis for email clicks that bring no sales separates a counting fault from a lost path to purchase.

Common mistakes

  • Opens as the goal. Machine-fetched images are counted; people reading the preview text are not.
  • Benchmarks from the industry. A list's own history is the only comparison with the same mail client mix, consent quality and send frequency.
  • Changing the attribution rule and reading it as performance. Print the rule on the report and keep it stable.
  • Crediting a flow with everything it touched. Without a holdout, the flow claims the orders that were coming anyway.
  • Tests on a trickle. A winner declared from a few dozen clicks is a coin toss with a chart.
  • Stopping a test early. The variant ahead on day one is often behind by the end.
  • Blended reports. Flows and campaigns in one total hide which one moved.
  • Re-engagement triggered on opens. It keeps dormant pre-fetching addresses and removes engaged people whose client blocks images.
  • No test log. The same subject line test is run every year by a new person, with the same inconclusive result.

Frequently asked questions

Why has my open rate risen while clicks have not?

The most likely cause is a change in the mix of mail clients on the list rather than a change in behaviour. Clients that pre-fetch images register an open for every message they receive, so as more of the list uses them, the open figure climbs with no extra reading behind it. Judge engagement on unique clicks and the outcome layer, and treat the open figure as approximate.

How big does an email A/B test need to be?

It depends on how common the outcome is and how small a difference you need to detect, which is why there is no single number. A subject line test judged on clicks needs far fewer recipients than a content test judged on purchases, and detecting a small improvement needs many more recipients than detecting a large one. Use the calculator on this page before sending to see what a given split could detect, and pool results across sends when a single send is too small.

Do I really need a holdout for every automated flow?

No. Keep a permanent small holdout on the flows that carry the most volume, spend or incentive, where the question of what they add has a budget attached, and run a one-off holdout when any other flow is redesigned or challenged. A flow with a trickle of entrants cannot support a readable holdout in any case, and its report should be read with that caveat printed beside it.

Terms used on this page

Tools to try

Frameworks

Where this page is used

Also related

Sources

  1. Email sender guidelines — Google Workspace Admin Help

Related