Vanity Compression: A Field Guide
A vanity metric is a metric that exists primarily to be looked at. It does not improve when the underlying thing improves. It does not worsen when the underlying thing worsens. It is, structurally, a dashboard tile in search of a reason to exist.
We have nothing against vanity metrics. We are not here to abolish them. We are here to catalog them, because a thing that has been named is a thing that can be politely declined.
What follows is a working taxonomy. It is incomplete. So is your dashboard.
1. The Cumulative Counter
Definition. A number that can only go up.
Examples. Total users ever registered. Total messages ever sent. Total miles ever driven by the fleet. Total lines of code committed to the repository since founding.
Identifying feature. It is reported in the present tense ("we have...") but describes a past state. It cannot reflect a regression. If usage drops by 90% tomorrow, this metric will rise.
Emotional function. Provides the comforting sensation that something is, on net, accumulating. Useful in fundraising decks. Less useful in product reviews. Of no use whatsoever to a customer.
2. The Ratio Without Denominator
Definition. A percentage reported without the absolute numbers it was calculated from.
Examples. "Conversion is up 200%." (From one user to three.) "Retention improved 40%." (Of the seven users who didn't churn the first week.) "NPS is at a record high." (Of the eleven respondents.)
Identifying feature. The chart is going up and to the right, and you cannot tell from any axis whether "up" means seven or seven million.
Emotional function. Functions as a moral victory. The team has done good work. The work has been measured. The measurement is favorable. Nobody has asked the wrong question.
3. The Engagement Composite
Definition. A single number, designated as the team's primary metric, computed from a weighted sum of other metrics. The weights are determined by which combination makes the number go up most reliably.
Examples. "Engagement score." "Health score." "Activation index." "Customer success velocity."
Identifying feature. Two people in the company can explain what it is. The first explanation contradicts the second. Both are slide decks.
Emotional function. Makes a complex system reducible to one digit. The digit goes on a poster. The poster goes in the office. The office is where the work is not actually done.
4. The Leading Indicator That Trails
Definition. A metric introduced as a leading indicator of business success, which over time becomes the thing the team is optimizing for instead of business success.
Examples. Daily active users (originally: a proxy for habit). Pull requests merged (originally: a proxy for shipping). Tickets closed (originally: a proxy for customers being helped).
Identifying feature. The metric is going up while the underlying outcome is going sideways. This is not a bug. The metric has been successfully delivered.
Emotional function. Allows the team to celebrate progress on a measurable thing in lieu of progress on an unmeasurable thing. The unmeasurable thing was the original goal. The original goal is no longer on the dashboard.
5. The Comparative Snapshot
Definition. A metric whose only meaningful interpretation requires a comparison group that the team has not provided.
Examples. "We have 1.2M monthly active users." (Compared to what? A year ago? The competitor? The total addressable market?) "Our churn is 4%." (Per month? Per year? Per cohort? Excluding which segments?)
Identifying feature. Reported as a fact. Functions as a vibe.
Emotional function. Looks like data. Behaves like a tweet.
6. The North Star That Pointed Elsewhere
Definition. A primary metric, chosen by the team as the singular north star of the business, which on subsequent inspection turns out to point in a direction not aligned with the business succeeding.
Examples. Time spent in the app. Total content created. Total connections formed. Number of A/B tests run per quarter.
Identifying feature. The metric has been going up for three quarters. So has the metric directly downstream of it. So has the team's confidence. So has the customer's irritation, but that is on a different dashboard, which nobody is paid to look at.
Emotional function. Provides directional certainty. Direction may be off, but team is, demonstrably, headed somewhere.
A field test
You can identify a vanity metric using a single question:
"If this number doubled overnight, what would actually be different?"
If the answer is "we'd feel better," it is a vanity metric. If the answer is "a customer would have a measurably better day," it is not.
Most metrics on most dashboards fail this test. This is not a scandal. This is the operating state of the modern dashboarded organization. We mention it here because mentioning it is the cheapest thing we can do.
There is no fix. There is only the practice of asking the question, every quarter, of every number on every wall. The asking is what we mean by keeping receipts.
— Filed by The Receipts Department