The Promise and Peril of Metrics

The goal of measurement is to not only do things right but do the right things and continuously improve doing that.

We can only manage what we measure. Metrics provide feedback. Metrics are part of transparent visual management allowing pulling. There are many great things about metrics. However, there are people extremely obsessed by metrics who end up creating a huge and sophisticated set of meaningless metrics and that some managers may put a lot of energy on getting better indicators, just because they want beautiful numbers to report and not because they're genuinely interested in helping their teams to improve. So what are the pros and cons, the promise, and peril of metrics?

A measurement system is a necessary foundation for continuous improvement: Metrics can help you get some objective perspective on what you are trying to manage, but they need to be crafted and interpreted well. That is hard. If your team or organization doesn't have a (preferably simple) way of figuring out whether over a short span of time your work is better than in the previous period (and created more/enough value for the end-users), how will you reason about where to invest in improving? Surely you won't just try stuff at random, will you?

Another reason to use metrics is to help stakeholders understand what is going on: All stakeholders may not be in a position to visit the team or talk to team members, so the right metrics can be helpful is to track progress in an improvement initiative. Without measurements, it can be hard to tell whether attempted improvements make the situation better or worse. The maxim "You can't manage what you don't measure" has come to be taken as a truism. The right metric is requested in the right context but without any explanation of why it is requested, without concern for whether the gathering is onerous, or without concern for whether there is a better or easier way to gather metric that achieves the same goal, the measurement can easily go wrong.

Metrics help a team "fail fast" or show value delivered: Metrics is useful only when you can act on the value. The value is just a form of knowledge when there is no action. So you need to know the exact purpose of the metrics and what follow-up action is needed if it does not meet the expectation. The different metrics only make sense at different stages in organizational maturity. Metrics is a tool in the toolbox, but just because you have a hammer, not everything is a nail. While it sounds a bit cliche, it is true. So focus on metrics that help you identify trends, outliers, ask informed questions, create conversation, but ultimately you manage relationships, not metrics. Metrics themselves won't influence the way people behave, but the way they are used will force people to change their behavior.

Metrics shouldn’t motivate a team to game the data: One of the biggest problems with metrics, in general, is that, once you have them, people will try to game them and they influence the mood and morale of the people that read them. We all have our biases. They are a necessary aspect of survival. However, they can filter and even distort qualitative evidence and prevent us from seeing the truth. Complementing qualitative insight with quantitative insight can sometimes give us a better picture of reality so we make better decisions and get better outcomes. Often you can see problems in using metrics when people aren't clear about what information they want to collect and how they intend to use the information to support decisions. This can lead to people trying to measure everything they can think of and display the information in every way possible, with no apparent reason for any of it.

There is a lot of metrics abuse and false assumptions of the metrics: But sometimes, people are obsessed with measuring things and asking for more and more measurements. Each new measurement takes some form of time to gather, collate, and interpret. There is a lot of metrics abuse. People use inappropriate metrics for the situation. People twist metrics to match their agendas or their preconceived notions. Some of these have led to metrics avoidance. Next, most measurements are exceptionally subjective, they have loose logic and reasonings and make many assumptions. In some organizations, measurement is being measured and then evaluated. Decisions are then made on these so-called metrics that complete and utterly wrong, based on many false assumptions of the metrics. Lots of people institute metrics without understanding them or understanding the assumptions upon which they are built. Why? Because some process told them to or because they worked for them in a completely different context.

The perception will come from the usage you're doing with metrics: Assuming an organization believes that metrics can lead to continuous improvement, it won’t be just a matter of explicit communicating the intention behind metrics, but a matter of coaching and leadership to guide the team to understand the purpose of doing that and engaging on that. The perception comes not only from what you're doing with the metrics, but from whatever the team suspects you might be doing with them, including a lot of irrational (or rational) assumptions based on relationships, trust, and past experiences. Never underestimate the capacity for the human mind to weave a story regardless of the explicit communication and especially when it's overwhelmed by the rumor mill. So trust is important to gain a positive mentality. Management wants people to understand what they are doing. If you want to do metrics, you want people to understand what those metrics are, what they are trying to achieve, and why you think it is appropriate in that specific context. Perhaps they will agree with you. When the team or the organization is measuring for the sake of getting numbers, you are creating waste and reducing productivity and team satisfaction...

What gets measured, gets managed. Metrics are not the end-all solution to management, but simply another set of tools, data, and information sets. Numbers permit one to collect and build out a quantifiable history for reference, particularly for trending. Businesses need to avoid vanity metrics and really focus on key metrics that correlate to better business outcomes. The goal is to not only do things right but do the right things (and continuously improve doing that).

Posted in: Digital Pitfalls,KPIs,Lessons from Failures