This series of two articles (this is part 1 of 2) was originally published in Serious Scrum in June 2020. Several years have passed, but „obsessing about velocity“ is still a thing. Read on to learn why velocity is not a (good) performance metric, and what to measure instead when you need to understand whether your are on track and moving in the right direction.
Recently I participated in a passionate team discussion about engineering performance KPIs. To illustrate a few points, I dug out a presentation about measuring engineering performance I did a while ago to a rather small audience. The title of the presentation was “Stop obsessing about velocity”, but that was more of a hook to talk about more generic topics such as metrics dichotomies and delivery performance indicators.
A lot of my research was based on the book “Accelerate” by Nicole Forsgren, Jez Humble and Gene Kim, which, in turn, is based on the DevOps Research and Assessment (DORA) program conducted (at least partly) by the same authors (before being taken over by Google). The team encouraged me to share the presentation with a broader audience, so I decided to transform it a blog post. If some of you can draw some inspiration from it, my efforts have not been wasted.
Why measure performance?
Before we talk about metrics, let’s keep in mind that questions like “what is a good (engineering) team performance” and “how do we measure it” are contextual and generally not easy to answer. And since product- and engineering teams usually have enough other difficult problems to tackle, we should make sure that if we choose to invest time to answer these questions, we do this for the right reasons. So let’s take a step back, and start by asking ourselves why we want to measure performance.
If you think the answer is obvious, namely because we want to learn what we can improve as a team and test whether our improvement efforts are having the desired effect, consider yourself lucky. You are apparently working in an organisation that believes in team ownership and trusts the teams to self-improve, understanding performance monitoring as a means to drive continuous learning. These are characteristics of a generative organisational culture which Ron Westrum, author of “A typology of organisational cultures”, characterised as “performance-oriented”.
In contrast, here is one widespread reason for measuring team performance that should ring the alarm bells: “Because we are asked to”. And another one, even more blatant: “to cover our asses”. Both of these reasons are quite common and characteristic of companies with so-called bureaucratic (rule-oriented) and pathological (power-oriented) organisational cultures.
What happens if a team measures performance “to cover its asses”? If the monitoring serves its purpose, it will protect the team from blaming, and redirect the blame to another team. It’s easy to see how such “performance monitoring” affects collaboration between teams and the overall organisational performance.
Recent research into team productivity and software delivery performance by Google’s Project Aristotle and DORA confirms that organisational cultures which are characterised by high levels of trust, open flow of information and psychological safety clearly predict better software delivery and organisational performance. In other words, if we are striving for high organisational performance, we should leave our teams to worry about their performance, and let the management focus on providing an environment that fosters trust, psychological safety and gives teams an aligned strategic direction.
Characteristics of good performance metrics
Now that we have a good reason to measure performance, let’s reflect on how we can actually do it. What metrics are good indicators for team performance? Remember that we are measuring performance to learn what and how we can improve, or test whether our previous improvements are working. Which indicators can help with these goals?
We want to take action based on what we learn — measuring things that will not help us drive any decisions or optimisations will usually be a waste of time. Hence, we prefer actionable over informational metrics. Informational metrics can highlight potentially interesting facts, however, they are unlikely to directly change our course of decisions. Often, actionable metrics are contrasted with vanity metrics — however, this refers more to the way data is used (to make things look good) rather than to the nature of the data.
We want to uncover knowledge that will lead us to better decisions. We therefore prefer leading indicators over lagging ones. Lagging indicators are output-oriented. They are easier to measure, but harder to improve. Leading indicators are the opposite: input-oriented, often hard to measure, but easier to improve in comparison.
For example, calories input and output are leading indicators for body weight. It’s easy to step on a scale and figure out how much you weigh. It’s a lot harder to measure how many calories we take in or burn throughout the day. However, while we cannot directly control our weight, we can control the input of calories by deciding not to eat that piece of chocolate cake (OK, I see how that is a really bad example — resisting a piece of good chocolate cake is beyond control for many of us… but I hope you still get the point).
We want metrics that help us to improve the global outcome, knowing the dangers of local optimisations. We also prefer to see trends over numbers, since it is more important to us to know the direction than to see the exact number on a sign post. And we want metrics that are not overly contextual, so that they provide value without too much context, or are at least easy to bring back into their context, because we are aware that contextual data without context can be treacherous.
Quick look at Sprint Velocity
Many engineering teams nowadays work in sprints, and there is a reason why my original talk was called “stop obsessing about velocity”. So let’s take a short digression into sprint velocity, probably the most famous and most commonly misunderstood Scrum metric (by the way, can you guess how many times the official Scrum Guide mentions velocity? You’ve guessed right, exactly 0 times).
Take a look at the two (real) velocity charts below. What do they tell you about the respective team performances?
If you’ve been able to extract a lot of useful information about performance from these graphs, I suppose I should congratulate you on your creativity and imagination.
Now, I am not saying that sprint velocity as a metric is useless. In fact, it has some really good characteristics. It’s actionable, although not for performance improvements, but for forecasting whether a sprint goal is achievable and limiting work in progress during a sprint. It’s very easy to measure, especially when you use tools such as Jira anyway, where you basically get it “for free”.
However, sprint velocity is also highly volatile and contextual. It is influenced by team changes, unplanned work and all kinds of unpredictable circumstances. It is prone to the “observer effect”, meaning that it is often manipulated even without ill intent, just because teams know that it is being observed. And it becomes ugly in the wrong hands, which, unfortunately, happens a lot in “Dark Scrum” land. Velocity not improving over time? Looks like the Scrum Master is useless, let’s get rid of that overrated role. Not reaching the number of stories you committed to? Need to work harder, put in some overtime! Needless to say that such “actions” do not help to sustainably improve performance.
In fact, and that is the main point in this discussion, sprint velocity does not correlate to any meaningful performance indicators. In ideal circumstances it describes the rate at which a product (not the team!) is moving forward, but in that regard it’s prone to error and too often misused. It also says nothing about the reasons a product is moving forward at a faster or slower rate. It does not say whether or how efficiently desired outcomes are being achieved. And it often deceives teams into focusing on the wrong problem, e.g. how to improve velocity rather than how to improve performance. It is useful as a means to improve forecasting quality within a team, but it is pretty useless as a performance metric.
So if sprint velocity does not help us with measuring performance, what does? In the next post, I’ll look into the answers the DevOps research by DORA provides us with, how these can be practically applied and why we usually need to dig even deeper
Schreibe einen Kommentar