The two design flaws that make your metrics easy to game

“Gaming the system” is a common objection to the introduction of metrics, with skepticism coming from executives and engineering alike. This resistance isn’t unwarranted. Goodhart’s Law is commonly cited in this circumstance: once a measure becomes a target, it ceases to be a good measure. It’s prudent to consider system design, and potential unintended consequences from the design, at the very start of any metrics program implementation.

At its core, Goodhart’s Law warns us against making a measurement a target. Specifically, these two design flaws create systems of measurement that carry the most risk:

• There is just one measurement (lines of code, security incidents)
• There is direct incentive tied to that measurement (promotion, bonuses)

To see this in action, we can look at the Cobra Effect, a classic example of Goodhart’s Law. The term comes from an event in colonial India, where the British government offered a bounty for every dead cobra, with the aim of reducing their population. The program had initial success. But before long, some entrepreneurs gamed the system, breeding cobras specifically to kill them for profit. When the government caught on, they abruptly ended the bounty program. Now with no financial incentive, these breeders simply released their now-worthless cobras into the wild, making the problem worse than before.

In this example, the system was poorly designed in that it met both criteria: there was only one measurement (number of dead cobras) and a direct incentive (bounty payments). Without any tension metrics, or measurements that would tip off overseers to the unintended consequences unfolding, it was nearly impossible to see the negative effects of the system.

Measuring the cobra population in addition too cobras brought in exchange for bounty could have more transparently shown Goodhart’s Law in action, because the population numbers were not decreasing, though the number of dead cobras were decreasing. 

In the Cobra Effect example, and many others just like it (like getting 10,000 tiny nails if you measure output by count, and one giant nail if you measure output by weight), we can see Goodhart’s Law play out.

But Goodhart’s Law tells us more about human behaviour and less about what a good metric can look like. Humans are hard-wired to maximize incentive for ourselves. So if we are presented with a system in which our bonuses, job security, or even just praise are tied to moving a metric up or down, groups of people will inevitably find the shortest, easiest way to do that – even if it goes against the spirit of what’s being measured, and leads to unintended consequences, or even making the original problem worse.

Goodhart’s Law warns us against what may happen with a poorly designed system, but doesn’t tell us what to do about it.

To avoid the pitfalls, we need to plan for them. We know how humans will react to a poorly designed system of metrics. But Goodhart's Law isn't a reason to avoid using metrics altogether. If you worry about people gaming the system, remember: it's a problem with the system, not with the people.

It’s our responsibility to design better systems, by using multi-dimensional systems of metrics, and by avoiding individual incentives tied to a single measurement.