Counting Vulnerabilities is Misleading

I saw a presentation celebrating an engineering team that reduced the number of vulnerabilities in their application by half. Tacitly implying that fewer vulnerabilities are better, and counting vulnerabilities is a way to assess that. It sounded reasonable, but it’s not nearly as useful as you think. While fewer vulnerabilities seem better, it's a misleading measure. This article explores why and what to count instead.

Units of measure facilitate the communication of quantifiable information. A two-yard rope is twice as long as a one-yard rope. Units are only meaningful relative to their intended purpose. The two-yard rope is longer, but we don’t know if it is more brown or more brittle. Yards don’t measure that. You might assume the two-yard rope is twice as expensive based on a heuristic that costs should correlate with length. But real-world pricing models don’t always work that way. Smaller items may cost more to manufacture or have lower market demand. Similarly, you might assume an application with 20 vulnerabilities is twice as risky as one with 10 vulnerabilities. Alas, that’s not a good assumption either.

Let’s play out why. In the above case, you have 30 vulnerabilities to address: 20 on one application and 10 on the other. However, most published vulnerabilities are not exploitable and pose no material risk to properly managed applications. But let’s imagine one of those is. Which app is it in? It could be either. Remediating 29 non-exploitable vulnerabilities won’t make you more secure. So why assume that the app with 20 vulnerabilities has the exploitable one? Where’s the data that suggested exploitability is evenly distributed? Moreover, it’s possible that the 20-vulnerability application is remediated with one simple library upgrade, whereas the 10-vulnerability application requires more work, or may not have a fix at all. Suddenly, the number of vulnerabilities is irrelevant to the risk calculation. 

Vulnerability remediation is a good habit. Bad things happen that make safe situations unsafe; the remediation ritual is helpful. But remember, most CVEs in the National Vulnerability Database have no published exploitation script. The CISA KEV list (a list the government maintains of known instances of CVE exploitation) is minuscule relative to the list of published CVEs (thankfully!). CVEs are published based on worst-case scenarios where the application in question lacks environmental controls that defend from attacks. We occasionally read about highly exploitable “celebrity CVEs” (like Log4Shell or Heartbleed) a few times a decade. But NVD publishes an average of 100-130 CVEs a day. Most CVEs, like most people, are not celebrities. 

Herein lies the problem: Vulnerabilities represent the potential for harm. They are not intrinsically bad. Some people mistakenly treat vulnerabilities like malware; you must get rid of them all. However, malware is designed to harm. If you find malware on your computer, you have a real issue. In contrast, vulnerabilities are design or implementation weaknesses that might enable harm, and only under certain conditions. Counting them as if they are a unit of measurement is deceptive and leads you to misdiagnose a vulnerability finding.

To illustrate, human skin has vulnerabilities – it can burn, freeze, and is prone to puncture when pressed against sharp things. Use sunscreen, wear gloves during a blizzard, and don’t walk barefoot on broken glass. Measuring skin area (e.g., tall and large people have more of it) isn’t a good way to quantify the amount of risk skin poses to a person. (Babies have less skin and are more vulnerable.) If you wear shoes and clothing, it is unlikely you will puncture your skin today, unless you deal with sharp things regularly. Declaring a vulnerability reminds us to take precautions. Firefighters wear special clothing – when they need to. Skin isn’t “bad” because it is vulnerable. You are much better off with skin than without! Vulnerability means “potential for bad,” which means you need a remediation plan to address the bad things that could take place – clothing, armor, bug spray, bandages, etc. Some are preventative, some reactive, and some reparative. Knowing the vulnerability allows you to practice the right habits.

The most common remediation for a software vulnerability in an open source library is to upgrade to a version that is not vulnerable, assuming such a version exists. It’s a good practice, yet it’s not perfect. The upgraded version may have other undisclosed vulnerabilities. But if you were able to upgrade this time, you can likely do it again. Make it a habit. Another remediation strategy is to determine that you are not exploitable due to configuration settings that protect you. There are others. And the point being: once you know you have to remediate, the conversation shifts from counting the problems to planning the solutions.

We create counting strategies for pragmatic reasons. It’s more effective to say “three pairs of socks” than “two sock-triplets” since we use socks in pairs, and the former counting tells us how many besocked days we have ahead. Saying “six socks” or “two sock-triplets” is factually correct, but impractical. If upgrading the software is your strategy, then the vulnerability count does not matter; performing the upgrade does. When I hear risk management folks talk about the number of vulnerabilities in an application, I roll my eyes the way I would if you told me you have six socks or two sock-triplets. It reveals how little they understand the context of the conversation they are involved in. Maybe it’s not their fault; they are just reading the data from the tools and not applying any critical thinking skills. Oh, that means it is their fault.

Imagine you have an application using Spring Framework version 6.0.16. A vulnerability detection tool reports three vulnerabilities. (CVE-2024-22243, 22259, and 22262) telling you to upgrade to 6.0.19. You do, but now you find CVE-2024-38809, 38820, and CVE-2025-41234. You are about to fix those, and then someone says that Spring Framework 6.0.x is unsupported. Ideally, the application needs to be on 6.2.8 (as of today – but this stuff ages like milk). It doesn’t matter if you have 100, 10, or zero vulnerabilities telling you to move to version 6.2.8. (Most of them are not exploitable in real-world situations anyway. Maybe some are, but who has the time to check?) You should move to the current version today and plan to move to version 7 next year. Pragmatically speaking, you have two outcomes: either your application can perform the upgrade and will be free from the known (and likely non-exploitable) vulnerabilities, or your application cannot be upgraded for some reason. That last outcome is a problem and is not measured by the number of vulnerabilities.

Some readers may suggest that they operate critical infrastructure, where they are always under attack by malicious actors who are constantly probing for vulnerabilities. Therefore, they assume all vulnerabilities are potentially exploitable. Okay, I see you. Let me use another analogy. Imagine after a wild off-roading trip, you blow out some tires. The mechanic tells you that one of your flat tires has 5 holes on the sidewall and the other has 3. You have to change the tires. The number of holes, albeit interesting, is not nearly as important as the number of tires that need replacing. Don’t count the holes, count the tires. Counting the holes helps you make the story you tell sound more dramatic, but it doesn’t get you back on the road.

Bottom line: When you see people presenting reports that use the number of vulnerabilities as a proxy for risk, inspire them to challenge their assumptions. It’s very risky to manage risk by using the wrong metrics. Counting vulnerabilities is the wrong metric, and it sends the wrong message when suggesting that halving the number of vulnerabilities on an app means they’ve cut the risk in half. Instead, count remediation actions. If application A can be remediated with a few actions that are easy to perform, but application B can only be remediated after many other time-consuming changes take place,  then A is better off. It’s worth measuring that.

Coda: I’m a fan of the DORA metrics. They are designed to measure something slightly different than the impact of vulnerabilities, but I’m inspired by their design. I wonder if there’s a way to throw out all the security metrics and just use DORA. I’m not sure. What do you think?

Prediction: Someone may respond with something about CVSS scores. If that's you, please read this article again. I have lots of thoughts about CVSS, perhaps for another article. But this one isn't about severity metrics.