Can we fix all the vulnerabilities?

No

Thank you for coming to my TED Talk

Let's reframe the question maybe. What if we ask "Are zero CVEs possible?"

Also No

But that "no" deserves an explanation. The answer is a bit like the old saying "You can fool some of the people all the time. You can fool all of the people some of the time. But you can't fool all of the people all of the time". Let's rewrite it. "You can fix some of the CVEs all the time, all the CVEs some of the time, but you can't fix all of the CVEs all of the time."

So what does that mean? It means the arrow of time is coming for us all. As software ages, the number of vulnerabilities will only ever increase if we do nothing.

Container images with a very small attack surface and very fast updates are all the rage right now. If you pull the latest image it will have no CVEs reported, usually by Grype, a vulnerability scanner I happen to be a fan of. If you're in a world that needs to make an auditor happy, this is pretty awesome. And we only have to make an auditor happy a few times a year (or "some of the time") so this works out pretty well. If we want to keep these images clean, we have to keep pulling the latest images as fast as they are being released (which is really fast). This is a good goal if for no other reason than being able to quickly update generally means you have a lot of good process and automation in place.

But it should be noted most of us don't use container images "as is". We install our own stuff on top. Our stuff might not be so simple to update constantly. Once we start to include our own things, it gets complicated pretty fast. There's not really a one size fits all solution.

We know we can "fix all the CVEs some of the time". But what would fixing all the CVEs all the time look like?

If we are able to update all our dependencies we might have a clean scan (maybe not, unfixed vulnerabilities is another topic we can unwind in the future). Updating dependencies often comes with API changes, new features we might not want, new default behavior we might not want. It's all complicated, but it's how open source works.

There are of course many possible solutions to the above problems. Depending on who you ask, it can range from "just engineer harder" to "write us a big fat check to make your problems go away". There's no single answer sadly. You'll have to figure out what makes the most sense for you.

To keep things simple, let's just use this model of container base image plus our own stuff. We'll ignore all the exciting solutions for the moment. I think using a base image that's extra small and hardened is a reasonable assumption.

So if we want to keep our vulnerability list at zero, we have to update our base image on a regular basis. A lot of these images try really hard to not break compatibility, so it's probably low risk to update these images often. Then we have to update our dependencies as needed. This won't be low risk. There are three ways you can update images.

1. Constantly update all dependencies, every time there is an upstream release, update what you use. This will break many things on a regular basis.
2. Update dependencies as needed to have zero vulnerabilities. You can rely on a tool like dependabot to only update packages with vulnerabilities. This is lower risk than installing updates constantly.
3. Cheat. Rather than actually updating things, change things in the image so scanners can't pick up contents.

You're probably wondering why on earth "cheat" is number three on the list. It's there because it's time to ask "what is the problem we're trying to solve?"

Why are you trying to have zero vulnerabilities? If your goal is to make an auditor happy, and you're getting an audit twice a year, you only need a clean scan twice a year. If your goal is better risk management, breakneck updates are almost certainly more risky than most of the vulnerabilities you're removing. If your goal is simply "zero", you're going to end up cheating eventually.

So let's go back to our title, what if we asked "can we fix all the bugs?" That would be a question, if asked, would end up with you getting laughed out of the room. Vulnerabilities are bugs we treat special (which I'm OK with). But just like bugs, some vulnerabilities are so lame, it's OK to not fix them quickly (or ever). Of course deciding how to classify vulnerabilities is yet another topic. When in doubt you could probably just do whatever engineering does to classify bugs, they've been doing this for a long time and are really good at it.