Software rotting

Editor's Notes

• #2: Welcome everybody to this session about Software Rotting. My name is Giulio Vian and I will start with a brief overview so you can see if this session suits your taste.
• #3: Today, I hope to convince you that we have serious problems in the way we patch and deploy applications, problems that we must address as an industry. At the core a perfectly working application today, is a huge risk tomorrow. That’s why I speak of decay and rotting, because it is not a slow process. Wear, erosion, rust… They do not convey the urgency and work required to preserve from decay. #1 unless you put it in a fridge or in a can, it starts smelling very soon  #2 those other processes requires time, while rotting requires quick action to stop it I am not sure big an effort is to fix processes and tool to cope with security-related problems – the one this audience is acquainted to --. Security is the main driver, although not the only one. To change process and invest in tools, we have to speak to leadership/executive using a simple but effective vocabulary, so I suggest using the word inflation to convey the idea and start a discussion. As you may guessed, this presentation is a bit visionary, high-level, I will talk about industry trends and process not technology. For those interested in technology details, I recommend the sessions of my friends Michael Kaufmann and Matteo Emili. Now you have a couple of minutes to switch if you are not interested.
• #4: Who am I? I work at Unum, a Fortune 500 company, with more than a thousand people in IT. I studied DevOps for over 10 years, so, no I am not an InfoSec professional. One thing I learned over the years: I try to solve a new problem each day, but some issues take years to go away. Awarded by Microsoft as Most Valuable Professional on Azure DevOps category in the last few years. I speak at international conferences. If you want to discuss today’s ideas or other DevOps topics you can reach me at Twitter as giulio_vian or email me directly.
• #5: Today’s presentation has four sections: The main problem (security) How it impacts developers’ work and Operations’ work and the DevOps perspective, so the overall impact
• #6: I bet you already know this, maybe you saw bits and pieces of it, so a recap should be useful to grasp the overall picture.
• #7: What these vulnerabilities have in common? They affected widely used libraries, generating major security storms. Log4J is a Java library used for two decades OpenSSL a C library at the core of HTTPS pac-resolver a piece of Javascript to configure an HTTP client WinSCPHelper a .NET library Each language stack had/has a major issue in a library.
• #8: …display the same pattern, even more. Why? Apps use a lot of open source libraries, increasingly. And those libraries have vulnerabilities.
• #10: Open a parenthesis. There are substantial differences on the number and depth of dependencies across different developer stacks. An average Javascript app uses hundreds of libraries, while a .NET app only a couple of dozens, Java is somewhere between. I could not find data about other platforms. How is the state for Python, Go, Ruby, etc.? Close parenthesis
• #11: You knew this was coming, ah?
• #12: Both graphs illustrate that we, as an industry, aren’t exactly great at reacting and fixing our applications. The one on the left is data about OSS projects. The one on the right is more interesting because based on telemetry data, a more significant insight on IT organizations.
• #13: When a new CVE appears, your organization’s Standard Reaction© is to send an urgent email to all developers and managers. Everyone has to stop any development activity and focus on the issue! Production must be patched NOW! Loosely related to Security Orchestration, Automation and Response (SOAR)
• #14: Fixing a library vulnerability is developers’ responsibility, but how it works in practice? Let’s see Kevin’s perspective.
• #15: Here we discuss how to identify:1. the code that needs to be patched 2. the pipeline that release that code in Production and some issues that one may face: If more than one branch can reach prod, which one you choose? How do you match the exact version of code? Software Composition Analysis kicks in only through pipelines? Is triggered by the deploy pipeline? The deploy pipeline hasn’t been used in months and doesn’t work anymore (e.g. a token expired, or there is no more an apt agent)
• #16: It does not require a big organization to have a lot of objects to handle. The trend in architecture is micro-services which translates in lot of independent code. It is quite normal to a 1:10 ratio or higher, i.e. one developer works on at least ten different repositories, each independently built and deployed. Things gets worse if teams manage their own CI/CD infrastructure. You must reach every team and ask them to redeploy! Distributed team => time zones, delay in communication.
• #17: Here we discuss how to identify:1. the code that needs to be patched 2. the pipeline that release that code in Production and some issues that one may face: If more than one branch can reach prod, which one you choose? How do you match the exact version of code? Software Composition Analysis kicks in only through pipelines? Is triggered by the deploy pipeline? The deploy pipeline hasn’t been used in months and doesn’t work anymore (e.g. a token expired, or there is no more an apt agent)
• #18: Can be automated? <pause>To my knowledge there are some tools that do some of the work, like GitHub dependabot It scans sources and proposes changes via a pull-request mechanism It does not support all package manager, though, and some features requires GitHub And clearly we need to input which is the correct version to use. We have seen toolchain attacks were the fix was to rollback, haven’t we?
• #19: Fixed the code we only care deploy it to Production, but many of us works in a regulated industry where a Release Management role, separate from developers, may be required by SOX, Basilea, HIPPA, and so on You need speed when it is a 0-day exploit. You should be able to deploy a patch within hours of its release from a 3rd party (an OSS project or a vendor). Thus, your organisation need special type of pipelines called fast-track or expedite. These must balance the audit requirements with the speed, so they must be restricted for urgent patching. Only a new CVE or a communication from the Security team can enable them with all pre-approval. They have special checks, for example that changes are limited to build scripts (pom.xml, build.gradle, *.csproj, Makefile, package.json, … name it) or few lines of code. This prevents changes in functionality.
• #20: It should be clear that it is impossible implement a fast-track/expedite pipeline without a thorough regression-test suite. Sometimes security patches introduce subtle changes in behaviour. Without a good test-harness, you increase the risk of disrupting production. A high percentage of code coverage is a good indicator that the tests minimize the risk.
• #21: I think I spoke enough of developers. Let’s see what means for Steven
• #22: Kevin stops and thinks: I need to look at my pom.xml (build.gradle, *.csproj, Makefile, package.json, … name it) for references to Log4J (or whatever is vulnerable). Oh, but I use SLF4J which in turns… indirect dependencies! I need a tool just to find all possible references recursively. Oh Oh, our Tomcat configuration is using Log4J! I must check more than my JAR file, says Kevin. And rebuild the Docker container…
• #23: Nowadays containers are a common vehicle to package applications so we have an additional piece to manage and we use automated pipelines for building container images. The base image version maybe in the source Dockerfile or as a pipeline parameter. In any case the Ops person must have a grasp of running pipelines or ask the developers. This won’t happen frequently, or would it?
• #24: You can reuse the stats I showed you to make people aware .NET Core 3.1 3.1.0 December 3, 2019 3.1.22 December 14, 2021 got 22 patch releases in 3 years i.e. every 45 days/6 weeks Node v14 (Fermium) Active LTS start 2020-10-27 v14.15.0 2022-02-01, Version 14.19.0 total 19 releases in 463 days or 66 weeks i.e. every 24.4 days JDK 11 Java SE 11 (LTS)September 25, 2018 11.0.13+8 (GA), October 19th 2021 total 13 releases(updates) in 1121 days i.e. every 12.3 weeks or 86.2 days Go 1.16 released 2021-02-16 go1.16.14 (released 2022-02-10)total 14 updates in 360 days i.e. 26 days go1 (released 2012-03-28) -> go1.17 (released 2021-08-16) 17 major releases in 3429 days or 490 weeks MongoDB 5.0 5.0.0 - Jul 13, 2021 5.0.6 - January 31, 2022 total 6 releases in 203 days or 29 weeks i.e. every 4.8 weeks
• #25: A simple pattern would be to refresh dependencies every night and redeploy. There are important caveat though, that severely restricts applicability. Some stacks are more fragile than others (Javascript/npm) and automatic update may very easily break applications. Deploying a new version is running against a wall if the pipeline has no automated testing, or testing is poor. Lastly, rebuilding everything has a cost (some ballpark measures: Microsoft-hosted is $40 per parallel job, GitLab $10 per 1,000 minutes, other have more complex formulas).
• #26: Current tooling may offer some information but a well-rounded process lot of cross-reference data. Dependency management is a weak spot in general, SCA (Software Composition Analysis) can identify vulnerabilities in libraries. Use of API may be caught by security scans Artifact management tool can track the source (build) of binaries if properly used. Pipeline knows which repositories they use, what we need here is ability to call a REST API that tell us the dependency. If you can use such tools, great. Maybe you need to follow a bit of conventions and write some query tools. In the worst scenario, you have to build and maintain your own database.
• #27: Annie’s perspective security is a cost, this continuously updating is expensive, right?
• #28: I won’t teach you all the costs of neglecting security and quality. A good source is 2018 report of the [US] Council of Economic Advisors titled “The Cost of Malicious Cyber Activity to the U.S. Economy”. It lists 13 different economical impacts of an adverse incident [see below] So how can we justify the budget increase? Loss of IP Loss of strategic information Reputational damage Increased cost of capital Cybersecurity improvements Loss of data and equipment Loss of revenue Public relations Regulatory penalties Customer protection Breach notification Court settlement fees Forensics
• #29: Can we explain it using technical debt? What is technical debt, precisely. Although the use started with Ward Cunningham in 1992, I found that scholars have a more precise definition. {{read}}it clearly does not match the observation: software decay due to external changes, not by developers’ action Should we use another term? Johannes Holvitie, Sherlock A. Licorish, Rodrigo O. Spínola, et al. - Technical debt and agile software development practices and processes: An industry practitioner survey - Information and Software Technology, issue 96 (2018) p.142
• #30: I think so, and I suggest to talk about inflation. As all metaphors it has limits: it is impossible to restore value for a currency like you can with software (unless you remove some digits and convert to a new currency).
• #31: All the mechanism suggested before (expedite pipelines, dependency metabase) plus Reducing the run-times Strong policies for quality
• #33: I should have demonstrated that there is an urgent need to automate rebuild and redeploy applications and underlying stacks at scale. We find new flaws in dependencies every day. Do we have enough resources to manage at scale or the process is still heavily manual? How quickly do we react without automation? Have we covered the entire application portfolio? Solving this problem requires all parties to collaborate: development, operations, management, and vendors. Hope I helped you understand toolchain problems in a broader way and will be easier for you to discuss with colleagues, managers and leadership.
• #34: I am open for questions. If you prefer you can each me via Twitter @giulio_vian or email me directly giuliovdev@hotmail.com.
• #35: I am open for questions. If you prefer you can each me via Twitter @giulio_vian or email me directly giuliovdev@hotmail.com. If you want to discuss today’s ideas or other DevOps topics you
• #42: …are there tools to support me and detect vulnerabilities in the code I deliver?Yes, there are BLAH Ma Guilio non si perde d’animo e sa come trovo vulnerabilita` nel codice e nelle librerie usate: SAST e SCA! Static Application Security Testing (SAST) analizza i sorgenti per errori come il mancato controllo dell’input o SQL injection. Software Composition Analysis (SCA) analizza i binari o i sorgenti per identificare le versioni di librerie in uso e controllare in un database continuamente aggiornato se hanno vulnerabilita` note. Quegli strumenti SCA che validano i binari sono in grado di indentificare anche componenti di runtime o del Sistema operative riguardo a vulnerabilita` note. Guilio non ha budget e quindi usera` un versione open source o freemium per la sua ricerca. E chiudiamo la parentesi https://docs.docker.com/engine/scan/
• #43: There are studies that quantified in million of dollars I mentioned before that a successful attack can significantly impact the bottom line, didn’t I?
• #45: Implementing the convention and tools, described before, can be expensive, depending on your situation. How to justify this work? It balance with the risk of falling down because of an attack. The last item is the reduction of velocity mentioned in “Business Impact” slide