The Illusion of Control: Seven Deadly Wastes in Your Devops Practice

Editor's Notes

• #3: Have you ever showed up to work on a Monday morning after a long weekend and felt a bit like this guy? I know I have – especially after this past Thanksgiving holiday weekend. Today we are going do a deep dive into what constitutes a “Software Supply Chain” and explore the radical concept that if you are developing and deploying modern software, you most definitely have one. At this point, you may be wondering the same thing as Homer, where is my software supply chain? Or even what is a software supply chain?
• #4: Before we start, I want so share a story with – not sure how many people have heard of Edwards Deming – but many in Japan credit Deming for what has become known as the Japanese post-war economic miracle of the 50’s, when Japan rose from the ashes of war to become the second most powerful economy in the world in less than a decade founded on the ideas Deming taught. He tried to share his quality message with Detroit automakers but was largely ignored – so he took his message to Japan and companies such as Sony and Toyota paid attention; the quality principles he espoused which changed the economy of an entire nation are embodied in the 3 tenets of supply chain management – principles that eliminate waste and rework, minimize technical debt, and dramatically improve the quality and efficiency of your cont. delivery operations. So what are those 3 principles?
• #5: If you leave this presentation with nothing else, remember this:
• #6: As we embark on our deep dive into our software supply chain and why you should care, we are going to be heavily relying on Sonatype’s State of the Software Supply Chain Report where Sonatype analyzed OSS component usage of over 106 thousand organizations – the findings tell a remarkable story. How reliant are we on OSS components? What tools are we using to find out about these components and utilize them? Are we concerned with the quality and risks associated with their use.
• #7: What we will discover on our journey boils down to this, the promise of Continuous Delivery is to produce software at unprecedented speed and efficiency – what I call “Raw Innovation” – and yet if our applications are buggy, susceptible to data breaches, or contain onerous license obligations what really is the net benefit to our organization? What we really want is not “Raw Innovation”, but instead “Net Innovation” - net innovation is continuous delivery producing high quality, low risk applications. We should look to move from the Illusion of Control focused on raw innovation with actual control that maximizes net innovation! So how do we accomplish this?
• #8: We start with taking a closer look into our “Software Supply Chain” – we certainly can’t efficiently manage something we don’t understand – Homer Simpson is a loveable character but we really don’t want him managing our cont. delivery operations, do we? <GO THROUGH SECTIONS>
• #10: Many of the companies I talk to don’t even realize how dependent they have become on 3rd party binaries – the idea of securing their custom code is prevalent, but the importance of securing the usage of 3rd party binaries is not even on their radar. The hidden gorilla in the room is their dependence on OSS components and having no idea how to manage that usage. When I show them this statistic and ask them what is the makeup of their applications – the typical answer is “not sure” or “probably a lot”, or even “would have to check” and yet they have no efficient process to even determine this.
• #12: <GO OVER STATS> The suppliers and the manufacturers need to share information. And right now that communication channel is not only broken, it simply doesn’t exist. These 11 million plus components often have problems reported against them, typically security vulnerabilities in the form of CVE’s. Components are updated an average of 4X a year quite often to fix these problems, but how do the manufacturers even learn about it?
• #15: So let’s take a look at the track record of the suppliers by looking at how quickly parts are “repaired”. There is one bright spot here, the best suppliers were remediating problems in less than a week – those be the suppliers you want to seek out and utilize!
• #18: Given the large number of components downloaded is in the billions, it would make sense to maintain a local cache of components to avoid round trips via the Internet? One would think so. I was at a meeting with SalesForce a few weeks ago and was discussing their new company wide continuous delivery initiative – they told me they were scrambling to trim a few seconds off their builds for their new initiative to be successful – so of course they utilize a local repository manager to ensure they meet their goals.
• #20: This is an astonishing statistic, particularly since the majority of repository managers are primarily OSS and FREE, including Nexus.
• #21: So let’s look at the Manufacturers, the software development teams writing – or may I say “assembling” applications – your dev teams.
• #22: So I mentioned that most companies are scanning their custom code for quality and security issues, but what percent do you think have a policy governing the quality and integrity of their binary components – which I mentioned earlier, make up 90% of a typical application?
• #23: Unfortunately, ½ of organizations still don’t have an open source policy – we all need to work together to change that and in a big way.
• #24: Large breaches seems to occur on a monthly basis, Target, Heartbleed, and JP Morgan Chase to name a few..
• #25: Let’s examine the CVE process and how that’s working to secure our applications. CVE’s are security vulnerabilities reported by the National Vulnerability Database and other sources – they always have a CVSS score – a risk assessment score of 1 to 10, 10 being the very worst – we call these “Known Vulnerabilities”. You can think of an 8-10 scored component as having a potential attack vector where an intruder can gain admin access to your server, run arbitrary code on your server, or obtain all your sensitive data. Unfortunately, even lower risk scores make your applications susceptible to attack – for example, Heartbleed had a risk score of 5. <GO OVER SLIDE> The process is clearly broken.
• #26: To put this in perspective, can you imagine Toyota using 27 different out dated versions of an alternator in their vehicle line?
• #27: Finally, let’s look at the finished goods, our applications utilizing the OSS components from the suppliers. Maybe the large number of bad components being downloaded are not getting into our applications?
• #28: Any venture to guess what we discovered? Of course, many of these bad components are being used in our applications.
• #30: One of the reasons I joined Sonatype, was to help make the world a bit more safe – by assisting companies increase the safety of their applications. So these examples alarm me, I would hope they would alarm you as well. As you can see, these high risk components are being downloaded and utilized in thousands and some times hundreds of thousands of applications. The Bouncy Castle component is an interesting one as it is used for encryption - so it means you care . And yet this vulnerability allowed someone to retroactively decrypt all the encrypted data – so all your data you thought you had secured is available to them. In 2014 over 11 thousand organizations downloaded it over 200 thousand times even though the alert was from 8 years ago – and companies are still downloading and using it in their applications today! The HTTP client component vulnerability was discovered in 2012 and yet last year was downloaded 3.7 million times, an astonishing statistic – this vulnerability is not as critical as the others – but opens up your application for “man-in-the-middle” attacks – its even likely you have this vulnerable component in your pocket on a smart phone app you use to connect to your bank, maybe even your pace maker or automobile navigation system. As far as Apache Struts, I am not going to name the vendor, but I scanned one of the more popular build and deploy tools in the market and found use of a CVSS level 10 component version of Struts 2 - what makes this even more risky is that it is so easy to find websites using struts – simply use google to find URL’s ending in “dot dew”. I continue to see vulnerable versions of struts in many of the company web applications I work with on a weekly basis. You may have heard about a recent zero day exploit in commons-collections within a Java deserialization class – Sonatype’s security team assessed this vulnerability to be a level 9.3 and the exploit has made headlines as attack vectors were found in such popular applications as JBoss, Web Sphere, Web Logic, and Jenkins. Sonatype’s security assessment team dived further and found this same vulnerability in 20 thousand other OSS components – an astonishing result a wider attack surface then even previously imagined. So clearly, these vulnerable components are still making their way into applications and creating attack vectors that can easily be exploited – even though CVE’s are reported on them years ago.
• #31: Ok, we’ve discussed what a software supply chain management and the problems we create when we ignore our software supply chain, let’s look into the SEVEN DEADLY WASTES for any devops practice around this – what NOT to do.
• #32: As I mentioned earlier, this is really the gap between RAW innovation and NET innovation – and it can be very wide. Whether it’s a vulnerable component or a risky license, losses often involve millions of dollars. According to the IBM/Ponemon Data Security report, the average total cost of a single data breach rose 23 percent to $3.79 million in 2014 and intellectual property lawsuits against companies like Cisco, and more recently VMWare, occur frequently and involve millions of dollars as well. It just drives home the point – for your software supply chain, each part matters and you are only as strong as your weakest link!
• #33: You might consider this the sin of “Pride” or “Sloth”, we often overlook the value (and risk) we receive through the use of OSS components and other 3rd party binaries. It’s very easy to ignore, but I would urge you not to do that.
• #34: Instead, to increase quality and leverage your developer’s and architect’s expertise, choose fewer, quality vendors with proven track records and specific versions of high use components. One reason for SW Airlines tremendous success and market penetration is they standardized on JUST ONE airplane vendor and model – the Boeing 737.
• #35: Not sure why I chose this graphic, other then I thought it was really cool looking… it might be a stretch, but when any of these guys need a part, they don’t want to have to go to an off-site warehouse for that part – they need it readily available. And for the compliance and quality folks, it’s great to have control and visibility over which parts are being utilized.
• #36: We frequently consider performance and features but often overlook security and license risk. This is probably the most ubiquitous and risky behavior – something we all need to improve upon. It’s important we shift from blame to empathy – developers are not intentionally choosing risky components. We need to give them the tools they need to pick the best components up front with clear policies to guide them and then continue to monitor our component usage across the entire SDLC including when applications are deployed.
• #37: Unfortunately, this is the approach I see many companies adopting and although I applaud their efforts; manual approval processes depending on “experts” don’t scale and won’t fit into a devops shop – instead think automate, automate, automate! Another problem with manual review processes is that once a component is deemed “good”, we often forget that it could go “bad” at any time just one of the many pitfalls associated with the so called “Golden Repository” – as components go bad, our so called golden repository soon becomes brown or worse. I was working with a major financial organization earlier this year – not going to name them – but they found out that some of their developers were actually renaming components so they could use them without going through the review board – the demands on your devops teams is tremendous and taking short cuts is certainly an attractive alternative when faced with the pressure of a deadline. An automated, continuous, policy driven approach is much more effective and efficient – what I like to call a “Golden Policy” approach.
• #38: As a minimum, you should have a BOM for all your applications. Ideally, given a particular component, you should also be able to quickly determine what applications that component is in – consider a new CVE on a component you use in one or more applications like a “recall” – how do you efficiently deal with that?
• #39: Our last deadly waste, focuses on the fact that components age like milk, not like wine. CVE’s often come out 2, 6, 12 months or years after a component is published – once a CVE is announced, we often see attacks based on that vulnerability as soon as a few weeks later which happened with struts a few years ago. Consequently, it’s critical to continuously monitor your released applications for new vulnerabilities including the use of alerts – and the only sure way to do that is to have a tool to automate that check and send those alerts!
• #40: Let’s dive into the 3 principles of Supply Chain Management
• #41: So what makes a quality supplier, it’s rather obvious of course, a track record of delivering quality parts and remediating quickly when there is a problem. In this screens shot, we see detailed information about a component, in this case, struts2-core. To the right we see the popularity, license risk, and security alerts for every version of struts 2 – when you have this kind of information about a component, it ‘s easy to see the track record and mean time to remediate – and determine for yourself if this is a quality supplier.
• #42: What are the characteristics of a quality component? In the automobile industry, the safety of the component is a critical aspect of quality, it’s the same in the software industry especially in the use of open source components. If a component has a known security vulnerability or a risky license – I would encourage you not to use it or at the very least, determine if the license is appropriate for your particular application. Usage is another factor of quality, if a large community of manufacturers are all choosing a particular part – it’s very likely there is a good reason. The converse is true is well, if a particular part is shunned by the manufacturers, there’s a reason for that as well. You may have heard about a recent zero day exploit in commons-collections within a Java deserialization class – Sonatype’s security team assessed this vulnerability to be a level 9.3 and the exploit has made headlines as attack vectors were found in such popular applications as JBoss, IBM Web Sphere, Web Logic, and Jenkins. Sonatype’s data security team dived further and found this same vulnerability in 20 thousand other OSS components – an astonishing result. Providing developers this information on the desk top is critical, here you see that there are 5 versions of commons-collections with high risk security vulnerability – but there are safe versions and a developer can quickly choose among the good versions of this component – companies such as Disney and Intuit find making this type of information available to all their developers a crucial first step in managing their software supply chain and producing high quality software.
• #43: For tracking what you use - a BOM is pretty much a no brainer – but you would be surprised how many companies fail to maintain one for their applications. In this screen shot, we see a BOM but also show the poor quality parts at the top of the list – this is also extremely beneficial in finding your quality issues and providing a clear path to increase the overall quality of your application – start with the low hanging fruit.
• #44: Remember Deming’s quote, “Change is not Necessary - Survival is Not Mandatory”. I would encourage you to follow Toyota’s lead, and start on this journey of managing your software supply chain – put processes in place and acquire tools to automate those processes. Remember these 3 tenets of supply chain management: use high quality, low risk parts, utilize fewer, higher quality vendors, and track what you use and where. By putting these principles to work in our devops practice, we can all be winners!