Security Vulnerabilities in Third Party Code - Fix All the Things!

Editor's Notes

• #5: (Jake) This is a hard issue for an organization to get their heads around.
• #6: Who here is familiar with a CVSS Score? Does your company have an active SDL program? Do you use the STRIDE Model or another vulnerability impact classification system? Do you know who at your company handles product security incident response? Do you have a product security incident response team?
• #8: Security is a subset of quality, but with often overlooked costs. This is very much a pay me now or pay me later situation.
• #10: (Jake) There are lies, damned lies, and statistics, and while this talk is stats heavy there will be a lot of contextual discussion about validity. Starting off with vulnerability trends, . But seeing trends can help here are some from us! =) If we take a step back and look at code quality. It is clear that vulns are not decreasing, we are still producing poor code. Doesn’t matter the complexity for this point but not all vulns are equal as we know. You can still see here that we are seeing around 10k vulns a year and this isn’t even showing cloud / multi-tenet issues or vulns produced at organizations.
• #11: (Jake) Prior to Heartbleed, we were not hearing very much about third party libraries A fancy logo can go a long way!
• #12: FREAK
• #13: POODLE – scary!
• #14: Logjam
• #15: (Jake) BASH
• #16: (Jake) BASH – ShellShock lame ------ BashBleed lives! https://news.ycombinator.com/item?id=8361574
• #18: Not just openssl…….. Other sexy logos to fear at well! And we recycle names of vulns! 117579 2013-01-12 GNU C Library (glibc) nss/digits_dots.c __nss_hostname_digits_dots() Function Heap Buffer Overflow (GHOST) 79214 2005-06-08 Opera Script Code Obfuscation (Ghost) 17334 2005-06-08 Microsoft IE Script Code Obfuscation (Ghost) GHOST: glibc vulnerability (CVE-2015-0235) GHOST is a 'buffer overflow' bug affecting the gethostbyname() and gethostbyname2() function calls in the glibc library. This vulnerability allows a remote attacker that is able to make an application call to either of these functions to execute arbitrary code with the permissions of the user running the application.
• #19: (Jake) How much was impacted by Heartbleed? Lets just agree a lot….
• #20: (Jake) Counting vulns part 2
• #22: (Kymberlee) 7 minutes for section – will see how long this section is when analysis is completed.
• #23: (Kymberlee) Minimum requirements to start talking to your engineering partners about third party library vulnerability trends. There are several sources of free vulnerability data. You can go straight to the vendor advisories, NVD, CVEDetails, OSVDB… It will be better than no data. Of course this IS easier with a paid vulnerability data source that is higher quality and a database and some SQLskills, but you CAN do this without those if you don’t know how to write SQL but are a Pivot Table Ninja.
• #24: NVD and OSVDB are making a valiant effort but you get what you pay for, and vuln data is no different. Here is an example: We e.g. know FFmpeg is a library of concern to many orgs. Compare our info on that library to CVEDetails/CVE/NVD. That should say it all. http://www.cvedetails.com/vendor/3611/Ffmpeg.html <-- CVE Details has total of 191 https://vulndb.cyberriskanalytics.com/vulnerabilities/search?vendor_id=157346  <-- VulnDB has over 1000
• #25: (Kymberlee) So who uses OpenSSL? Gmail, Tor, LastPass, AWS, GitHub, Bitcoin, Yahoo, Instagram, Flickr, Etsy, Tumbler, Netflix, GoDaddy, YouTube, WordPress, Pinterest, Dell, VMWare, BlackBerry, and Dropbox. Oh, wait, that’s not 100. Let’s add these to the list. And multiple products by HP, Oracle (including Java), F-Secure, IBM, MySQL, Dell, Novell, OpenBSD, Intel, Juniper, VMWare, Rapid7, nginx, Huawei, Trend Micro, Linux, Tableau, McAfee, F5, Cisco, Fortinet, Sophos, Python, Citrix, SUSE, Ubuntu, Debian, FreeBSD, RedHat…
• #26: CVE2011-0226 is a CVSS 9.3 vulnerability in the FreeType library that was used in JailbreakMe for the iPhone in 2011. If this were a locally exploitable vulnerability, the jailbreak would primarily be a terms of use violation. But it wasn’t. A missing parameter check allowed a malicious font program to move a stack pointer outside of its bounds, providing read and write access to various fields of the stack. This vulnerability was remotely exploitable via anything that uses FreeType to parse fonts.
• #27: (Kymberlee) We’ve tracked LibPNG to over 130 applications including products from Google, Apple, Adobe, Microsoft, VMWare, BlackBerry, Linux, and Oracle. This thing is eeeeverywhere. Oh, and one of those icons? That’s Webkit. That’s right, another third-party library in a third-party library.
• #29: Adobe’s efforts to bring Flash to connected TVs, Blu-ray players and other devices, like its mobile Flash plans, were part of its Open Screen Project, which aimed to create a consistent app runtime across multiple devices. The idea was that developers would be able to create a Flash application once and be able to distribute it across web browsers, mobile devices and TVs. “Adobe will continue to support existing licensees who are planning on supporting Flash Player for web browsing on digital home devices and are using the Flash Player Porting Kit to do so.” -2011
• #30: I think we can all agree that Java wins the vulnerability spread competition, its on Mars.
• #31: METHODOLOGY: Used free public data, each individual vendor’s release notes. Paid intelligence feeds like VulnDB will have higher counts, because they are actually analyzing the updates to uncover silently patched vulnerabilities the vendor did not publicly acknowledge For example, 71 listed for OpenSSL, VulnDB has 84 as FYI. (Kymberlee) Let’s start with the poster child, OpenSSL.
• #32: Just because Java and Flash have the highest counts of vulnerabilities does NOT necessarily mean their libraries are less secure than Freetype or libpng. Both Oracle and Adobe have the resources to hire dedicated security teams, consultants, run bug bounty programs, etc. The point of sharing this data is so you can have a conversation with your development teams during the design phase of SDL about their vulnerability management plan for the libraries they choose to integrate.
• #34: So you’re saying “I thought we weren’t supposed to be using vuln counts or number of releases as the measure of security” and we totally agree. Today there is no One Measure to Rule Them All, just a group of metrics about cadence of release, volume of code churn, severity of issues, and to help you make informed SDL decisions. BUT WHAT ABOUT WINDOW OF OPPORTUNITY? Does it matter how long the vulnerability was known for? (transition to Jake)
• #36: (Jake) There are some current attempts at accountability already in place.
• #37: Customers don’t really feel this way about your products.
• #38: (Jake) Companies that consume third party libraries are not allocating dev resources to security test ‘someone else’s code’. Right or wrong, everyone seems to think that someone else will find the vulns.
• #39: (Jake) Companies that consume third party libraries are not allocating dev resources to security test ‘someone else’s code’. Right or wrong, everyone seems to think that someone else will find the vulns.
• #40: (Jake) There are some current attempts at accountability already in place.
• #41: https://en.wikipedia.org/wiki/Vulnerability_management This is the entirity of the wikipedia entry on vuln mgmt The second paragraph starts with network security, mentions using fuzzing to identify buffer overflow test cases, and then goes back to network security for the remainder of the article. Its pretty awful, I should edit this and clean it up so Appsec and Netsec are not conflated.
• #42: There are even some very fancy models online for Vulnerability Management. Intrusion Monitoring, Event Correlation, Asset and Patch Management…
• #44: Well ok, so fixing code you haven’t written yet IS pretty low cost. The main thing to take away from this is that the earlier you fix vulnerabilities, the lower the development cost. Many people think of Vulnerability Management as post-release patch management, but effective vulnerability management starts during development and is an integral part of the Secure Development Lifecycle. Next slide: GOAL, not SDL
• #45: This sounds simple, but
• #46: This sounds simple, but some development and security teams take an adversarial approach or worse, development just ignores their security team entirely. Why does this happen? Is it just that developers want to work on cool new features and security professionals are holding them back, always saying no to their neat new designs and telling them how bad their code is? It might be some of that, but fundamentally there is a lack of understanding between builders and breakers. This leads to a lack of respect in some cases, which is counterproductive. And vulnerability management is a space where partnership and mutual respect is crucial.
• #47: Some SDL models merge Requirements & Design for a 6 Phase Lifecycle instead of 7 Where are vulnerabilities introduced in products? Design and Implementation When are they identified? Verification and Response. But wait… Do you use third party libraries in your development? Then vulnerabilities are being identified all the time, even before you’ve started the design process.
• #48: How does vuln mgmt in app sec compare to net sec? In Network Security you frequently see the first step in Vuln Mgmt as “Scan for Vulns”. That’s just a very specific way of saying “identify an issue”. We’ll come back to scanners in a moment. The second step in most Net Sec Vuln Mgmt process flows is to assess vulnerabilities for criticality. Triage, repro, and prioritization. Net Sec Vuln Mgmt models typically then go to “Patch Applications”, without acknowledging that testing is required prior to deployment Fourth in NSVM is to report on patch deployment, which is typically to executives and leadership teams, not customers. In application security if you are pre-release, internal reporting is consistent, but if the issue was identified externally after product release you’ll have to document it publicly with an advisory.
• #49: in addition to planning regular maintenance to keep your third party library up to date, you have to have robust reactive incident response for when your third party libraries have an emergency event. How does vuln mgmt in app sec compare to net sec? In Network Security you frequently see the first step in Vuln Mgmt as “Scan for Vulns”. That’s just a very specific way of saying “identify an issue”. We’ll come back to scanners in a moment. The second step in most Net Sec Vuln Mgmt process flows is to assess vulnerabilities for criticality. Triage, repro, and prioritization. Net Sec Vuln Mgmt models typically then go to “Patch Applications”, without acknowledging that testing is required prior to deployment Fourth in NSVM is to report on patch deployment, which is typically to executives and leadership teams, not customers. In application security if you are pre-release, internal reporting is consistent, but if the issue was identified externally after product release you’ll have to document it publicly with an advisory.
• #50: So lets talk about where those vulnerabilities come from. Internal Security Team may be FTEs or consultants, but typically they are doing white box vulnerability assessments, running static analysis tools, fuzzing source code… Depending on your organizational structure they may be in a centralized security team, embedded in your product development team, or both. External Security researchers typically report after product release to your SIRT team on exploitable vulnerabilities they’ve found through black box testing. Maybe they’ve run some tools, but unless your product is open source this is typically black box testing. Who owns monitoring and managing vulnerability reports in your third party libraries?
• #51: If your internal security team found a CVSS 9.0 is that higher priority for your dev team to fix than a CVSS 7.5 in a third party library that you use? What if that CVSS 7.5 vulnerability is being actively exploited on a competitor’s platform? I give a whole talk on vulnerability prioritization with Michael Roytman of RiskIO and David Severski of Seattle Children’s Hospital, I could spend an hour on this alone. You need to determine what your priorities are beyond just CVSS score which is inherently flawed and is not a meaningful predictor of exploitability.
• #53: Sustainment Planning? Yeah, especially if you use third party libraries or are re-using code from a legacy product that external security researchers are looking for vulnerabilities in. Who is responsible for monitoring 3rd party libraries? Who is repro testing newly found vulns in legacy products? Do you know every component that uses OpenSSL? The versions? When is the last time you patched OpenSSL? And the time before that? Do you have any idea how often OpenSSL releases updates? If you only patched OpenSSL last week, during Shellshock, and Heartbleed, you missed patching a bunch of critical vulnerabilities in between.
• #54: You can’t prioritize your vulnerabilities if you don’t have early agreement on what is and is not acceptable to ship with. This is where you also define SLAs for time to fix based on vulnerability priority. The legal team is your friend, they want to know what you use where for compliance reasons – partnership can help you secure your products! Blackduck, Palamida, Sonatype, others also offer source code scanning for 3rd party libraries.
• #56: You can have a very real impact on how software vendors handle vulnerabilities by making it clear that you care about security. Vendors have goals around satisfied customers. If you’re unhappy with how many issues are not getting fixed prior to and asking hard questions that makes a difference.
• #58: (Kymberlee) Your legal team is your friend here – both for documentation as well as funding for better tooling like these: Legal team: compliance requires documentation, this is how we got started.
• #59: (Kymberlee)
• #60: (Kymberlee) Can’t implement a scanning tool? You can still monitor security releases for the libraries you are impacted by. Some of these are higher quality than others.
• #61: (Kymberlee) Don’t know if anyone from VMWare is here today, but we’d be interested in working with them on VTEM data modeling.