Webinar – Using Metrics to Drive Your Software Security Initiative
0 likes97 views
The document discusses the importance of metrics in driving software security initiatives, highlighting the key differences between metrics and measures. It outlines best practices for using metrics effectively, common missteps organizations make, and how to leverage metrics for decision-making and resource allocation. The document also emphasizes the need for clarity and accuracy in metrics to avoid misinterpretation and mismanagement of security risks.
![CONFIDENTIAL© 2019 Synopsys, Inc.5
What does BSIMM tell us about metrics?
[SM2.1] Publish data about software security internally.
• Just 39% of firms in BSIMM9 are publishing data about their SSI within the organization,
compared to 84% that have identified gate locations (SM1.4).
[SM3.3] Identify metrics and use them to drive budgets.
• Only 15% of firms are using metrics formally enough to drive fiscal decisions about their
software security initiatives.](https://image.slidesharecdn.com/metrics-webinar-final-190702140347/85/Webinar-Using-Metrics-to-Drive-Your-Software-Security-Initiative-5-320.jpg)
Webinar – Using Metrics to Drive Your Software Security Initiative
- 1. CONFIDENTIAL© 2019 Synopsys, Inc.1 Using Metrics to Drive Your Software Security Initiative June 18, 2019 Kevin Nassery
- 2. CONFIDENTIAL© 2019 Synopsys, Inc.2 Intro Me: • Lead BSIMM ops & SSI practice at Synopsys Software Integrity Group • 20+ years (50/50 in consulting) across infrastructure security, software security, and program consulting • 50+ BSIMM assessments Metrics talk: • The good • The bad • The must-haves • But…not in that order • Real stories (but anonymized to protect the guilty)
- 3. CONFIDENTIAL© 2019 Synopsys, Inc.3 Metrics vs. measures Used interchangeably in business—can’t assume what someone is talking about when they use either term. However: –A measure is a numerical observation independent of the process defining how it was taken –A metric is a numerical observation based on standard systems, methods, calculations, and data sources Metrics provide better comparative value over collections of measures: – Consistency in observation process – Consistency in meaning
- 4. CONFIDENTIAL© 2019 Synopsys, Inc.4 What you need to know about SSI metrics in 2 minutes Must • Measure SDL compliance • Get telemetry from primary gates • Create feedback loop to SSDL enhancement • Look at the data! • Consider factors of the program execution and risk Should • Correlate data • Strive for good time series data • Compare value of efforts • Test your theories/ intuition • Tell your AppSec story • Fix your metrics when you know they’re broken • Maximize the security impact of spend Should avoid • Accidentally complicating the numbers • Intentionally complicating the numbers • Rushing metrics development Must not • Make up numbers • Make up stories about made-up numbers
- 5. CONFIDENTIAL© 2019 Synopsys, Inc.5 What does BSIMM tell us about metrics? [SM2.1] Publish data about software security internally. • Just 39% of firms in BSIMM9 are publishing data about their SSI within the organization, compared to 84% that have identified gate locations (SM1.4). [SM3.3] Identify metrics and use them to drive budgets. • Only 15% of firms are using metrics formally enough to drive fiscal decisions about their software security initiatives.
- 6. CONFIDENTIAL© 2019 Synopsys, Inc.6 Misstep #1: Not enough context CEO to CISO: “I see January was a real setback in our open issues from penetration testing. Let’s not let that happen again.” CISO to CEO: “Yes, ma’am.” CISO to AppSec director: “Cancel that new pen test contract ASAP!”
- 7. CONFIDENTIAL© 2019 Synopsys, Inc.7 Misstep #1: Not enough context (cont.) Date % PT # PT open H # PT open M # PT open L 6/2/19 66 1438 796 2148 5/2/19 66 1778 856 2540 4/2/19 66 1619 844 2320 3/2/19 66 1720 905 2450 2/2/19 66 2861 1435 4160 1/2/19 66 2841 1496 4211 12/2/18 66 1381 739 1950 11/2/18 66 1307 651 1880 10/2/18 66 1270 632 1875 9/2/18 66 927 447 1240 8/2/18 47 758 451 1100 7/2/18 47 748 405 1000 6/2/18 48 731 387 950 5/2/18 52 655 388 900 4/2/18 52 627 360 925 3/2/18 52 678 403 933 2/2/18 50 655 364 850 1/2/18 50 606 284 800 12/2/17 50 683 315 900 11/2/17 50 832 414 1200 10/2/17 10 97 35 50 9/2/17 10 105 69 150 8/2/17 10 228 75 200 7/2/17 10 101 48 50 0 10 20 30 40 50 60 70 3/6/17 6/14/17 9/22/17 12/31/17 4/10/18 7/19/18 10/27/18 2/4/19 5/15/19 8/23/19 % Portfolio Tested in Last 90 days AppSec director to CISO: “In January 2019, we began authenticated application testing, which gave us new visibility into vulnerabilities we previously weren’t aware of. Most of these issues were present long before they were discovered, and we’ve started the remediation progress with our business and technical owners. Before that, in August, we were able to bring our testing coverage across our portfolio up from 50% to 66%.” Issues like this are best managed by ensuring executives are familiar with the concepts of “managed” vs. “unmanaged” risk and providing visibility into both areas wherever possible.
- 8. CONFIDENTIAL© 2019 Synopsys, Inc.8 Misstep #2: Bad definition & calculation 0 50 100 150 200 250 300 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec MTTR (days) Total AppSec director to CISO on January 1: “Our most important application security metric is our mean time to remediate, which represents our organization’s ability to remediate our known findings.” October 1: “Good news: We were able to finally close out a critical design flaw discovered in January!” CISO to AppSec director: “That’s great. Unfortunately, your bonus has been canceled due to the September MTTR.”
- 9. CONFIDENTIAL© 2019 Synopsys, Inc.9 Misstep #2: Bad definition & calculation (cont.) Row Labels Sum of TTR (days) Jan 13 Feb 11 Mar 124 Apr 16 May 10 Jun 20 Jul 73 Aug 63 Sep 253 Oct 17 Nov 12 Dec 15 Finding Open Date TTR (days) EOMonth-Remed 4006 1/7/18 240 9/30/18 2360 3/23/18 100 3/31/18 4726 8/7/18 60 8/31/18 1372 7/9/18 50 7/31/18 3345 1/13/18 7 1/31/18 0 50 100 150 200 250 300 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec MTTR (days) Total By not representing “open” issues in the MTTR calculation, the most “difficult” problems were unaccounted for until they were closed. These types of lagging issues are quite common, unless great care is taken in metrics development. Metrics, like software, often contain bugs.
- 10. CONFIDENTIAL© 2019 Synopsys, Inc.10 Misstep #3: Hidden factors, hidden risk “Our severity scale reduces risk to low, where CVSS indicated the issue is not exploitable.” • Misinterpreting “Not Defined” as “Not Exploitable” hid hundreds of critical security flaws from the remediation efforts. • The original intent was to facilitate “prioritization” of risk, but this carelessness resulted in major program setbacks and provoked leadership changes. Image source: CVSS v3.0: Specification Document
- 11. CONFIDENTIAL© 2019 Synopsys, Inc.11 Misstep #4: Breaking the incentive model “Because I’m a penetration tester, my compensation is driven by making applications more secure. The key metric we use is if the applications tested have fewer findings over time.” Every published metric may have an unintended influence on stakeholder incentives. Pressure check how your organization is using data to ensure the incentives align with the overall goals of the organization and security initiative.
- 12. CONFIDENTIAL© 2019 Synopsys, Inc.12 Highlight #1: Using metrics to drive vendor selection “We provided the same application, artifacts, and source code to multiple vendors and our internal testing team with the same allotted time. The complete superset of results was then aggregated and deduplicated, and each vendor was given a percentage score against the baseline. This score was combined with other comparative factors, such as quality of guidance, reporting factors, and cost, and was used to drive our vendor schedule for the next 24 months.” Designing metrics that can inform budgetary decisions is a key behavior of successful SSI leaders.
- 13. CONFIDENTIAL© 2019 Synopsys, Inc.13 Highlight #2: Using a security activity to measure another “We weren’t sure if our secure development training was effective, so we analyzed our defects from multiple discovery sources against our developer curriculum content. Each high and critical issue was cross-referenced with our training content, which led to us identifying both areas where we didn’t have any coverage and areas where we did have coverage but developers were still making mistakes. This gave us clear direction on expanding the security development curriculum and refreshing problematic existing content.”
- 14. CONFIDENTIAL© 2019 Synopsys, Inc.14 Highlight #3: Making the case for SSI growth “After our acquisition, the value of our SSI investment was unclear to our new executive leadership team. Fortunately, we had the tools and automation in place to sample a number of applications not managed by our SSI and compare against the SSI’s portfolio. This demonstrated to our leadership teams the importance of our upstream program elements such as training, IDE integrated secure code review, and threat modeling.”
- 15. CONFIDENTIAL© 2019 Synopsys, Inc.15 Fun resources to inspire a new mindset A fun primer on how to look at data differently, and a fun introduction to thinking about incentives A documented approach that may help you break some of the institutional inertia regarding “bad” numbers How objectives and key results (OKRs) has helped tech giants from Intel to Google achieve explosive growth—and how it can help any organization thrive
- 16. CONFIDENTIAL© 2019 Synopsys, Inc.16 Discussion
- 17. CONFIDENTIAL© 2019 Synopsys, Inc.17 Interested in learning more? Our fourth annual FLIGHT Boston 2019 conference is just around the corner. On Sept. 17–19, Synopsys will bring together leading experts from around the world to help you take your software security/development practice to new heights. You’ll learn about the latest insights and best practices in application security, DevOps and the cloud, and open source license compliance. Visit https://snps.sw-sec.co/FLIGHTBoston for more information and use code FLIGHT19 for 50% off your registration.
- 18. Thank You