Key Findings from the 2019 State of DevOps Report

Editor's Notes

• #2: Hi, I’m Alanna Brown, Sr. Director of community and developer relations at Puppet. I started something called the State of DevOps Report back in 2012 before anyone really knew what DevOps was or what it would become. Like many of you, I thought DevOps would be dead in a year.
• #3: And here we are today, still talking about DevOps. In fact, I’ve spent the past eight years, surveying over 33,000 technical professionals from around the world, and working with the larger DevOps community and Puppet customers to understand how organizations adopt and scale DevOps practices and the outcomes they’re seeing.
• #4: In just a few hours, we’re releasing our 2019 State of DevOps Report. This year’s report focuses on one of the most challenging aspects of DevOps: integrating security practices into the software delivery lifecycle.  Security is often seen as a necessary evil, but it doesn’t have to be that way. This year’s research shows us that integrating security early and often delivers results even if the path to get there isn’t straightforward.
• #5: Intro slide - Welcome to the topic; introduce context; what we decided to focus on; introduce ourselves, what we found interstin
• #6: Testing in production / driving down costs Tech advancements; microsfervices, containers; cheaper to fix in microservice vs. monolith 
• #7: Why? What do you see about landscaipe? Myike - doesn’t matter; Org issues / feature development
• #8: How do we solve this problem? We’ve seen successful patterns emerge from DevOps that have enabled organizations to build quality and deployability into the software delivery life cycle. There are also organizations out there having success baking security in from the start. We wanted to know if integrating security throughout the software delivery life cycle actually delivers positive outcomes.  I’ll be referring to “levels of security integration” throughout this presentation. The way we defined those levels was by asking people to select all the phases where security is involved.  We then broke those answers out into five levels:  Level 1 - No integration of security in any of the phases Level 2 - Minimal integration (one of five phases) Level 3 - Selective integration (two of five phases) Level 4 - Significant integration (three or four of five phases) Level 5 - Full integration (all phases) 60% of firms include security in two or fewer phases of their software delivery cycle. So most organizations aren’t at a very high level of integration. Combine 2 slides Nigel ask Andi about bell curves -
• #9: Roundtable Yeah, I know, some of you are like “Obvious. Move on.” But most research exists just to  prove what we all already know.  If you’ve already started on your DevOps journey, good news, you are building the capabilities that will enable you to deliver software more securely. Use of version control and continuous integration, automated testing and automated deployment provide an amazing foundation to build other capabilities. It makes it so that making a security-related change is the same as making any other change.   Last year, we discovered that at the highest levels of DevOps evolution, security policies and incident response are highly automated and security teams were involved early in technology design and development.  We wanted to know if this works the other way around.  As you can see from this trendline, the more integrated security becomes the more likely a firm will also be at a high stage of DevOps evolution. 22 percent of firms at the highest level of security integration are also at an advanced stage of DevOps evolution.
• #10: We found that as security become more integrated, confidence in security posture improves.  Does confidence equote to actual security
• #11: 81% of respondents at firms with high integration felt that their security policies and practices significantly improve their security posture. Compare this with respondents at firms with no security integration — just 38 percent had that level of confidence.   Now that doesn’t necessarily mean that they actually are more secure. What’s more important here is the shift in mindset.  When there’s no integration, there’s usually a lack of understanding of the work involved in security and a lot of cynicism. But when teams are fully integrated, there’s a shared understanding and people feel like the things they’re being asked to do really do matter, which makes them more likely to actually do them. If you were to guess what the most impactful practices are for improving confidence in security posture, what would they be?
• #12: What does it mean to actually collaborate across teams How do you make this happen? How do you get people to collaborate on threat model: how do you get vp engaged? Inviting security expert on scrup, having htem sign off on release? Practical and actionalable Anectdotes; anti patters; if you don’t do this what blows up; What works what doesn’t work; We threw in a bunch of practices and I thought for sure that the more tactical practices, like testing would be at the top of the list. Me and my fellow authors were delighted to find that the practices that require strong collaboration and sharing amongst teams were actually the top confidence-builders. The top five practices that improve security posture are: Security and development teams collaborate on threat models. Security tools are integrated in the development integration pipeline so engineers can be confident they’re not inadvertently introducing known security problems into their codebases.  Security requirements — both functional and non-functional — are prioritized as part of the product backlog.  Infrastructure-related security policies are reviewed before deployment.  Security experts evaluate automated tests, and are called upon to review changes in high-risk areas of the code (such as authentication systems, cryptography, etc.).
• #13: Make a list of things - We mapped all of the practices on a quadrant. The x axis represents the importance of the practice as it relates to improving confidence in security posture and the y axis represents the frequency of these practices. The practices on the left are tablestakes. Those are the things you should be doing anyway.  The practices on the right are where you should be focusing your effort. Again, these practices require deep collaboration and happen early in the development cycle. It's not just about shifting security checks left, it's about fundamentally changing the way everyone works earlier in the pipeline. Our recommendation is to start focusing on the practices in the bottom right quadrant because those are the ones that are infrequently used but are incredibly important to improving confidence in security posture.
• #14: Ok, so this is all great feel good stuff, but you might be wondering if any of this actually has an impact on business outcomes?
• #15: Just because you can doesn’t mean you should, but beting able to makes you agile in good ways; Question: deployment frequency bullshit vnity metric discuss? Goes up then goes doen then goes up again; pain is real; gets worse betfore it gets better; Best thing you can do is not try and then not fail This year, we asked two questions about deployment frequency. How often can you deploy versus how often do you deploy. We know of enough organizations now that can deploy more frequently than the business or their customers require. This is a far cry from a few years ago when everyone was obsessed with improving their deployment frequency. I actually think it’s a huge measure of success for a DevOps initiative if you can now deploy so frequently that your marketing team asks you to slow down because they can’t keep up. If you focus on this teal green bar here to the left, this is the percentage of respondents at each level of integration that are able to deploy on demand. Firms at the highest level of security integration are able to deploy to production on demand at a significantly higher rate than firms at all other levels of integration — 61 percent of highly integrated organizations are able to deploy on demand compared to 49 percent of organizations with no integration.
• #16: Doing remedation is good; doing it well is hard; Doing more security finding more problems Unpack level 1 - not integration - same team that does everything; just ops work; small org so can move quickly; security team just process security people and you handle all the other stuff; 45 hae rigor in rpodess A couple of interesting things to point out here. First only 7% of total respondents are able to remediate a critical vulnerability in less than one hour. The majority of respondents are able to remediate in less than a week.  The main takeaway for me is that it’s really hard to actually reduce the time it takes to remediate vulnerabilities because there are too many factors involved, too many stakeholders, handoffs and approvals.  The differences you see here between each of the levels are statistically significant, but they’re not as dramatic as we’d like. But still, any reduction is a good thing because that does reduce your company's risk and exposure. 
• #17: Low severeity different from other results: two of the most clustered; lowest severity and crtical ones why is that? Critcal security is blank check to do whatever it tkes to get it done; med and high interestin and prove processes work; drop everything and fix everything is simple; Gap between level 1 and level 5; ; hard to do Now imagine with me for a minute that you were asked to choose between making a security improvement or delivering a critical new feature that your customers have been asking for and will help you make your quarterly number. Which one would you choose?  We asked a series of questions about like this and found that firms with deeper security integration were more likely to prioritize security improvements over feature delivery.
• #18: So far, I’ve painted a rosy picture, but the reality is that integrating security is messy work, especially in the early stages.  When you’re just starting you don’t know what you don’t know. As you dig deeper though, it can feel a bit like opening Pandora’s box. All of the duct tape and glue that’s been holding everything together is suddenly laid bare before you. We’ve seen this in past reports, too. It’s called the j-curve, which means things start out well because you’re seeing quick wins and then they take a turn for the worse before they get better again.
• #19: Crossing org boundaries We asked if security teams encounter friction when collaborating with delivery teams. Friction is higher in the middle and it never really goes away, even when teams are fully integrated. When we compared those in security roles vs. non-security roles, we found that friction was even higher for security teams in the middle stages.  Understanding that DevOps is fundamentally about cultural change, it’s imperative to remember that different teams will experience this change differently.
• #20: Mike rant Increasing security integration also doesn’t reduce the number of issues that require immediate correction that arise from audits. The path is hardest at levels 2 and 3. We looked at number of audits firms were performing at each level and it turns out that at higher levels of integration, firms are actually doing more audits per year, since often times audits are elective and you can select the degree of difficulty for some audits.
• #21: Add devopssurvey.@puppet.com
• #22: So far, I’ve painted a rosy picture, but the reality is that integrating security is messy work, especially in the early stages.  When you’re just starting you don’t know what you don’t know. As you dig deeper though, it can feel a bit like opening Pandora’s box. All of the duct tape and glue that’s been holding everything together is suddenly laid bare before you. We’ve seen this in past reports, too. It’s called the j-curve, which means things start out well because you’re seeing quick wins and then they take a turn for the worse before they get better again.