![#RSAC
Challenge: Managing Diversity of Scanners and
Security Information Sources
8
Numerous scanners and scanner types available
– Open Source, Existing Enterprise Licenses for on-prem and SaaS models
– Lightweight (take a few minutes) vs Exhaustive (may take hours to run)
Selection of appropriate scanners for comprehensive coverage
– SAST [Static Code Security] Scanner
– SCA [Software Composition Analysis] Scanner
– Embedded Secrets Scanner
– DAST [Dynamic App Security] Scanner
– Infrastructure mis-configurations (cloudformation, kube deploy, etc.)
Ability to process vulnerability feeds from various input sources
Coping with scanner noisiness/chattiness](https://image.slidesharecdn.com/tech-m03buildenterprisedevsecopsinfrastructurefinalv2-notes-220608174349-dddffd76/85/Building-an-Enterprise-scale-DevSecOps-Infrastructure-Lessons-Learned-8-320.jpg)
![#RSAC
High Level Logical Architecture
12
Jenkins based Pipeline
SAST [Static Code
Security] Scanner
SCA [Composition
Analysis] Scanner
Embedded
Secrets Scanner
Layered Integration
Application
Security
Workbench
2
1
4
3
Security
Organization
Findings
Additional Security
Information Sources
JIRA
Code Repository
BU Leader
Architect / PM
App Lead
Developer
Business Unit
Build Applications
Security
Scan Reports
(Immediate)
View & Prioritize
Security
Vulnerabilities
Status Updates
Ingestion (at a cadence)
Create tickets for
Remediation
Updated status
post Remediation
DAST [Dynamic
App Security]
Scanner
Kubernetes
Checker
Scanner+
(Cloudformation,
nodeJS,
python,…)
Offline
Enterprise
Scanners
Team-specific
Scanners
L
a
y
e
r
e
d
I
n
t
e
g
r
a
t
i
o
n](https://image.slidesharecdn.com/tech-m03buildenterprisedevsecopsinfrastructurefinalv2-notes-220608174349-dddffd76/85/Building-an-Enterprise-scale-DevSecOps-Infrastructure-Lessons-Learned-12-320.jpg)
![#RSAC
Apply #1 [Immediate]
18
Agree on an enterprise-wide approach with all stakeholders (Dev
teams, leadership, CSO office)
– Identify DevSecOps efforts that are on-going in different teams and capture their
requirements
Identify if there are any existing enterprise license agreements with
security vendors
– Supplement with use of well-respected open source systems
– Security scanners are a commodity, examine vendor claims of superiority with caution!
Agree on a base level set of security scanners and information sources
– Getting off the ground is key; don’t try to achieve nirvana!!](https://image.slidesharecdn.com/tech-m03buildenterprisedevsecopsinfrastructurefinalv2-notes-220608174349-dddffd76/85/Building-an-Enterprise-scale-DevSecOps-Infrastructure-Lessons-Learned-18-320.jpg)
![#RSAC
Apply #2 [30-60 days]
19
Ensure that the selected tools support varied team development
methodologies and different tech stacks
– This is why a pluggable framework is important
– This will also guide your BUY vs BUILD decision
Agree on how security vulnerabilities should be surfaced
– E.g., Webapp, webex, email, CI/CD links, bitbucket annotations
Identify and engage with early adopters / champions
– Establish regular feedback mechanism](https://image.slidesharecdn.com/tech-m03buildenterprisedevsecopsinfrastructurefinalv2-notes-220608174349-dddffd76/85/Building-an-Enterprise-scale-DevSecOps-Infrastructure-Lessons-Learned-19-320.jpg)
![#RSAC
Apply #3 [180 days]
20
Develop a process for remediation timelines and priority
– Who determines impact and risk and how will it be manifested by your
tools?
– Not every security issue can be fixed easily or quickly, important to have a
tracking process
Culture shift through office hours/trainings
– Developers to become familiar with security vulnerability patterns common
within the enterprise
– Train developers on becoming proficient at remediating these vulnerabilities](https://image.slidesharecdn.com/tech-m03buildenterprisedevsecopsinfrastructurefinalv2-notes-220608174349-dddffd76/85/Building-an-Enterprise-scale-DevSecOps-Infrastructure-Lessons-Learned-20-320.jpg)
Building an Enterprise-scale DevSecOps Infrastructure: Lessons Learned
- 1. SESSION ID: #RSAC Building an Enterprise-scale DevSecOps Infrastructure: Lessons Learnt TECH-M03 Prateek Mishra Senior Director - Security Architecture Developer Experience, ADP Gaurav Bhargava Director of Product Management Developer Experience, ADP
- 2. #RSAC Disclaimer Presentations are intended for educational purposes only and do not replace independent professional judgment. Statements of fact and opinions expressed are those of the presenters individually and, unless expressly stated to the contrary, are not the opinion or position of RSA Conference LLC or any other co- sponsors. RSA Conference does not endorse or approve, and assumes no responsibility for, the content, accuracy or completeness of the information presented. Attendees should note that sessions may be audio- or video-recorded and may be published in various media, including print, audio and video formats without further notice. The presentation template and any media capture are subject to copyright protection. ©2022 RSA Conference LLC or its affiliates. The RSA Conference logo and other trademarks are proprietary. All rights reserved. 2
- 3. #RSAC Agenda What is DevSecOps? Enterprise DevSecOps - Problem Statement Solution Proposal and its Characteristics Demo! Learnings Conclusion 3
- 4. #RSAC What is DevSecOps? 4 DevSecOps is the integration of security into emerging agile IT and DevOps development (and deployment) as seamlessly and as transparently as possible (Gartner) Seems straightforward enough, so why do we need this session? Threat Modeling Static Analysis Component Analysis Dynamic Analysis Build Integrity and Authenticity Configuration Validity Logging Alerting Runtime Tracking
- 6. #RSAC Problem Statement 6 • Many autonomous development groups (10s -100s), developing apps using several different technology stacks • Some groups are working on well-established ”legacy” apps, others on next generation applications with modern tools • Different approaches to detecting and closing security vulnerabilities: with every commit or build or at sprint boundaries or all of the above 6 JS + Java Stack | Jenkins CI | Data Center Development Group A DevSecOps Tools JS + Node.js | Cloud Native CI | Cloud Development Group B DevSecOps Tools Xamarin + Mobile Platform | Another CI | Consumer Devices Development Group C DevSecOps Tools
- 7. #RSAC Challenge: Lack of Uniform View across Departments 7 Lack of a uniform view and understanding of security state of applications – Difficult to assess and compare security posture of various products across departments – Difficult to enforce a uniform level of compliance with enterprise guidelines Cost of maintaining separate departmental infrastructures – Headcount/License/Training/Maintenance
- 8. #RSAC Challenge: Managing Diversity of Scanners and Security Information Sources 8 Numerous scanners and scanner types available – Open Source, Existing Enterprise Licenses for on-prem and SaaS models – Lightweight (take a few minutes) vs Exhaustive (may take hours to run) Selection of appropriate scanners for comprehensive coverage – SAST [Static Code Security] Scanner – SCA [Software Composition Analysis] Scanner – Embedded Secrets Scanner – DAST [Dynamic App Security] Scanner – Infrastructure mis-configurations (cloudformation, kube deploy, etc.) Ability to process vulnerability feeds from various input sources Coping with scanner noisiness/chattiness
- 9. #RSAC Challenge: Developer Enablement 9 How to ensure that developers and development teams are tasked only with security vulnerabilities relevant to them? Rich set of application artifacts - git branches, repositories, build processes, docker containers, application assemblies, cloud accounts - create difficulties in linkage to devs or dev teams Lack of information sharing between development teams Actionability of remediation guidance by app developers – App developers are not security experts!! Process inefficiencies in consultation between development teams and CSO (Security SMEs)
- 10. #RSAC Solution
- 11. #RSAC Solution: Enterprise-scale DevSecOps 11 Shared infrastructure “plugs-in” to the specific development tech stacks and CI+CD frameworks used by different groups Layered architecture accommodates new tech stacks, languages and target platforms Ability to process and manage security vulnerabilities from diverse scanners and security information sources 11 JS + Java Stack | Jenkins CI | Data Center Development Group A JS + Node.js | Cloud Native CI | Cloud Development Group B Xamarin + Mobile Platform | Another CI | Consumer Devices Development Group C
- 12. #RSAC High Level Logical Architecture 12 Jenkins based Pipeline SAST [Static Code Security] Scanner SCA [Composition Analysis] Scanner Embedded Secrets Scanner Layered Integration Application Security Workbench 2 1 4 3 Security Organization Findings Additional Security Information Sources JIRA Code Repository BU Leader Architect / PM App Lead Developer Business Unit Build Applications Security Scan Reports (Immediate) View & Prioritize Security Vulnerabilities Status Updates Ingestion (at a cadence) Create tickets for Remediation Updated status post Remediation DAST [Dynamic App Security] Scanner Kubernetes Checker Scanner+ (Cloudformation, nodeJS, python,…) Offline Enterprise Scanners Team-specific Scanners L a y e r e d I n t e g r a t i o n
- 13. #RSAC Solution Software framework built out of stock components and open source – Scanner layer provides a uniform way to package scanners into docker images We selected a standard set of open source scanners familiar to us Can be replaced by alternatives or licensed versions as needed – Dockerized scanners can be plugged into many different CI pipelines Current focus is on Jenkins-based pipelines, but others are in our backlog Linkage between repositories/projects to products and teams – Based on machine readable meta-data added by teams to repositories/projects – Ensures that code/artifacts/assemblies/docker images/cloud accounts can be linked to products and teams 13
- 14. #RSAC Solution (contd.) Workbench provides a generic data model and uniform GUI for all security vulnerabilities – Allows new scanners or security information sources to be added as needed Including off-line scanners and security information sources that are available asynchronously – Standardized display of vulnerabilities Severity, Remediation Guidance, False Positives, Acceptable Risk Main focus is helping application developers to act on the information • Including ways of sharing information between teams acting on similar vulnerabilities Workbench provides division/product level rollups – Fine grained vulnerability reports at repositories/artifact level – Aggregate vulnerabilities at department/product level – Exposes extent of security maturity across division/products 14
- 16. #RSAC
- 17. #RSAC Learnings
- 18. #RSAC Apply #1 [Immediate] 18 Agree on an enterprise-wide approach with all stakeholders (Dev teams, leadership, CSO office) – Identify DevSecOps efforts that are on-going in different teams and capture their requirements Identify if there are any existing enterprise license agreements with security vendors – Supplement with use of well-respected open source systems – Security scanners are a commodity, examine vendor claims of superiority with caution! Agree on a base level set of security scanners and information sources – Getting off the ground is key; don’t try to achieve nirvana!!
- 19. #RSAC Apply #2 [30-60 days] 19 Ensure that the selected tools support varied team development methodologies and different tech stacks – This is why a pluggable framework is important – This will also guide your BUY vs BUILD decision Agree on how security vulnerabilities should be surfaced – E.g., Webapp, webex, email, CI/CD links, bitbucket annotations Identify and engage with early adopters / champions – Establish regular feedback mechanism
- 20. #RSAC Apply #3 [180 days] 20 Develop a process for remediation timelines and priority – Who determines impact and risk and how will it be manifested by your tools? – Not every security issue can be fixed easily or quickly, important to have a tracking process Culture shift through office hours/trainings – Developers to become familiar with security vulnerability patterns common within the enterprise – Train developers on becoming proficient at remediating these vulnerabilities
- 22. #RSAC Closing Thoughts, Credits and Demerits Enterprise-scale DevSecOps requires going beyond selecting or creating a toolset – Culture change for both Development Teams and Security SMEs – Requires process changes in Development Teams and movement away from a model of “security as punishment/shaming” Credits – OWASP organization and community esp. open source tools – Vendors supporting open source tools (SonarQube, Anchore - Grype/Syft) Demerits – Vendors with unrealistic and unreasonable claims pushing proprietary solutions 22
Editor's Notes
- #2: Welcom to this session! I am xyz and with me is Gaurav Bhargava, we both work for ADP where we have been building a DevSecOps infrastructure across the company. Today we will share some of our experiences and insights from this journey. We look forward to your questions - and for those who want to dig deeper - we are leading a BOF session on Wednesday afternoon.
- #5: During the last five years, we have seen an increasing focus on integrating security with agile development and deployment So by DevSecOps we mean the processes and tools needed to achieve this integration In this diagram, we show categories of security processes integrated with each stage of the DevOps lifecycle -- call out each stage
- #6: So lets dig deeper to understand the challenges of enterprise-scale devsecops
- #7: Now if you have a couple of dozen or even a couple of hundred developers creating and maintaining a handful of products on a single technology platform - yes, you can pull together a DevSecOps toolset and program relatively easily. Our focus is on large organizations with dozens of development groups creating 10s or 100s of products using varying technology stacks And these products are at different stages of maturity - some are established ”legacy” products - others are newer apps - for example they may be completely cloud native Finally, team cultures and development practices in teams can be quite different - some teams address security vulnerabilities within each sprint - others accumulate security debt and use a security sprint to address security vulnerabilities.
- #8: With teams owning and implementing independent devsecops programs, its difficult to have a uniform view of the security state of applications … Read slide
- #9: Application security requires a range of scanners and information sources for comprehensive coverage Need to identify and make available scanners in each different category Inline or lightweight scanners that can provide feedback within minutes Offline scanners that may take hours to find a more exhaustive set of vulnerabilities -- final 2 bullets -- There is no magic formula that will help in choosing the right tools - enterprise securlty policies and compliance needs will drive the selection of scanners
- #10: Developer and dev team enablement is key to the DevSecOps program linkage of security vulnerabilities from all the different application artifacts - build processes, docker containers, cloud accounts - to devs and dev teams and products is a challenge that has to be solved Application dev teams and developers aren’t security experts and we shouldn’t expect them to become security SMEs As vulnerabilities are discovered, we need practical guidance for remediation When different teams deal with similar vulnerabilities, we need to support information sharing and experiences across teams We need defined workflows when the CSO office should be involved (consultations, exceptions, risk acceptance)
- #11: So now I am going to turn to Gaurav Bhargava to talk about our solution to these requirements
- #19: The first and most important step is to drive consensus around the value of an enterprise-wide approach - The best way to achieve this is going to vary, it really depends how your organizational structure Our approach was to create a POC and show some workflows that teams found helpful ---- Switch the last two bullets
- #20: The key to success here is to meet dev teams where they are What this means is that you have to understand their development culture and tech stacks - and this is why your solution needs to be able to plug security into their processes with only a small investment in time and effort for them - Bullet 2 How you communicate vulnerabilities has to be low impact and fit with team practices Understanding when/if a vulnerability has been successfully remediated should be straightforward - Bullet 3