Securing the Container Pipeline at Salesforce by Cem Gurkok
4 likes5,556 views
The document discusses securing container pipelines, outlining key threats, mitigation strategies, and the importance of monitoring throughout the pipeline process. It emphasizes hardening both host and container environments, implementing strict access controls, and employing vulnerability management techniques. Additionally, it covers digital forensics and network infrastructure monitoring as essential components of maintaining container security.
![Memory Forensics: Process Hierarchy
• pstree_hash [new]: View Docker
processes in a tree view based on
the PID hash table vs. linked list
• Use case: Detect rogue or injected
child processes/containers](https://image.slidesharecdn.com/dockercon-2016-cg-securingthecontainerpipelineatsalesforce-sf-6-23-2016-public-160627171137/85/Securing-the-Container-Pipeline-at-Salesforce-by-Cem-Gurkok-34-320.jpg)
![Memory Forensics: Process Integrity
• process_compare [new]: Detect if user space binary has
been tampered with in memory (in memory binary vs. on
disk) [5]
• Works when binary symbols can’t be extracted](https://image.slidesharecdn.com/dockercon-2016-cg-securingthecontainerpipelineatsalesforce-sf-6-23-2016-public-160627171137/85/Securing-the-Container-Pipeline-at-Salesforce-by-Cem-Gurkok-37-320.jpg)
Securing the Container Pipeline at Salesforce by Cem Gurkok
- 1. Securing the Container Pipeline Cem Gürkök Lead InfoSec Engineer (NOTE: PASTE IN PORTRAIT AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP)
- 2. Securing the Container Pipeline Cem Gürkök Lead InfoSec Engineer Salesforce @CGurkok
- 3. Agenda • Threats • Container pipelines and integrity • Monitoring containers, hosts, apps, networks • Digital Forensics • Vulnerability Management • Hardening • Demo
- 4. Threats
- 5. Container Threats & Challenges Run-time • Container exploit and resource exposure (App) • Breaking out of container • Cross-container attacks • Resource overuse (DoS) At-rest or transport • Tampering of images • Unpatched OS or applications
- 6. Mitigations
- 7. “As we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns—the ones we don’t know we don’t know.” — Donald Rumsfeld
- 9. The Pipeline
- 10. Container Pipeline & Security Base OS and Docker File Base OS Image DEV Docker Trusted Registry + Notary Developer RelEng Image DEV Docker Trusted Registry + Notary Continuous Integration PROD Docker Trusted Registry + Notary Running in PROD Monitoring in all steps. 1. Security Review and Hardening 2. Signing, Authentication, Image Vulnerability Scans 3. Authentication, Verification 4. Signing, Authentication, Image Vulnerability Scans 5. Authentication 6. Authentication, Verification 7. Authentication, Verification, Vulnerability Scans 8. Incident Response, Digital Forensics, Patching
- 11. Access Control: Authentication • LDAP over SSL for Docker image transactions: • Users (Devs, RelEng) • Service accounts • Mutual TLS Authentication for registry replication Dev Systems Dev Registry Build & Test Master Registry Prod Registry DMZ Services TLS
- 12. Container Integrity Docker Trusted Registry (DTR) • On-premise • Authenticated transactions with LDAPS authentication • DEV and PROD user and image separation • Users will not be able to disable signing validation • Validation will be transparent to the users
- 13. Container Integrity Docker Notary • Enable Docker Content Trust on consumers • Can enable signing checks on every managed host • Signature verification transparent to users Build & Test Notary Master DMZ Services Prod Services Sign Validate Validate
- 14. Master Docker Registry DEV DMZ Notary Master LDAPS Auth Notary Signing Dev Docker Registry PROD Mirrored Read-only Registry or Caching Proxy Docker packaged services Mutual TLS Auth HTTPS Pull Validate Dev Systems LDAPS user acct HTTPS Push to Dev Authenticated pulls LDAPS Auth HTTPS Push and Sign HTTPS Pull Sign LDAPS Auth HTTPS Push Already Signed Docker packaged services in DMZ HTTPS No Auth Pull Validate RelEng promotes to DMZ Release case * Andrey Falko, Salesforce Ticketing System
- 15. Hardening
- 16. Hardening: Host • Frequent patching • Install only needed components and libraries (i.e. no gcc or bash) • Grsecurity/PaX for the kernel • File system integrity monitoring • Leverage Linux isolation capabilities!!
- 17. Hardening: Container • Base image and app with latest updates/patches • Leverage User namespaces (run as low priv user on host) • Install only needed components and libraries (i.e. no gcc or ssh)
- 18. Hardening: Container • Avoid using Docker with the --privileged flag • Use --read-only when running containers (immutability) • Avoid providing access to the docker user and group • Limit and/or separate host and kernel device access
- 19. Hardening: Docker Bench for Security • Docker Bench for Security to the rescue! • https://github.com/docker/ docker-bench-security • Checks based on best practices for hosts and containers * https://github.com/docker/docker-bench-security
- 20. Hardening: Vulnerability Management Image Scans with tools, such as Docker Security Scanning: • Operating System • Application source code and libraries Network Scans with traditional vuln scanners: • Discovery • Exposed services Auto and Manual source code audits * “Securing the Software Supply Chain with Docker, ” May 2016, Nathan McCauley
- 21. Hardening: Vulnerability Management • Scanning • Docker Images • Applications • Remediation • Prioritization and SLAs for Patching • Relaunching containers after patching Δt
- 22. Monitoring
- 23. Network Infrastructure • Bridged networking on Host • Containers assigned VNICs, IP addresses, and hostnames • Containers isolated via VLANs (i.e. DB, Web App) • Tap interface for monitoring • Security Policies per VLANs and Zones
- 25. Monitoring: Network Network traffic captured for: • Inter-container communications • Host communications • Resource communications (i.e. DB, Public Internet) Network traffic sent to: • IDS (Intrusion Detection System) • Netflow generator • Output sent to SIEM for analysis
- 26. Monitoring: Hosts Logs: • All host logs are saved • SIEM agents consume and forward the logs from hosts • Monitoring, Dashboarding, Alerting at SIEM Host SIEM
- 27. Monitoring: Containers & Apps • Logs are monitored similar to host • OS + Application logs • Network activity monitoring • IP address assignments • Netflows • IDS (Intrusion Detection System) • Raw Network Traffic Capture
- 28. Monitoring: Host, Containers & Apps Disk activity monitoring • File system integrity • Run time layer monitoring Memory monitoring • Docker and container process activity • Process integrity: Engine + Container
- 30. Digital Forensics • Incident Response Plan/Policies • Live/Post-mortem Memory Forensics • Disk Forensics • Network Monitoring/Forensics
- 31. Disk Forensics • Build supertimeline to have integrated view of events • Data Sources: • Raw Disk Image • Log Files • Binaries • Tools • The Sleuth Kit: File system analysis • Plaso: Build supertimeline • dd: Raw disk image dd Sleuth Kit Plaso
- 32. Memory Forensics Why Memory Forensics? • Nothing can hide in memory! • Faster artifact discovery vs. disk forensics
- 33. Memory Forensics Analyze host memory • Live /dev/*mem • VM memory file • Memory dump/sample Tools: • Analysis (most OS and sample format): • The Volatility Framework • Memory sampling on Linux: LiME, linpmem LiME linpmem
- 34. Memory Forensics: Process Hierarchy • pstree_hash [new]: View Docker processes in a tree view based on the PID hash table vs. linked list • Use case: Detect rogue or injected child processes/containers
- 35. Memory Forensics: Temporary File Systems • tmpfs: lists and recovers tmpfs file systems from memory • Use case: monitor file systems
- 36. Memory Forensics: Loaded Libraries • linux_proc_maps: shows process memory maps, their permissions and original file paths (executable and libraries) • Use case: Detect Shared Library Injections
- 37. Memory Forensics: Process Integrity • process_compare [new]: Detect if user space binary has been tampered with in memory (in memory binary vs. on disk) [5] • Works when binary symbols can’t be extracted
- 38. Summary Platform Security Isolation Hardening Best Practices Vulnerability Scans Content Security Registry Notary Image/Code Signing Image/Code Scanning Access Controls LDAPS User Authentication System Authentication Monitoring and Response IR Plan & Testing Vulnerability Management Network Logs Forensics
- 39. thank y u
- 40. References 1. “CIS Docker 1.6 Benchmark,” Center for Internet Security 2. “Introduction to Container Security,” Docker.com 3. “Understanding and Hardening Linux Containers,” NCC Group 4. “The Volatility Framework,” https://github.com/volatilityfoundation/volatility 5. “Identifying the Unknown in User Space Memory,” Andrew White 6. “LiME,” https://github.com/504ensicsLabs/LiME 7. “linpmem,” http://www.rekall-forensic.com/docs/Tools/ 8. “The Sleuth Kit,” http://www.sleuthkit.org/ 9. “Plaso,” https://github.com/log2timeline/plaso