Monitoring & Securing Microservices in Kubernetes
1 like603 views
This document outlines five key principles for monitoring and securing microservices in Kubernetes, emphasizing container monitoring over traditional VM-focused approaches. It highlights the importance of alerting on service performance, monitoring APIs, securing the container runtime, and conducting incident and forensic analysis. The content also differentiates between containers and VMs, explaining the implications for application management in a microservices architecture.
Monitoring & Securing Microservices in Kubernetes
- 1. Monitoring & Securing Microservices in Kubernetes Michael Ducy - @mfdii - Sysdig
- 2. 5 Principles 1. Monitor containers and what’s inside them. 2. Alert on service performance, not container performance. 3. Monitor APIs. 4. Secure the container runtime environment. 5. Provide incident and forensic analysis for containers.
- 3. Let’s talk about Microservices
- 4. A quick refresher • Let’s define microservices: The term "Microservice Architecture" has sprung up over the last few years to describe a particular way of designing software applications as suites of independently deployable services. While there is no precise definition of this architectural style, there are certain common characteristics around organization around business capability, automated deployment, intelligence in the endpoints, and decentralized control of languages and data. http://www.martinfowler.com/articles/microservices.html
- 5. Stop, It’s not SOA • Services tend to have smaller concerns than SOA Services • Architectural Concepts incorporates Innovations: – In Infrastructure – In Automation – In Continuous Delivery – In Development – In Monitoring http://www.martinfowler.com/articles/microservices.html
- 6. What you think it looks like http://microservices.io/i/Microservice_Architecture.png
- 7. What it really looks like. https://www.appcentrica.com/wp-content/uploads/2016/11/Microservices-Architecture-1.png
- 8. But it’s probably more like https://i.stack.imgur.com/y27IV.png
- 9. Let’s talk about Containers
- 10. What’s a Container? Containers ARE NOT lightweight VMs.
- 11. What’s a Container? A combination of: ● An image ● Cgroups ● Namespaces ● Linux Security Modules: ○ SELinux, App armor, seccomp
- 12. Containers vs. Zones vs. Jails vs. VMs Thanks to Jessie Frazelle (@jessfraz) for this slide https://goo.gl/7fVKFa Containers Cgroups Namespaces LSMs Zones First class concept Jails First class concept VMs First class concept Read more about this here: https://blog.jessfraz.com/post/containers-zones-jails-vms/
- 13. What’s a Container The Old Way The New Way Heavy, Non Portable Light, Easily Portable
- 14. What’s a Container The Old Way The New Way Heavy, Non Portable Light, Easily Portable Application Artifact Virtual Machine Artifact Container Artifact
- 15. cgroups Control Groups Limits the quantity of a resource that can be consumed. Memory, CPU, Net, etc
- 16. namespaces namespaces Limits what resources can be accessed Cgroup CLONE_NEWCGROUP Cgroup root directory IPC CLONE_NEWIPC System V IPC, POSIX message queues Network CLONE_NEWNET Network devices, stacks, ports, etc. Mount CLONE_NEWNS Mount points PID CLONE_NEWPID Process IDs User CLONE_NEWUSER User and group IDs UTS CLONE_NEWUTS Hostname and NIS domain name
- 17. Linux Security Modules SELinux System wide execution policy AppArmor System wide execution policy, focused on processes Seccomp Per process system call isolation
- 18. Why is it magic? Container Image Container Runtime cgroup namespace LSM
- 20. Containers + Microservices Containers Limited scope Easily deployable Immutable Versionable Microservices Limited Scope CD used for deployment Service Guarantees Versionable API
- 25. 5 Principles 1. Monitor containers and what’s inside them. 2. Alert on service performance, not container performance. 3. Monitor APIs. 4. Secure the container runtime environment. 5. Provide incident and forensic analysis for containers.
- 26. 5 Principles 1. Monitor containers and what’s inside them. 2. Alert on service performance, not container performance. 3. Monitor APIs. 4. Secure the container runtime environment. 5. Provide incident and forensic analysis for containers.
- 27. A Paradigm Shift Traditional management focuses on apps running on VMs and Hosts. Containers shift the management focus to apps running as isolated process. vs.
- 28. Monitor Containers & What’s Inside Monitoring needs to focus on: ● Isolated processes in the context of the container ○ Resource limits ● Isolated processes in the context of the orchestrator ○ Resource limits ○ Scaling limits
- 29. Monitor Containers & What’s Inside Monitoring needs to focus on: ● Monitoring the application itself. ● Discovery of metrics for your language runtime, application services, database, etc ● Discovery of metrics exposed by developers ○ Prometheus, JMX, etc
- 30. 5 Principles 1. Monitor containers and what’s inside them. 2. Alert on service performance, not container performance. 3. Monitor APIs. 4. Secure the container runtime environment. 5. Provide incident and forensic analysis for containers.
- 31. Average VM life: 23 Days Average container life: 2.5 Days Alert on service performance, not container performance.
- 32. Alert on service performance, not container performance. ● Containers don’t last long. ● Orchestration systems should monitor individual containers and take action. ○ Resource limits are important! ● Operators and Developers should worry about: “Is the thing doing the thing it’s supposed to be doing, and in a timely manner”
- 33. 5 Principles 1. Monitor containers and what’s inside them. 2. Alert on service performance, not container performance. 3. Monitor APIs. 4. Secure the container runtime environment. 5. Provide incident and forensic analysis for containers.
- 35. Monitor your APIs ● APIs are the primary communication path for your Microservices ● Monitoring API health is important to: ○ Catch bottlenecks in certain methods/functions/endpoints ○ See frequently used methods/functions/endpoints ○ Trace behavior of your application through multiple systems
- 36. 5 Principles 1. Monitor containers and what’s inside them. 2. Alert on service performance, not container performance. 3. Monitor APIs. 4. Secure the container runtime environment. 5. Provide incident and forensic analysis for containers.
- 37. Secure the Runtime Environment Current container security paradigms: ● Image Based ○ Image Scanning ○ Trusted Images ● Orchestrator Security ○ RBAC ● Service Based ○ SPIFFE, Cilium, Service Mesh ● Runtime Security ○ Sysdig Secure, Sysdig Falco
- 38. Image Based Security Image Scanning: ● Scan image for known vulnerabilities ● Still requires process for bringing in upstream security releases ● “Point in Time Compliance” ● Integrate into container build process ● Clair & Anchore Trusted Images/Registries: ● Cryptographically Sign Images ○ Still “Point in Time” ● Only trust particular container registries. ○ Only as good as your container build process
- 39. Orchestrator Security Kubernetes RBAC: ● Stable as of Kubernetes 1.8 ● Rich ability to limit users access to Kubernetes resources Securing APIs, Dashboards, etc: ● Common exploitation point ● Open API ports to internet ● Exposed dashboards
- 40. Service Based Service Mesh ● Ingress, Egress, Service Discovery, Failure Handling, Service Identity Admittance Control ● Service Mesh, SPIFFE Kernel Level Access Control ● API Level Access Control ● Cilium
- 41. Runtime Security ● What happens when a container starts/runs? ● Container Images are Immutable ● Runtime environments often aren’t
- 42. Runtime Security ● Watches Containers for “Abnormal Behavior” ○ Rules defined by policy ■ CIS, PCI, Organizational Best Practice, etc ○ Rules defined by container baselining ● Action taken based on severity ○ Stop/Kill container ● Should be part of your CD pipeline ○ Ensure developers haven’t added anything new that voilates runtime policy.
- 43. 5 Principles 1. Monitor containers and what’s inside them. 2. Alert on service performance, not container performance. 3. Monitor APIs. 4. Secure the container runtime environment. 5. Provide incident and forensic analysis for containers.
- 44. Incident & Forensics Analysis Incident & Forensics traditionally difficult: ● Bare Metal - Save the Drives ● VM - Snapshot VM state, Keep VM Containers make Incident & Forensics easier: ● Smaller footprint ● Limited objects in the container Orchestrators make Incident & Forensics difficult: ● Ephemeral nature of containers ● Rapid deploy cycles
- 45. Incident & Forensics Analysis Container Incident & Forensics Analysis requires: ● Capturing the container before it’s gone ● Capturing the system activity for analysis ● Correlation between abnormal behavior and system activity
- 46. To recap. Containers change how we monitor applications. Microservices change how we monitor services. 1. Monitor containers and what’s inside them. 2. Alert on service performance, not container performance. 3. Monitor APIs. 4. Secure the container runtime environment. 5. Provide incident and forensic analysis for containers.
- 47. Thank You! Twitter: - @sysdig - @mfdii Sysdig Blog: - https://sysdig.com/blog/ Sysdig Container Intelligence Platform: - https://sysdig.com/ Sysdig Trial: - https://sysdig.com/sign-up
- 48. Thank You