Flight East 2018 Presentation–A DevOps State of Mind: Continuous Security with Kubernetes
0 likes909 views
This document discusses continuous security with Kubernetes. It introduces the concept of DevSecOps which integrates security practices into DevOps workflows. It discusses how to secure container images, container builds, container registries, container hosts, and networking. It also covers continuous integration, delivery, monitoring and logging in container environments. The goal is to enable innovation speed while maintaining security and efficiency at scale.
Flight East 2018 Presentation–A DevOps State of Mind: Continuous Security with Kubernetes
- 1. A DevOps State of Mind: Continuous Security with Kubernetes Chris Van Tuin Red Hat Chief Technologist, NA West / Silicon Valley cvantuin@redhat.com
- 2. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co “Only the paranoid survive” - Andy Grove, 1996
- 3. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Static & Planned Dynamic & Policy Driven ExecutionInnovation Old New ExecutionInnovation THE CHALLENGE: ENABLE INNOVATION AT SPEED, WHILE EXECUTING AT SCALE WITH EFFICIENCY
- 4. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co SECURITY MUST EVOLVE & KEEP UP
- 5. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Applications & devices outside of IT control Cloud computing Software-defined infrastructure Dissolving security perimeter Menacing threat landscape TRADITIONAL NETWORK-BASED DEFENSES ARE NO LONGER ENOUGH SECURING THE ENTERPRISE IS HARDER THAN EVER The way we develop, deploy and manage IT is changing dramatically led by DevOps, Cloud Native Applications, and Hybrid Cloud
- 6. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co DEVSECOPS + + Security DEV QA OPS Culture Process Technology Linux + Containers IaaS Orchestration CI/CD Source Control Management Collaboration Build and Artifact Management Testing Frameworks Cloud Native Applications Hybrid Cloud OpenSource
- 7. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co DEVSECOPS Continuous Security Improvement Process Optimization Security Automation Dev QA Prod Reduce Risks, Lower Costs, Speed Delivery, Speed Reaction
- 8. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co LAPTOP Container Application OS dependencies Guest VM LINUX BARE METAL Container Application OS dependencies LINUX VIRTUALIZATION Container Application OS dependencies Virtual Machine LINUX PRIVATE CLOUD Container Application OS dependencies Virtual Machine LINUX PUBLIC CLOUD Container Application OS dependencies Virtual Machine LINUX CONTAINERS - Build Once, Deploy Anywhere Reducing Risk and Improving Security with Improved Consistency
- 9. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD Security Platform
- 10. AUTOMATION
- 11. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Web Database role=web role=db role=web replicas=1, role=db replicas=2, role=web ORCHESTRATION Deployment, Declarative Pods Nodes Services Controller Manager & Data Store (etcd)
- 12. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Web Database replicas=1, role=db replicas=2, role=web HEALTH CHECK Pods Nodes Services role=web role=db role=web Controller Manager & Data Store (etcd)
- 13. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Pods Nodes Services Web Database replicas=1, role=dbreplicas=3 role=web AUTO-SCALE 50% CPU role=web role=db role=web role=web Controller Manager & Data Store (etcd)
- 14. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Network isolation API & Platform access Federated clusters Storage {} CI/CD Monitoring & Logging BuildsImages SECURING YOUR CONTAINER ENVIRONMENT Container hostRegistry
- 15. CONTAINER IMAGES
- 16. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co docker.io RegistryPrivate Registry FROM fedora:1.0 CMD echo “Hello” Build file Physical, Virtual, Cloud Container Image Container Instance Build RunShip CONTAINER WORKFLOW
- 17. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co JAR CONTAINER IMAGE Application Application Language runtimes OS dependencies 1.2/latest 1.1 CONTAINER IMAGE
- 18. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Config Data Kubernetes configmaps secrets Container image Traditional data services, Kubernetes persistent volumes TREAT CONTAINERS AS IMMUTABLE Application Language runtimes OS dependencies
- 19. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co •Authenticating authorship •Non-repudiation •Ensuring image integrity CONTAINER IMAGE SIGNING Validate what images and version are running
- 20. CONTAINER BUILDS
- 21. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co A CONVERGED SOFTWARE SUPPLY CHAIN
- 22. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co • Treat build file as a Blueprint • Version control build file • Don’t login to build/configure • Be explicit with versions, not latest • Always list registry pulling FROM • Specify USER, default is root • Each Run creates a new layer BUILD FILE BEST PRACTICES FROM registry.redhat.com/rhel7 RUN groupadd -g 999 appuser && useradd -r -u 999 -g appuser appuser USER appuser CMD echo “Hello” Build file
- 24. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co 64% of official images in Docker Hub contain high priority security vulnerabilities examples: ShellShock (bash) Heartbleed (OpenSSL) Poodle (OpenSSL) Source: Over 30% of Official Images in Docker Hub Contain High Priority Security Vulnerabilities, Jayanth Gummaraju, Tarun Desikan, and Yoshio Turner, BanyanOps, May 2015 (http://www.banyanops.com/pdf/BanyanOps-AnalyzingDockerHub-WhitePaper.pdf) WHAT’S INSIDE THE CONTAINER MATTERS
- 25. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co PRIVATE REGISTRY
- 27. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Best Practices • Don’t run as root • If you must, limit Linux Capabilities • Limit SSH Access • Use namespaces • Define resource quotas • Enable logging • Apply Security Errata • Apply Security Context and seccomp filters • Run production unprivileged containers as read-only http://blog.kubernetes.io/2016/08/security-best-practices-kubernetes-deployment.html Kernel Hardware (Intel, AMD) or Virtual Machine Containers ContainersContainers Unit File Docker Image Container CLI SYSTEMD Cgroups Namespaces SELinux Drivers seccomp Read Only mounts Capabilities CONTAINER HOST SECURITY
- 29. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co SECURITY IMPLICATIONS What’s inside matters…
- 30. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Security CONTINUOUS INTEGRATION WITH SECURITY SCAN
- 31. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co AUTOMATED SECURITY SCANNING with OpenSCAP ReportsScan SCAP Security Guide for RHEL CCE-27002-5 Set Password Minimum Length Content Scan physical servers, virtual machines, docker images and containers for Security Policy Compliance (CCEs) and known Security Vulnerabilities (CVEs)
- 32. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co CONTINUOUS INTEGRATION + BUILDS
- 34. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co CONTINUOUS DELIVERY WITH CONTAINERS
- 35. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co CONTINUOUS DELIVERY DEPLOYMENT STRATEGIES DEPLOYMENT STRATEGIES • Recreate • Rolling updates • Blue / Green deployment • Canary with A/B testing
- 36. RECREATE
- 37. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Version 1 Version 1Version 1 Version 1.2 ` Tests / CI RECREATE WITH DOWNTIME
- 38. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Version 1 Version 1Version 1 Version 1.2 ` Tests / CI RECREATE WITH DOWNTIME
- 39. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Version 1.2 Version 1.2Version 1.2 RECREATE WITH DOWNTIME Use Case • Non-mission critical services Cons • Downtime Pros • Simple, clean • No Schema incompatibilities • No API versioning
- 40. ROLLING UPDATES
- 41. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Version 1 Version 1Version 1 Version 1.2 ` Tests / CI ROLLING UPDATES with ZERO DOWNTIME
- 42. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Deploy new version and wait until it’s ready… Version 1 Version 1 V1.2 Health Check: readiness probe e.g. tcp, http, script V1
- 43. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Each container/pod is updated one by one Version 1.2 50% Version 1 V1 V1.2
- 44. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Each container/pod is updated one by one Version 1.2Version 1.2Version 1.2 100% Use Case • Horizontally scaled • Backward compatible API/data • Microservices Cons • Require backward compatible APIs/data • Resource overhead Pros • Zero downtime • Reduced risk, gradual rollout w/health checks • Ready for rollback
- 45. BLUE / GREEN DEPLOYMENT
- 46. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Version 1 BLUE / GREEN DEPLOYMENT Route BLUE
- 47. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Version 1 BLUE / GREEN DEPLOYMENT Version 1.2 BLUE GREEN
- 48. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Version 1 Tests / CI BLUE / GREEN DEPLOYMENT Version 1.2 BLUE GREEN
- 49. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Version 1 Version 1.2 BLUE / GREEN DEPLOYMENT Route Version 1.2 BLUE GREEN
- 50. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Version 1 BLUE / GREEN DEPLOYMENT Rollback Route Version 1.2 BLUE GREEN Use Case • Self-contained micro services (data) Cons • Resource overhead • Data synchronization Pros • Low risk, never change production • No downtime • Production like testing • Rollback
- 52. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co ”only about 1/3 of ideas improve the metrics they were designed to improve.” Ronny Kohavi, Microsoft (Amazon) MICROSERVICES RAPID INNNOVATION & EXPERIMENTATION
- 53. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co CONTINUOUS FEEDBACK LOOP
- 54. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co A/B TESTING USING CANARY DEPLOYMENTS
- 55. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Version 1.2Version 1 100% Tests / CI Version 1.2 Route 25% Conversion Rate ?! Conversion Rate CANARY DEPLOYMENTS
- 56. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co 50% 50% Version 1.2Version 1 Route Version 1.2 25% Conversion Rate 30% Conversion Rate CANARY DEPLOYMENTS
- 57. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co 25% Conversion Rate 100% Version 1 Version 1.2 Route Version 1.2 30% Conversion Rate CANARY DEPLOYMENTS
- 58. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Version 1.2Version 1 100% Route Rollback 25% Conversion Rate 20% Conversion Rate CANARY DEPLOYMENTS
- 59. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Network isolation API & Platform access Federated clusters Storage {} CI/CD Monitoring & Logging ImagesBuilds Container hostRegistry SECURING YOUR CONTAINER ENVIRONMENT
- 60. NETWORK SECURITY
- 61. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Network Namespace provides resource isolation NETWORK ISOLATION Multi-Environment Multi-Tenant
- 62. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co NETWORK POLICY example: all pods in namespace ‘project-a’ allow traffic from any other pods in the same namespace.”
- 63. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Kubernetes Logical Network Model NETWORK SECURITY • Kubernetes uses a flat SDN model • All pods get IP from same CIDR • And live on same logical network • Assumes all nodes communicate Traditional Physical Network Model • Each layer represents a Zone with increased trust - DMZ > App > DB, interzone flow generally one direction • Intrazone traffic generally unrestricted
- 64. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co NETWORK SECURITY MODELS Co-Existence Approaches One Cluster Multiple Zones Kubernete Cluster Physical Compute isolation based on Network Zones Kubernete Cluster One Cluster Per Zone Kubernete Cluster B Kubernete Cluster A Kubernetes Cluster B C D https://blog.openshift.com/openshift-and-network-security-zones-coexistence-approaches/
- 66. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co KUBERNETES MONITORING CONSIDERATIONS Kubernetes* Container* Host Cluster services, services, pods, deployments metrics Container native metrics Traditional resource metrics - cpu, memory, network, storage prometheus + grafana kubernetes-state-metrics probes Stack Metrics Tool node-exporter kuberlet:cAdvisor Application Distributed applications - traditional app metrics - service discovery - distributed tracing prometheus + grafana jaeger tracing istio
- 67. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Aggregate platform and application log access via Kibana + Elasticsearch LOGGING
- 68. STORAGE SECURITY
- 69. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Local Storage Quota Security Context Constraints STORAGE SECURITY Sometimes we can also have storage isolation requirements: pods in a network zone must use different storage endpoints than pods in other network zones. We can create one storage class per storage endpoint and then control which storage class(es) a project can use
- 70. API & PLATFORM ACCESS
- 71. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Authentication via OAuth tokens and SSL certificate Authorization via Policy Engine checks User/Group Defined Roles API & PLATFORM ACCESS
- 72. FEDERATION
- 73. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Amazon East OpenStack FEDERATED CLUSTERS Roles & access management (in-dev)
- 74. WHAT’S NEXT
- 75. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Monitoring & Metrics -prometheus (logs) -grafana (visual) Access Control & usage policies -mixr (policy decisions) Encryption & Auth -citadel -service 2 service -user auth Traffic routing - pilot - circuit breaker - a/b testing - traffic mirroring Fault injections -envoy corner cases: abort & delays SERVICE MESH
- 76. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co OPERATORS
- 77. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Deployment Frequency Lead Time Deployment Failure Rate Mean Time to Recover 99.999 Service Availability DEVSECOPS METRICS Compliance Score
- 78. THANK YOU linkedin: Chris Van Tuin email: cvantuin@redhat.com twitter: @chrisvantuin