Batten Down the Hatches: A Practical Guide to Securing Kubernetes - RMISC 2019
1 like533 views
James Condon presented a guide to securing Kubernetes. He began with an overview of Kubernetes architecture and then discussed major risk vectors like exposed Kubernetes components and pod compromise. He demonstrated finding exposed Kubernetes dashboards, API servers, kubelets, and etcd clusters. Condon recommended 10 essential practices for securing Kubernetes, including network security, role-based access control, security boundaries, upgrading, and audit logging. He concluded with resources for further information.
Batten Down the Hatches: A Practical Guide to Securing Kubernetes - RMISC 2019
- 1. @laceworklabs Batten Down the Hatches: A Practical Guide to Securing Kubernetes James Condon CSA June 18th, 2019
- 2. @laceworklabs whoami • James Condon, Director of Research @ Lacework • Former USAF OSI, Mandiant, and ProtectWise • Network Forensics, Incident Response, Threat Intelligence, Cloud Security Twitter: @laceworklabs, @jameswcondon Email: james@lacework.com Blog: www.lacework.com/blog/
- 3. @laceworklabs AGENDA Kubernetes Overview Risks and Threats Securing Kubernetes
- 7. @laceworklabs Master API Server etcd Scheduler Control Manger Proxy Node Proxy Kubelet Container Runtime Pod 1...n Node Proxy Kubelet Container Runtime Pod 1...n UI Dashboard CLI BASIC ARCHITECTURE
- 8. @laceworklabs RISKS & THREATS TO KUBERNETES
- 9. @laceworklabs MAJOR THRE AT VECTORS Exposed K8s Components Pod Compromise • UI Dashboard • API Service • etcd • Kubelet • Application Attacks • Supply Chain Attacks • Orchestrator and container CVEs
- 10. @laceworklabs Node Proxy Kubelet Container Runtime Pod 1...n Node Proxy Kubelet Container Runtime Pod 1...n UI Dashboard CLI EXPOSED DASHBOARDS Master API Server etcd Scheduler Control Manger Proxy
- 11. @laceworklabs EXPOSED DASHBOARDS • Web-based cluster management UI • Risks & Threats: • Default service accounts needs RBAC • Can expose CSP keys • Cryptojacking attacks • Information & data leaks
- 13. @laceworklabs DASHBOARD FINDINGS 500+ 75% AWS 10% GCP + Azure Ports 80, 443, 8080
- 14. @laceworklabs Node Proxy Kubelet Container Runtime Pod 1...n Node Proxy Kubelet Container Runtime Pod 1...n UI Dashboard CLI KUBE-APISERVER Master API Server etcd Scheduler Control Manger Proxy
- 15. @laceworklabs EXPOSED API SERVER • Handles all client interactions to the K8s API • REST API • Handles authentication and authorization • Secure & insecure port by default • Risks & Threats • Access to insecure port allows complete access of cluster • CVE-2018-1002105 • Information leaks
- 17. @laceworklabs API SERVER FINDINGS 21K+ 92% AWS, ~3% GCP + Azure ’18: 21K+ for K8s, Meso, OpenShift, & Swarm Cert CNs: kubernetes-master 88%, system:apiserver 4%, apiserver 2%
- 18. @laceworklabs API SERVER FINDINGS (INSECURE PORT) 800+
- 19. @laceworklabs Node Proxy Kubelet Container Runtime Pod 1...n Node Proxy Kubelet Container Runtime Pod 1...n UI Dashboard CLI KUBELET Master API Server etcd Scheduler Control Manger Proxy
- 20. @laceworklabs EXPOSED KUBELET • Daemon on nodes to bridge compute resources, facilitate communicates, and aide in pod health • Risks & Threats • Allows anonymous requests by default • “AlwaysAllow” is the default for authenticated requests rest by default • Contains credentials that can be used to access other components in the cluster
- 21. @laceworklabs EXEC ON RUNNING CONTAINER THROUGH KUBELET • PoC by Security Engineer @ Handy (K8 v1.9) • Issue POST request to targeted Pod • Follow with GET request via SPDY or websocket client
- 22. @laceworklabs REPLAYING KUBELET CREDENTIALS • SSRF in vulnerable service used by Shopify • Kubelet credentials leaks via vulnerability • Credentials replayed to gain root access in any container
- 23. @laceworklabs Node Proxy Kubelet Container Runtime Pod 1...n Node Proxy Kubelet Container Runtime Pod 1...n UI Dashboard CLI etcd Master API Server etcd Scheduler Control Manger Proxy
- 24. @laceworklabs ETCD • Distributed key value datastore • REST & gRPC APIs • Responsible for storing objects, state, etc. • Risks & Threats • No authentication or encryption at rest by default • Maintains cluster secrets • The Luke Hemsworth of unsecured DBs
- 27. @laceworklabs POD COMPROMISE & LATERAL MOVEMENT Pod Compromise Application Vulnerabilities Supply Chain Attacks Known & Unknown CVEs
- 28. @laceworklabs Node Proxy Kubelet Container Runtime Pod 1...n Node Proxy Kubelet Container Runtime Pod 1...n UI Dashboard CLI PODS Master API Server etcd Scheduler Control Manger Proxy
- 31. @laceworklabs • allows containers using subPath volume mounts to access files or directories outside of the volume, including the host’s filesystemCVE-2017-1002101 • Flaw in runc, allows potential container escapeCVE-2019-5736 • Options for accessing host systemPrivileged Containers • Default service accounts are overprivileged and have too much access that an attacker could leverageService Accounts • Authenticated users with permission to exec/attach/portforward could escalated to run additional commands against Kubelet APICVE-2018-1002105 LATERAL MOVEMENT
- 33. @laceworklabs 10 ESSENTIALS SECURING K8S Upgrade Network Security POD Security PoliciesNode Security Hardening Audit Logging Security Boundaries RT Compliance / Auditing Image Security RBAC Host Logging / HIDS
- 34. @laceworklabs NETWORK SECURITY Restrict Open Internet Access TLS, VPN, Bastion Network Policy for pods Host Firewalls
- 35. @laceworklabs IMAGE SECURITY Container vulnerability scans Scan for poor configurations in containers Scan for keys in containers Combine pre-deploy with runtime
- 36. @laceworklabs ROLE BASED ACCESS CONTROL Critical for division on access Segregates roles and permissions Decreases attack surface Reduce default permissions of service accounts
- 37. @laceworklabs SECURITY BOUNDARIES Utilize multiple namespaces Separate sensitive workloads Utilize node pools to separate Ex: kube-public
- 38. @laceworklabs UPGRADE! CVE-2018-1002105 DEMO Upgrading should be seamless No runtime patching Patch = redeploy Vulnerabilities != vulnerable often
- 39. @laceworklabs POD SECURITY POLICIES Huge win in securing K8s Allow centralized cluster level security controls / configuration Controls growing frequently Common Examples: privileged volumes network
- 40. @laceworklabs NODE SECURITY HARDENNING Minimal OS footprint Restricted file system access Upgrades and Patches Disabled root login Kernel Protection Security Defaults
- 41. @laceworklabs AUDIT LOGGING Audit Logging for ALL API requests API is largest attack surface Log as much as you can afford Store, glacier, have them avail/query Audit logs big forensics firehouse
- 42. @laceworklabs RT COMPLIANCE / CONFIG CIS Benchmarks Realtime / runtime auditing critical Infrastructure as code = wider paper cuts Security vulnerabilities often config’s Identify, alert, fix, measure (repeat)
- 43. @laceworklabs HOST LOGGING / HIDS / EDR Ephemeral workloads make logging more important Understand process, applications, network Building net “sensors” hard / blind Correlate IOC’s + events (ML+) Opensource + SaaS options Build / buy centralized warehouse Auditd, /proc, pcap,etc..
- 44. @laceworklabs FINAL THOUGHTS • K8s is complex “5 minutes to deploy, 5 years to learn” • Reported attacks are primarily cryptojacking, pivoting to CSP, and data leak • Misconfiguration and pod compromise are the major vectors • Use traditional security, DevSecOps, and K8s features to harden your cluster
- 45. @laceworklabs resources 1. Tesla Exposed Dashboard https://redlock.io/blog/cryptojacking-tesla 2. Weight Watchers Exposed Dashboard https://kromtech.com/blog/security-center/weightwatchers-exposure-a- simple-yet-powerful-lesson-in-cloud-security 3. Lacework Containers at Risk Report https://info.lacework.com/hubfs/Containers%20At- Risk_%20A%20Review%20of%2021,000%20Cloud%20Environments.pdf 4. CVE-2018-1002105 Github Page https://github.com/kubernetes/kubernetes/issues/71411 5. Kubelet Reference Page https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet- authentication-authorization/ 6. Compromising Kubernetes Through Kubelet Blog https://medium.com/handy-tech/analysis-of-a-kubernetes- hack-backdooring-through-kubelet-823be5c3d67c 7. Shopify Hack https://hackerone.com/reports/341876 8. Exposed etcd Clusters Blog https://elweb.co/the-security-footgun-in-etcd/ 9. Lacework exposed etcd Clusters Blog https://www.lacework.com/etcd-thousands-of-clusters-open/ 10. Backdoored Docker Images https://arstechnica.com/information-technology/2018/06/backdoored-images- downloaded-5-million-times-finally-removed-from-docker-hub/ 11. Twistlock Blog on CVE-2017-1002101https://www.twistlock.com/labs-blog/deep-dive-severe-kubernetes- vulnerability-date-cve-2017-1002101/ 12. Attacking and Defending a Kubernetes Cluster Webinar https://vimeo.com/277901517 13. Kubernetes Illustrated Children's Guide: https://youtu.be/4ht22ReBjno
- 46. @laceworklabs QUESTIONS Twitter: @laceworklabs, @jameswcondon Email: james@lacework.com Blog: www.lacework.com/blog/