Understand, verify, and act on the security of your Kubernetes clusters - Scaleway's expertise
0 likes119 views
The document outlines critical aspects of securing Kubernetes clusters, emphasizing the need for a security-centered approach that encompasses all roles—developers, architects, analysts, and users. It discusses common vulnerabilities, security policies, the importance of data protection, and designing applications with security as a fundamental principle. Finally, it highlights the necessity of managing third-party services and ensuring data compliance with regulations like GDPR.
Understand, verify, and act on the security of your Kubernetes clusters - Scaleway's expertise
- 2. 19/11/2020 - Emmanuelle Demompion Understand, verify, and act on the security of your clusters
- 3. What’s the plan? Introduction Secure your k8s cluster at its root Security as a coding philosophy Managing third parties security 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion 1 2 3 4 5 6 7 Lock your data Security cheat sheet Conclusion 3
- 4. From the Cloud to your code Introduction 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion
- 5. Scaleway Private Infrastructure High-end dedicated Servers Public Cloud Ecosystem 3 specialized brands to cover all cloud and infrastructure needs 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion
- 6. 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion 6 Compute Storage NetworkIoT AI Steering Scaleway ELEMENTS
- 7. Kubernetes Kapsule Slack #k8s - @ben - @jth - @Sh4d1 - @ema Just a high availability, fully managed, up-to-date Kubernetes environment 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion
- 8. TL;DR; of the webinar Security issues Everybody should care about security Very few really do Be aware of the most common issues Solutions Certificates Admission controllers Pod security policies Security by design Responsibilities The boss The analyst The architect The developer The user On a clock? You don’t have time to go through the presentation or you fell asleep during it, go to the end, there is a Kubernetes Cheat Sheet waiting for you… ;) 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion8
- 9. How will this webinar go? On each slide, I will use these images to illustrate who should feel the most concerned about each topic. You are all concerned, you just might not be the one dealing with it directly. First Strike: A creature with first strike will deal its fight damage before a creature it doesn’t. Trample: Static ability. After attacking, the creature with Trample can assign the rest of the damage not taken by the first target to target other creatures. Creature: Boss “No one can defeat the decision maker.” “Scaleway’s mascot rules” - Marc-Etienne Peintre 5 / * Landwalk: Static ability. The affected creature may not be obstructed providing the protecting player settings a land associated with the specified type. Shroud: Static ability. The impacted permanent or player cannot be the goal of every means or abilities, together with your own. Creature: Analyst “Data is everything and more.” “Connections and correlations” - Marc-Etienne Peintre 3 / 2 Lifelink: Fixed capability. Whenever a card with lifelink deals damage, the controller of this card additionally gains life corresponding to the amount of damage dealt. Banding: When several creatures with banding, temporally 'band' together, it can become one creature to either assault or block. Creature: Developer “It’s not a bug, it’s a feature!” “Scaleway’s mascot space cowboy” - Marc-Etienne Peintre 2 / 2 Indestructible: Any permanent which has indestructible can't be damaged, and cannot perish as a consequence of deadly harm. Intimidate: Fixed capability. A creature with intimidate cannot be blocked, except by artefact creatures and also by creatures that share a colour with it. Creature: User “I’m a leaf on the wind, watch how I soar…” “The magnificent” - Marc-Etienne Peintre * / * Creature: Architect “You need strong foundations for a building to last.” “The builder” - Marc-Etienne Peintre 1 / 4 Cover: Static ability. The affected permanent is granted with the inability becoming dealt damage, enchanted, blocked, or targeted by any card aided by the reported high quality. Vigilance: Static ability. The affected creature does not touch as soon as declared as an attacker. 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion9 The takeaways I hope you’ll get from this webinar: You know solutions to address security issues, you feel responsible about security, whatever your job is, and you are willing to accept that cloud-native apps should include security by design.
- 10. Terraform Helm CI/CD K8s Third parties Your Service Code & Services Deployment Cluster Your Data Everything we should take a look at… 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion10
- 11. How bad can it really be? My cluster: 63 issues A bunch of tutorials, an old Github project of mine and on-the-shelf application installed using Helm. My old unmaintained Github project: 1054 vulnerabilities Security issues spread 3 % 57 % 40 % Cluster Images Exposures 13 pods without security policies 1 service running without data encryption 6 namespaces without network policies associated 6 docker images with vulnerable packages 1 image including a private key 3 docker images built with ROOT user 1 service exposing software information 1 service using a self signed certificate 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion ok if trusted, mine was not 11
- 12. Securing the infrastructure of your cluster itself Secure your k8s cluster at its root 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion
- 13. How a Kubernetes cluster looks Creature:DeveloperCreature:ArchitectCreature:Boss 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion Worker nodes Namespaces Pods Control-Plane ETCD 13
- 14. Certificates Managing your Kubernetes cluster yourself Use automatically generated certificates (stored on the API-Server) or generate them yourself. Protect the location of your certificates. Think about a process if you need to revoke your certificates and regenerate new ones. Using a managed Kubernetes engine Certificates are generated and managed by the cloud provider. Depending on the provider, the revoking policy of certificates can be different and should be part of your choice of provider. Creature:ArchitectCreature:Boss 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion14
- 15. Admission controllers 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion Control every request reaching the API-Server1 2 3 Creature:DeveloperCreature:Architect Limit creation, deletion, modification, or connection rights Define the behaviour of every Kubernetes object running on your cluster Set security strategies on pods or containers 4 15
- 16. Admission controllers LimitRanger • Any incoming request should not violate the Namespace limit ranges constraints. • Limit Ranges objects enforce min and max resources, storage, requests, for k8s objects in a Namespace (Pod, Container, PersistentVolumeClaim…) ServiceAccount • Automation implementation for ServiceAccounts. • Mandatory if you are using ServiceAccount objects in your cluster (to manage different users accesses to your cluster) NamespaceLifeCycle • New objects cannot be created in a terminating namespace. • Requests on a non-existing namespace are rejected. • Prevents deletion of system reserved namespaces (default, kube-system, kube- public). 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion DefaultStorageClass • Observes the creation of PersistentVolumeClaim objects which does not explicitly provide the storage class to use. • This is usually set by default on managed Kubernetes engine to use the provider's Block storage volumes (bssd) Creature:DeveloperCreature:Architect 16
- 17. Admission controllers AlwaysPullImages Every new Pod will have its pull policy set to “Always”. • Valid credentials are always required. • Images will always be pulled before starting a container. 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion NodeRestriction • Limits the Node and Pod objects a kubelet can modify. • Kubelets must use credentials in the system:nodes group with a specific username. • Ensures kubelets have the minimal set of permissions required to operate correctly Creature:DeveloperCreature:Architect ResourceQuota • Ensures that incoming requests do not violate constraints listed in a namespace’s ResourceQuota object. • Mandatory if you use ResourceQuota objects on your namespaces. PodSecurityPolicy • Determines if the creation or modification of a pod should be admitted. • This admission controller should be set to allow usage of Pod Security Policies (more details later). 17
- 18. Network policies 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion Creature:DeveloperCreature:Architect Worker nodes Namespaces Pods By default, pods are non-isolated and accept traffic from any source. By setting a NetworkPolicy that selects pods, they become isolated. Network policies are additive. The network policy applied on a pod is the intersection of all applicable policies egress/ingress rules. Network policies require a CNI supporting them. Worker nodes Namespace Pods Pods NetworkPolicy1 NetworkPolicy2 NetworkPolicy1 Namespace 18
- 19. Pod security policies 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion Creature:DeveloperCreature:ArchitectCreature:Analyst 19 Pod security policies allow you to define rules in order to accept or reject the creation of pods. This enables the cluster administrators to decide what should run, and what shouldn't, in the cluster. For instance, they could decide to deny all pods that are running as root or that runs in privileged mode.
- 20. Role-Based Access Control 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion Creature:DeveloperCreature:ArchitectCreature:Analyst A standard authorization mode that can easily be used within a Kubernetes cluster is RBAC, by defining a Role or a ClusterRole. Once Roles are defined, they can be bound by defining a RoleBinding (or ClusterRoleBinding) and assigned specific ServiceAccounts. Accesses can be defined by user or by groups of users. 20
- 21. Include security concerns in your cloud native application by design Security as a coding philosophy 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion
- 22. Can we code now? 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion Creature:DeveloperCreature:Architect No! Your cluster is safer now that you defined restrictions and rules within the cluster itself, but what about the accesses from outside that you definitely need? I am talking here about your CI/CD, which should have dedicated accesses to the Container Registry, and to your namespaces. One pipeline does not need access to the entire cluster, but only to a dedicated part of it, with a dedicated user defined. 22
- 23. Now that your CI/CD is set, you may code :) 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion Creature:DeveloperCreature:Architect It may feel like there are a lot of restrictions going on here, but restriction means security. And by designing a production environment with all the necessary security constraints, the value you provide your customers and end users with your services and applications will only be greater. On the development side, there is also the topic of packages vulnerabilities. It may be useful to ask ourselves, on a regular basis, a few simple questions: • Are our lists of required packages/libraries up-to-date? • Do we force a specific version of each package, and how often do we upgrade to the newest version? • If one of our dependancies becomes deprecated, should we invest development time to replacing it so we can keep upgrading our packages? Or should we keep it, no matter the security risk, and freeze our package version from this point? 23
- 24. Security as a coding philosophy Security should not be seen as a constraint, but as an opportunity to learn, improve, and we should be willing to do it by default. It will never be a waste of time. 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion24
- 25. Managing third parties security 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion Tl;dr; Being on the shelf does not make it safe
- 26. Third parties: do you really know what you use? 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion Creature:DeveloperCreature:ArchitectCreature:AnalystCreature:Boss We need third parties because we cannot nor do we want to reinvent the wheel. On-the-shelf services and applications allows us to go faster, open source or on-premise, businesses use third parties for the purpose they serve: monitoring, logging, database, caching, analytics, and much more. Those services, like all others, have been coded by humans, and they are not immune to vulnerabilities. 26
- 27. Third parties: do you really know what you use? 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion Creature:DeveloperCreature:ArchitectCreature:AnalystCreature:Boss Using a container on the shelf might have been built by ROOT user, making it accessible for anyone, including an attacker who got into your production system if it is not secured enough. Those thirds parties must not be forgotten while defining security policies (pods and network), namespace isolation, and user access rights. Is the security level of an “on-the-shelf service" a criteria when designing your architecture and software? 27
- 28. How to put your data in a safe Lock your data 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion
- 29. Your data 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion Creature:UserCreature:DeveloperCreature:ArchitectCreature:AnalystCreature:Boss Users data • Can be stored the same way as your software data, but contain valuable, critical and personal information about your customers. • A leak of your users' data can kill your business. Software data • Often stored in a managed database where its security, redundancy, and availability is ensured by the cloud provider. • Sometimes managed on a dedicated server(s), implying managing the security of the server(s), as well as the redundancy. Analytics data • Statistics, metrics, sensors, valuable data that can be treated and in multiple ways, from simple statistics to transformation pipelines and even machine learning algorithms. • These data can be the added value you have compared to your competitors, and they often require dedicated storage because of their volume, format, and specific querying requirements. 29
- 30. Access restriction, encryption 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion Creature:UserCreature:DeveloperCreature:ArchitectCreature:Analyst Kubernetes and data storage • Kubernetes is often described as stateless, meaning that it is not meant to host persistent data directly on the nodes, which is logical once we consider the node auto-healing and node autoscaling features. • Though, data storage is possible in a Kubernetes cluster, using persistent volumes, such as Block Storage. • Even if running a database in a Kubernetes cluster is not recommended, it is possible and quite easy to do so. • Persistent volumes can be protected by restricting their access right solely to the pod to which they are connected. • Data stored in persistent volumes can also be encrypted if it is supported by the CSI of the cloud provider you use. 30
- 31. Your customer’s data: laws are coming 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion Creature:ArchitectCreature:AnalystCreature:Boss Data under protection • GDPR: General Data Protection Regulation (Europe) • LGPD: Lei Geral de Proteção de Dados Pessoais (Brazil) Customers, users and people are now increasingly protected by laws like the GDPR. These laws state that personal data, such as web browsing information, social information, actions tracked on a website or software... are the property of the user, and that, at any time, they have to be deleted upon the user’s demand. This request has a huge effect on Software and data pipelines, because not only is it a specific feature, but if your software is communicating this data to another software, the deletion order should follow. This also asks the question of anonymised data for machine learning or statistics. For now, a machine learning model based on anonymised data does not have to be re-trained without the removed data, but if the law evolves, it could be the case and have a drastic impact on some software and businesses. The best thing to do is to ensure that this data is safe, anonymous, and legitimate. Only keep what you need, and justify what it is used for. 31
- 32. Tl;dr; if you did not listen nor read, at least take this home with you Security cheat sheet 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion
- 33. Security Cheat Sheet 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion Choose wisely • Your cloud provider: what is its revoke certificate strategy? • Your third parties: are they secured, vulnerable, mandatory? Design before code • The production infrastructure of your software will define the roles, users, and best practices to code with serenity. • Learn and be curious about vulnerabilities. • Know your priorities when it comes to security. • Who can access what, and should they be able to? Start fresh • Set your admission controllers (at least): NamespaceLifeCycle, LimitRanger, ServiceAccount, DefaultStorageClass, PodSecurityPolicy, ResourceQuota. • Define Pod security policies to define rules for pods' creation. • Define Network security policies to isolate pods within your cluster. Be responsible • A production environment is everyone's responsibility • You are in the same boat, don't let other people row when things get complicated 33
- 34. An engaged team makes for a protected customer 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion FirstStrike:Acreaturewithfirststrikewilldealits fightdamagebeforeacreatureitdoesn’t. Trample:Staticability.Afterattacking,thecreature withTramplecanassigntherestofthedamagenot takenbythefirsttargettotargetothercreatures. Creature:Boss “Noonecandefeatthedecisionmaker.” “Scaleway’smascotrules”-Marc-EtiennePeintre 5/* Landwalk:Staticability.Theaffectedcreaturemay notbeobstructedprovidingtheprotectingplayer settingsalandassociatedwiththespecifiedtype. Shroud:Staticability.Theimpactedpermanentor playercannotbethegoalofeverymeansorabilities, togetherwithyourown. Creature:Analyst “Dataiseverythingandmore.” “Connectionsandcorrelations”-Marc-EtiennePeintre 3/2 Lifelink:Fixedcapability.Wheneveracardwith lifelinkdealsdamage,thecontrollerofthiscard additionallygainslifecorrespondingtotheamountof damagedealt. Banding:Whenseveralcreatureswithbanding, temporally'band'together,itcanbecomeone creaturetoeitherassaultorblock. Creature:Developer “It’snotabug,it’safeature!” “Scaleway’smascotspacecowboy”-Marc-EtiennePeintre 2/2 Indestructible: Any permanent which has indestructible can't be damaged, and cannot perish as a consequence of deadly harm. Intimidate: Fixed capability. A creature with intimidate cannot be blocked, except by artefact creatures and also by creatures that share a colour with it. Creature: User “I’m a leaf on the wind, watch how I soar…” “The magnificent” - Marc-Etienne Peintre * / * Creature:Architect “Youneedstrongfoundationsforabuildingtolast.” “Thebuilder”-Marc-EtiennePeintre 1/4 Cover:Staticability.Theaffectedpermanentis grantedwiththeinabilitybecomingdealtdamage, enchanted,blocked,ortargetedbyanycardaidedby thereportedhighquality. Vigilance:Staticability.Theaffectedcreaturedoes nottouchassoonasdeclaredasanattacker. 34
- 35. Better safe than sorry KAPSLOCKED 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion
- 36. Conclusion 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion You should be safer than 50% of other production systems. Good enough? That’s for you to decide. Have a look at the official Kubernetes documentation about the different topics related to this presentation • Certificates • Admission Controllers • Service Account • Network Policies • Pod Security Policy Feel free to join us on slack (https://slack.scaleway.com/) #k8s. Our Kubernetes community now has more than 650 members. 36
- 37. Conclusion “Care, but don’t overthink it” 19/11/2020 - Understand, verify, and act on the security of your clusters - Emmanuelle Demompion37
- 38. #k8s slack channel @ema - @ben - @jth - @Sh4d1 Merci !