Container security Familiar problems in new technology

Editor's Notes

• #4: It’s impossible to protect what you don’t know is out there which makes visibility into your environment and specifically visibility that can differentiate containers an essential starting point The easiest way to do this through an infrastructure scan that can detect docker hosts The average host is running 8 containers Once you detect container hosts, it’s important to harden them based on best practices like the CIS benchmark for docker but more importantly, this is where the real work begins It’s easy to think of containers security and focus on the the container itself but containerization is more than just a little container, it’s an entirely new application development and deployment paradigm that comes with an entire new ecosystem to protect
• #5: Docker Daemon checks the client request and communicates with the Docker components in order to perform a service whereas, Docker Engine or Docker is the base engine installed on your host machine to build and run containers using Docker components and services Image - a read-only template with instructions for creating a Docker container. Generally comprised of a base image (minimal version of Ubuntu, linux, etc.) additional layers vary by container purpose and resemble modern web applications the most commonly used images are Nginx for running HTTP servers, redis for caching, and postgres Image also contains the declarative manifest for how the container runs e.g. the dockerfile Container - a runnable instance of an image Registry/Repository - stores Docker images. Docker Hub is a public registry that anyone can use, and Docker is configured to look for images on Docker Hub by default. Can also use private registries or registries provided by cloud platforms AWS ECR, Google container reg, azure container registry Client -  the primary way that many Docker users interact with Docker Daemon - listens for Docker API requests and manages Docker objects such as images, containers, networks, and volumes. A daemon can also communicate with other daemons to manage Docker services
• #6: Per datadog 50% of all deployments run in orchestrated environments Pod - A Pod is the basic execution unit of a Kubernetes application–the smallest and simplest unit in the Kubernetes object model that you create or deploy. A Pod represents processes running on your Cluster. Node - A node is a worker machine in Kubernetes, Each node contains the services necessary to run pods and is managed by the master components Cluster - A cluster is a set of machines, called nodes, that run containerized applications managed by Kubernetes. A cluster has at least one worker node and at least one master node. Kubectl - a command line interface for running commands against Kubernetes clusters Kubernetes Master API Server -REST API that validates and configures data for API objects such as pods, services, replication controllers Scheduler - Scheduler that manages availability, performance, and capacity. Controller Manager - Daemon that embeds the core control loops shipped with Kubernetes. etcd – distributed storage system that manages cluster state Kubelet - The primary node agent that runs on each node. The kubelet takes a set of PodSpecs and ensures that the described containers are running and healthy. Kube-proxy - Can do simple TCP/UDP stream forwarding or round-robin TCP/UDP forwarding across a set of back-ends
• #7: Vulnerabilities have been present in every software component since time immemorial and now with increased complexity and software defined everything the number of components that can and will have vulnerabilities is only increasing Containerized environments are susceptible to all manner of component and code vulnerabilities as well as some unique ones that we’ll take a look at next
• #8: Runc is a container escape vulnerability runc through 1.0-rc6, as used in Docker before 18.09.2 and other products, allows attackers to overwrite the host runc binary (and consequently obtain host root access) by leveraging the ability to execute a command as root within one of these types of containers: (1) a new container with an attacker-controlled image, or (2) an existing container, to which the attacker previously had write access, that can be attached with docker exec.  I highlighted vulnerabilities within docker, docker community, and Kubernetes to show that these vulnerabilities can exist within any stage of container adoption from docker community where it might be one developer or a small team to full fledged orchestrated Cryptojacking Identify front facing systems and websites vulnerable to remote code execution and inject code via API or through webform. Code traverses to container environment Code executes when container is spun up, code is executed and commands are sent directly to the shell Cryptomining malware is downloaded through a wget command
• #9: If you ever needed evidence that “trust but verify” isn’t going away any time soon this should be it. Even official versions of images from reputable sources in the official docker hub repository are riddled with vulnerabilities.
• #10: Cyber hygiene – doing the basics right is fundamental to security, more important the protecting yourself immediately from the next 0 day attack Cyber hygiene becomes more difficult as more and more complexity is introduced to the environment. And containerization is very complex with several moving parts, microservices, and network overlays creating an obfuscation layer
• #11: Increased velocity is generally a byproduct of making more, smaller changes and it’s tempting to say that if there are only small changes being made then testing becomes less important This thinking is a recipe for lots of small vulnerabilities getting through and causing havoc especially in microservices and containers that are meant to be deployed in bunches and built for autoscaling, one vulnerability in a container can quickly become 5 vulnerabilities in your production environment Testing remains very important and done right it can remove security bottlenecks throughout the development process
• #12: there are a number of arguments to be made in favor of finding and fixing vulnerabilities as early as possible and that’s even more relevant with containers There shouldn’t be differences between environments which removes the reason of dev/test/qa being behind production and different From the developer’s laptop there’s generally an automated toolchain used for continuous integration and continuous deployment, as soon as an image is built it should go through automated testing An important one and one that’s generally more convincing across stakeholder units is the economic and time argument for finding and fixing vulnerabilities as soon as possible. This leaves minimal time for the person who wrote the initial code to move on to new projects or move on all together so a new person needs to familiarize themselves with the codebase before implementing any fix. This time and money argument can be very persuasive but it needs to be implemented correctly.
• #13: Ensure the tool you’re using to scan images gives layer level detailed information. A lot of patching happens in development and falls to the developers who don’t want to waste cycles guessing what to patch where and falling victim to false positives Suppose that you have a base image that includes mypkg 1.2.7 (without the fix), so we know it is vulnerable. If the scanner simply reports vulnerabilities layer by layer, it will appear that the issue is present in any child image, even if the package gets replaced with version 1.3.0 in a different layer. Another false positive. Let’s say that there’s a fictional package mypkg version 1.2.7 with a vulnerability (that I’m making up) called CVE-999. The maintainers of mypkg fixed the vulnerability, and the fix made it into the release of mypkg 1.3.0. So the vulnerability database might say that CVE-999 is known to exist in any version of mypkg before 1.3.0. Now, several other fixes and features went into 1.3.0 upstream, and let’s imagine that the owner of the image being scanned didn’t want all of them. Instead, she decided to cherry-pick just the fix for CVE-999. She rebuilt her own version of the mypkg and called it 1.2.7.12345. According to the database, it would seem that the vulnerability is still there, because 1.2.7.12345 is lower than 1.3.0 where the fix is known to have been applied. But it’s a false positive. Do not get complacent with images in repositories that have been scanned and patched before. As with any software, vulnerabilities will constantly be found in the software that makes up your image layers. Consistent scanning, patching, and re-deploying new secure versions of any containers running the image is required. This is where the immutability and portability of containers actually plays well with security. There’s no need to deploy complex patches to running applications, you can simply tear down the container and replace. The orchestrator will automatically pull the most recent (secure) version of the image to rebuild the container.
• #15: Access control here comes down to what different users and components can see and what they can do Access control has two purposes, keep malicious actors out and if they get in, limit what they can do and where they can go to limit the blast radius of any breach Now let’s get into several ways you can implement a containment strategy for your containers
• #18: https://medium.com/@mccode/processes-in-containers-should-not-run-as-root-2feae3f0df3b Avoid running containers as uid 0, if possible. Containers leverage two concepts to govern access control and resource utilization – namespaces and cgroups Namespace – what you can see Cgroups what you can do/what resources you have access to – useful for preventing buffer overflow type attacks by limiting the amount of memory a given process/container is allocated
• #19: In addition to these tools, you can drop individual capabilities from your container as part of the build CAP_SYS_ADMIN is a specially nasty one in terms of security, it grants a wide range of root level permissions SecComp, AppArmor and SELinux are linux kernel security modules that can be applied to containers to limit capabilities but one thing to remember is they’re meant to be broadly applicable. So in a sense these are made for “less privilege” but they’re not aware of how your containers will be used in your environment so even after applying these profiles or instead of applying these profiles if they’re not available it’s recommended that you list and remove individual capabilities that aren’t necessary
• #20: Kubernetes namespaces were developed as a method to help provide workload isolation. Running multiple, potentially multi-tenant, workloads in the same namespace sidesteps the protections of namespaces, resulting in a single large and flat namespace. In Docker EE with Swarm mode, administrators have the ability of influencing these scheduling decisions by using labels that are securely attached to the individual node identities. These labels allow administrators to group nodes together into different security zones limiting the exposure of particularly sensitive workloads and any secrets related to them.
• #21: If you’re just starting out with containers it’s good to have a guide and some best practices, the good news is these exist for every level of the container ecosystem Start with the hosts because if you build bullet proof applications on top of swiss cheese infrastructure you’re still extremely vulnerable Next move on to the containers themselves with a detailed overview of common issues and countermeasures as well as things to keep in mind from the design of your first container to decommissioning