Getting started with Docker
Download as PPTX, PDF1 like1,423 views
The document is a presentation by Ravindu Fernando that explores the evolution and technology of Docker, explaining its function as a container engine and the differences between virtual machines and containers. It covers the architecture of Docker, how containers operate, and provides basic commands for managing Docker containers. The presentation also discusses advanced topics like Docker networking, data persistence through volumes, and the enhancements offered by Docker BuildKit and Buildx for efficient image builds.
Getting started with Docker
- 1. GETTING STARTED WITH DOCKER FOSS LK DEVOPS EVENT SERIES – AUGUST 2021
- 2. ABOUT ME • Hi, I’m Ravindu Fernando • Associate Tech Lead @ Emojot Inc. • Cloud Computing Enthusiast
- 3. AGENDA • Brief History – Infrastructure shifts over the decades • VMs vs Containers • What are containers and what problem does it solve? • What is Docker? • Deep dive into Docker Internals • Demo
- 4. BRIEF HISTORY – INFRASTRUCTURE SHIFTS OVER THE DECADES Mainframe to PC 90’s Baremetal to Virtual 00’s Datacenter to Cloud 10’s Host to Container (Serverless) Now Let’s Recap MAJOR INFRASTRUCTURE SHIFTS
- 5. VMS VS CONTAINERS Host Operating System Hypervisor (Type 2) APP 1 Bins/ Libs Host Operating System Infrastrucutre Infrastrucutre Guest OS Guest OS Guest OS Container Engine APP 3 Bins/ Libs APP 2 Bins/ Libs APP 1 Bins/ Libs APP 3 Bins/ Libs APP 2 Bins/ Libs -Virtual Machines- -Containers-
- 6. WHAT ARE CONTAINERS AND WHAT PROBLEMS DOES IT SOLVE?
- 7. MATRIX FROM HELL INCREASES THE COMPLEXITY
- 8. CONTAINERS REDUCES THE COMPLEXITY
- 9. IN SUMMARY A CONTAINER IS, • Just an isolated process running on the host machine. And a restricted process. • Will share OS and, where appropriate, bins/ libraries and limited to what resources it can access. • It exits when the process stops. “Containers are the next once-in-a-decade shift in IT infrastructure and process”
- 10. WHAT IS DOCKER?
- 11. • So what’s Docker? – In 2021, Docker means lot’s of things, let’s just clear things out. • Docker as a “Company” • Docker as a “Product” • Docker as a “Platform” • Docker as a “CLI tool” • Docker as a “Computer Program”
- 12. WHAT IS DOCKER? • Docker provides the ability to package and run applications within a loosely isolated environment which is a a container. Simply it’s a container engine. • It provides tooling and a platform to manage lifecycle of your containers, • Develop your apps and supporting components using containers • Distribute and test your apps as a container • Ability to deploy your app as a container or an orchestrated service, in whatever environment which supports Docker installation • It shares the same OS kernel • It works on all major Linux Distributions and containers native to Windows Server (specific versions)
- 13. UNDERLYING TECHNOLOGY IN DOCKER • Docker is an extension of LXC’s (Linux Containers) capabilities and packaged it in a way which is more developer friendly. • It was developed in Go language and utilizes LXC, namespaces, cgroups and the linux kernel itself. Docker uses namespaces to provide the isolated workspace called a container. Each aspect of a container runs in a separate namespace and its access is limited to this namespace.
- 16. • Docker CLI structure, • Old (Still works as expected) docker <command> options • New – docker <command> <sub-command> (options) • Pulling Docker Image • docker pull nginx • Running a Docker Container • docker run –p 80:80 --name web-server nginx • Stopping the Container • docker stop web-server (or container id)
- 17. • Check what’s happening in a containers, • docker container top web-server – Process list in 1 container • docker container inspect web-server – Details of one container config • docker container stats – Performance stats for all containers • Getting a shell inside containers, • docker container run –it – Start a new container interactively • docker container exec –it <container_id_or_name> echo “I’m inside the container” – Run additional commands in the container • Listing, removing containers and images • docker images • docker container ls | docker ps • docker <object> rm <id_or_name>
- 18. DEEP DIVE INTO DOCKER INTERNALS
- 20. WHAT HAPPENS WHEN YOU RUN A CONTAINER? • docker run –p 80:80 nginx | docker container run –p 80:80 nginx 1. Looks for that particular image locally in image cache, if its not found pulls it from the configured registry (image repository). Downloads the latest version by default (nginx:latest) 2. Creates a new container based on that image and prepares to start 3. Docker allocates read write filesystem to the container, as its final layer. This allows running container to modify files and directories in its local filesystem. 4. Gives it a virtual IP on a private network inside docker engine 5. Opens up port 80 on host and forwards to port 80 in container. 6. Starts container by using the CMD in the image Dockerfile.
- 21. DOCKER OBJECTS • Docker Images • A read-only template with instructions/ metadata for creating a Docker container. • Can create your own image or use images created and published in a registry by others. • Dockerfile can be used to define steps required to create and run the image. • Each instruction in Dockerfile creates a layer in the image, only those layers which changes each time are rebuilt – What makes images so lightweight, small and fast.
- 22. • Docker Containers • Runnable instance of an image. • Can create, start, stop, move, or delete a container using the Docker API or CLI. • Can connect it to one or more networks, attach storage to it, or even create a new image based on its current status. • A container is defined by its image as well as any config options provided to it when you create or start it. Note that when the container is removed any data associated with it will be deleted unless those are not stored in a persistent storage.
- 23. UNDERSTANDING DOCKER IMAGES/ CONTAINERS INTERNALS • Docker Filesystem • Boot file system (bootfs) – Contains the bootloader and the kernel. User never touches this. • Root file system (rootfs) – Includes the typical directory structure we associate with Unix-like OS.
- 24. • In traditional Linux boot, kernel first mounts the rootfs as read-only, checks its integrity, and then switches the rootfs volume to read-write mode. • Docker mounts the rootfs and instead of changing the file system to read-write mode, it then takes advantage of union mounts service to add a read-write filesystem over the read-only file system. • In Docker terminology, a read-only layer is called an image. An image never changes and is fixed. • Each image depend on one more image which creates the layer beneath it. The lower image is the parent of the upper image. Image without a parent is a base image. • When you run a container, Docker fetches the image and its Parent Image, and repeats the process until it reaches the Base Image. Then the Union File System adds a read-write layer on top. • That read-write layer, plus the information about its Parent Image and some additional information like its unique id, networking configuration, and resource limits is called a container
- 26. • A container can have two states, it may be running or exited. • When a container is exited the state of the file system and its exit value is saved. • You can start, stop, and restart a container. The processes of restarting a container from scratch will preserve its file system is just as it was when the container was stopped. But the memory state of the container is not preserved. • You can also remove the container permanently. • A container can also be promoted directly into an image using the docker commit command. Once a container is committed as an image, you can use it to create other images on top of it. • docker commit <container-id> <image-name:tag>
- 27. • Based from the UFS, Docker uses a strategy called Copy on Write to improve the efficiency by minimizing I/O and the size of each subsequent layers, • If a file or directory exists in a lower layer within the image, and another layer (including the writable layer) needs read access to it, it just uses the existing file. • The first time another layer needs to modify the file (when building the image or running the container), the file is copied into that layer and modified.
- 28. • Docker Image Creation and Storage • You can create an image using a Dockerfile or by committing a container’s changes back to an image. • Once you create an image, it will be stored in the Docker host’s local image cache. • In order to move images in/out of the local image cache, • Export/ Import it as a tarball • Push/ pull to a remote image registry (ex - DockerHub)
- 29. DOCKER OBJECTS CONT… • Docker Networks • Each container is connected to a private virtual network called “bridge”. • Each virtual network routes through the NAT firewall on the host IP. • All containers on a virtual network can talk to each other without exposing ports. • Best practice is to create a new virtual network for each app.
- 30. • Docker enables to: • Create new virtual networks. • Attach container to more than one virtual network (or none) • Skip virtual networks and use host IP (--net=host) • Use different Docker network drivers to gain new abilities. • Docker Engine provides support for different network drivers – bridge (default), overlay and macvian etc.. . You can even write your own network driver plugin to create your own one. • Docker Networking – DNS • Docker deamon has a built in DNS, which consider container name as equivalent hostname of the container.
- 33. PERSISTENCE DATA • If we want to use persistence data as in like databases or unique data in containers, Docker enables that using two ways, • Volumes – Make a location outside of container UFS. • Bind Mounts - Link host path to the container path.
- 34. DOCKER COMPOSE • Another Docker client, that lets you work with apps consisting of a set of containers. • This saves docker container run settings in easy to read file, which can be committed to VCS. • Can use this to create one-line development environments • Consists of two components • YAML formatted file that describes – Images, Containers, Networks, Volumes etc… • A CLI tool docker-compose used to automate/manage those YAML files
- 35. DOCKER BUILDKIT & BUILDX • BuildKit enables higher performance docker builds and caching possibility to decrease build times and increase productivity for free. (https://github.com/moby/moby) • Standard docker build command performs builds serially, which means reads and builds each line or layer of the Dockerfile one layer at a time. With Buildkit enabled, it allows for parallel build processing resulting in better performance and faster build times. • It also enables the use of cache and storing cache in remote container repositories like DockerHub for better build performance as we don't have to rebuild every layer of an image. • You can enable BuildKit in places you already uses docker build including within your CI/CD pipelines to reduce the build times.
- 36. DOCKER BUILDKIT & BUILDX CONT… • Docker Buildx is a CLI plugin that extends the docker command with the full support of the features provided by BuildKit plus additional features. (Included within Docker Desktop versions & Docker Linux packages. You can even download as a source from Github) • Features of buildx, • Familiar UI from docker build • Full BuildKit capabilities with container driver • Multiple builder instance support • Multi-node builds for cross-platform images • High-level build constructs (bake)
- 37. DOCKER BUILDKIT & BUILDX PERFORMANCE Case Classic Builder BuildKit + buildx Dependency Change 6 min 6 min Code Change 6 min 3 min No Change 6 min 1 min Jiang Huan BuildKit timings (Look for references section)
- 38. DEMO • Running/ Stopping/ Removing a NGINX container using Docker CLI • Building/ Running/ Shipping a NodeJS app with Docker • Running multi-component app with Docker Compose • Buildx demo with BuildKit – Multi-platform image creation
- 39. Q/A
- 40. REFERENCES • https://docs.docker.com/get-started/overview/ • https://www.docker.com/blog/containers-and-vms-together/ • https://www.redhat.com/en/topics/containers/containers-vs-vms • Docker Storage Drivers - https://docs.docker.com/storage/storagedriver/ • https://docs.docker.com/storage/storagedriver/select-storage-driver/ • https://www.youtube.com/watch?v=cjXI-yxqGTI • Docker Buildx - https://docs.docker.com/buildx/working-with-buildx/ • Jiang Huan BuildKit timings - https://medium.com/titansoft-engineering/docker-build-cache- sharing-on-multi-hosts-with-buildkit-and-buildx-eb8f7005918e • What is Docker BuildKit - https://brianchristner.io/what-is-docker-buildkit/
- 41. THANK YOU! LinkedIn - https://lk.linkedin.com/in/ravindufernando
Editor's Notes
- #5: Why containers are such big deal? Mainframe to PC – PC distributed arch/ Changing networks and putting in Fiber/ TCP-IP Baremetal to Virtual – Servers were too powerful, had lot of idle time. Better ways to utilize it was virtualizing. Lots of OS within single piece of H/W. Those are defaults, we are still using them. DC to Cloud – Easy, cheap, disposable compute power via an internet connection. Everyone is know using cloud in some form. Containers – Serveless/ FaaS made posible by containers. Because they are running within containers. We are allready at levels where this is the default way to run apps These migration waves has been happening so quickly.
- #6: Explain highlevel components of both VMs and Containers. Note on the Type 2 VMs as it more common to you. Type 1 virtualization runs directly on the H/W while Type 2 uses host OS to provide virtualization management and other services Type 1 – Hyper V/ Type 2 – VirtualBox/ VMWare In summary VM is isolation of machines and containers are isolation of processes. Not two competing technologies, can be used hand in hand based on the scenario
- #8: Matrix of Hell - it is the challenge of packaging any application, regardless of language/frameworks/dependencies, so that it can run on any cloud, regardless of operating systems/hardware/infrastructure.
- #9: How containers solves the matrix of hell problem
- #10: After talking about points --> This isolation of containers processes and limiting to what resources it can access, is done using main two features called namespaces/ cgroups in the linux kernel level. Will talk more on this on upcoming slide After talking about the slide --> Containers are more useful in App modernization efforts and micro services as they offer speed, lightweight and portability - advantages of containers over the VMs.
- #12: Started in 2013 – Open soruce project by company called dotnet-cloud. Started a new company called Docker Inc. When you say Docker, lots of people means lots of things, but on this session we mainly focus on the Docker as a platform. From this point onwards when we mean Docker, we refer to the Docker platform
- #13: Gret UX for developers to interact with containers
- #14: Linux kernel provides cgroups that allow the host CPU to better partition memory allocation into isolation levels called namespaces Docker gives something more on top of LXC to manage and use it more user friendly
- #15: How docker solves the matrix of hell problem
- #20: Docker uses Client Server Architecture. Docker client talks to the Docker deamon which may run on the same system or can event connect to a remote one Client and deamon communication is done using a REST API, over unix sockets or a network connection Docker deamon (dockerd) – Listens to Docker API req and manages Docker objects (images, containers, networks, volumes). It can also communicate with other deamons to manage services which happens in Docker swarm Docker Client (docker) – primary way of users interacting with Docker. Command scenario. docker command uses Docker API Docker Registry – Stores docker images. Docker Hub is a public one which anyone can use and by default docker checks for images here. There are lot of other options out there and you can even host your own private registry.
- #24: Understanding how things work internally allows allows you to understand why Docker is able to perform more faster and efficient. Plus also note some important points for writing Dockerfiles After rootfs point In Linux and other UNIX like systems everything is based on the Filesystem Hierarchy Standard
- #25: Union mounting - UFS is a way of combining multiple directories into 1 that appears to as a all those combined . The Docker storage driver is responsible for stacking these layers (overlay2, aufs etc..) Images in next slide will help you to understand it better
- #26: You can see the writable layer is the container. So the only diff between image vs container is the top read/writable layer. When container is deleted read/writable layer is deleted Since each container has its own read/writable container layer, this means multiple containers can share access to the same underlying image and yet have their own data sate
- #31: Bridge network – Default one, containers are by default deployed here limited to a single host running Docker Engine. Overlay Network - An overlay network can include multiple hosts
- #32: Docker0 is the default bridge network, you can even create your own networks
- #33: Docker networking allows you to attach a container to as many networks as you like. You can also attach an already running container.
- #35: I won’t cover Docker Swarm on this session as it comes under container orchestration. I felt that it will drag this session as we have a demo session planned. But in simple SWARM is a container orchestration solution which allows you to manage multiple containers deployed across multiple host machines. Bit simillar to K8’s but less complex which means can’t extend into the levels we see with K8’s. Begineer friendly.
- #36: Moby project is an open framework created by Docker Inc. to assemble specialized container systems without reinventing the wheel.
- #37: Multi-node builds for cross-platform images – Before version 19.03 building multi platform images required you to manually create manifest files and build images separately. With Buildx, all these are included and allows you to create multi-platform images Building multi platform images will comes in handy if you are an image publisher and requires to build images for multiple architectures, like for raspberry-pi with linux/arm/v7 | v8 or other archs like linux/amd64, linux/arm64. The bake command supports building images from compose files, similar to docker-compose build, but allowing all the services to be built concurrently as part of a single request.
- #38: All in all these are bit advanced features. I added this specifically to talk about creating multi-platform images and performance improvements you get with BuildKit. These are not features you will use upfront when starting with Docker, but just wanted to point out theses features exists for you to look into and use to optimize. Will do a simple demo on this section. But I invite you to read about this and try this out and then you can use it in prod with confidence. Refer the reference section
- #39: Demo uses basic versions, just to showcase the capabilities. To know more I invite you to read the documentation and try out new things. Best way to learn