Baking docker using chef

Editor's Notes

• #2: Hello Chefs! Good afternoon!  I will be talking about using Docker with Chef! Baking docker using chef! 
• #3: Currently I am working with IHT. At IHT we are building a new product with an exciting technology stack. The product is primarily written in Scala, uses Akka framework and Triple store (triple rush). Similarly We are implementing a solid DevOps strategy. My job and role perfectly suits my passion for technology, innovation and the thirst to keep learning. Flew 20 hrs from india
• #4: We will go through a quick introduction to docker Why Chef and Docker are needed together to build an awesome solution We will Build a very simple CD pipeline that uses knife-ssh Quickly delve into using push jobs , quickly since yesterday we had a very good session on push jobs by See how we can Use the docker chef cookbook to manage docker And Conclude with Chef Containers
• #6: lightweight virtualization provided by libraries inside linux kernel Docker engine is a portable, lightweight runtime and packaging tool Docker Hub: is a cloud service for sharing applications and automating workflows Docker images: layered file system
• #7: guest operating system - which may weigh 10s of GB. Docker runs as an isolated process in userspace on the host operating system, sharing the kernel with other containers. Thus, it enjoys the resource isolation and allocation benefits of VMs but is much more portable and efficient.
• #8: dockerfiles help you codify your configuration, its a set of bash commands which results in a docker image. Docker images are diff to maintain. not idempotent This is a very basic example of config management done inside docker image. Consider repo HelloScala with a dockerfile & Hello-scala.conf. The right side shows the dockerfile which picks up the conf and uses it for the apache config. There is a need to do high quality config management inside docker images, while taking advantage of the many benefits docker containers provide. Example – *instead of long bash script in your dockerfile, you can use chef recipe *if you have already invested in chef, don’t have to make big changes
• #9: Shared Hosting – PaaS Many hosted CI servers like Travis-CI or internet search providers use Docker for infrastructure virtualization and application isolation Microservices microservices architecture has small services which have independent deployment but depend on each other. Docker provide an ideal environment for deployment of these services with respect to speed, isolation, and lifecycle ,management. Lightweight Testing Docker is widely used in testing applications. Docker containers can be destroyed post testing and new ones are created for every test run. *There is support in test kitchen to test cookbook inside docker containers
• #11: we want to automate everything. In DevOps. Iteratively progress to achieve automation We want to include all the awesome features. We want to ensure we stay in the market even after a couple of years. And that IS an important consideration - when the industry and technology are moving at such a fast pace! We want to do it all.. and yet deliver on time! We can achieve this is by defining a solution that would give us the best of both the worlds! 
• #12: Chef Vs Docker is like considering CM Vs golden images. With CM you can control the environment by using tools like Chef. Docker belongs to the school of golden-images which favours clones of a single and well maintained system image. Generally, people look down on Golden Images as The state of the machine tends to change over time and becomes an unmanageable mess. Golden images were gradually discarded for more advanced methods of config management! But having said that, golden images have been pulled off successfully for a long time. Now With Docker or containerization generally, the concept of immutable infrastructure has again come to the fore. Immutable infrastructure is a stack that you build once, run any number of times as new instances, and never change again. If you want to change deployment then you need to terminate the instance or container and start over from step one: that is, build a new image.
• #13: Docker attacks a subset of problems in DevOps, while an umbrella of solutions is provided by Chef. Lets see why & how we can do that Complementary
• #14: Chef was born to replace human tasks. It was based on 3 core principles: idempotence - configuration should be defined such that it can run on any machine, “pull” philosophy thereby maintaining a thin server, and the importance of the order in which the configuration is defined. Chef grew up to be one of the most prefered CM tools, with a large community contribution. Its tried and tested. Its safe. Then came the container era. Containers are faster. With Docker, there have been debates that complex CM tools are not needed. Docker makes your life simpler, they say. But Docker is new and not as much explored. There is risk in getting docker to production. Lets bake the fresh fish into a hot and healthy meal!
• #16: Consider you have a github repo which triggers build on your CI server with a git push. Most of the build tools now have docker support. With a successful build & maybe tests, a docker image is created by the build tool. The docker image can be saved to the docker registry. It can be given a unique tag in the docker registry. This docker image needs to run on your node. As we know, it is just 3 commands that need to run: docker pull latest images (or pull a particular tag), stop the running container, start a new container. Note – docker stop may or may not be needed, or it can be stopped later as per the requirement or design. How can you take your deployment to your nodes? Use knife-ssh!
• #17: Here are the steps in detail, of how this can be done. A sample demo project, HelloScala triggers a Travis-CI build. SBT is “scala build tool”. Sbt docker command creates a docker image. Muktaa/hello-scala is a repo on the docker registry where we would save the docker images. Other build toolsl ike maven offer docker integration too. Run the command mvn docker:build to create docker image That’s it! Did I hear using Chef is complex? Esp in this particular case?
• #18: That’s how the command execution looks…
• #19: We talked about docker registry which woud save docker images. Docker hub provides a hosted docker registry. The links above show 2 different repos on the docker registry. We can setup automated builds in that registry. With git push, this build can be triggered, or the build can be triggered from your CI server using the API call. However this feature is in a crude phase with very less flexibility, it takes a long time for the docker image to build. You can setup the docker registry in your data centre too. It works very similar to the docker hub.
• #21: The CD pipeline we saw last was almost like a push model. Almost. Because the changes were pushed into the node by the CI server and not your chef server! The world seems to be going round and round, traversing the same path again and again! Golden images are coming back. Chef too had a reason for the pull model, to keep the server “thin”. But the changing challenges demand that there is a need for a pull model. So chef has introduced push jobs by keeping the server is thin! Chef push jobs is an extension of the Chef server that allows jobs to be run against nodes independently of a chef-client run – that’s how push jobs are defined. A job is a set of commands that need to be run on the target node. From docker perspective the commands are docker pull, stop & run.
• #22: As mentioned in the previous slide, push jobs fit in the real “push model”. <Read out the diff>
• #23: You need either Enterprise Chef or Chef Server 12. It relies on the ACL system that was open sourced with Chef Sever 12. Also, the install command was introduced with Chef Server 12. Push Jobs does not work with Open Source Chef Server 11.  Can be setup as standalone or as HA Run the commands on your chef server, to set it up to use the push jobs feature.
• #24: Install the knife push plugin Push jobs cookbook would be used. So download it from the site or git clone the cookbook. You would have to fetch its dependency cookbooks as well. Update the attributes to add the push jobs package URL and checksum as mentioned. Upload the cookbook to your chef server
• #25: Simply run the chef client with the recipe Run the knife node status commands to check the node status. It will just show the status “available” at this stage which confirms that the node is prepared for push events. Create the pushy_job_writers and pushy_job_readers on the organization of the Chef server and add your workstation user to that group.
• #27: Knife-ssh won for us. Push jobs need to mature. They are still under development and complex to implement. I believe if people are comparing Ansible with Chef for this feature, it should be compared with knife-ssh! 
• #28: We looked at the demo use case, but in actuality the systems are complex
• #29: Docker image is made up of the Application and the configuration.
• #30: By configuration, we mean the setups needed for the application to run as well as the environment specific configuration. Packages: apache. Softwares: jdk Files: xml Log levels Eg: There is a database that the application reads. This DB is different for the 3 environments. Consider the case where the application reads some reference data. This ref data source and volume varies in all environments. More specifically consider an Akka cluster, which is a group of nodes and few nodes are defined as the SEED nodes. This akka config is different for diff environments
• #31: Docker containers that host the application, or components of the application. The workflow is such that the docker containers need to be deployed continuously to diff environments. Dev, pre prod and prod. Each env has a different configuration. Now also consider the case, that initially the docker container runs on DEV. When its performance meets expectations, it needs to be promoted to the pre prod and then to the prod env. Since the environment conf is embedded inside the docker image, any idea how we can promote the docker container from DEV to preprod? Docker containers can run on any machine or VM so the “environment” should not matter, you say? Let me explain further...
• #32: Managing Credentials is an unsolved problem with Docker today Hard coding or setting variables is not secure, they are the workarounds. So if in the prev slides you thought you could pass the configuration to the docker containers, then reconsider the approach for passing the credentials in that manner! Well, one can be adamant and find workarounds, which would only get more complex. If you decide to use Chef, this can be managed using databags.
• #33: This worked well for us, until we had to keep making changes to the configuration
• #35: The docker cookbook is available in chef supermarket Using it, you can install docker, build docker images, commit & push to docker registry, pull image and run container. For the docker image management and deploy, the 3 LWRPs are useful.
• #37: Using lwrp docker_image to build and push the image to docker registry
• #40: Elegant and working solution
• #42: Package that provides config managemet for your containers
• #43: chef-client Latest chef-client that runs within the container. runit - RUNit - lightweight cross-platform init scheme to ensure all child processes are properly managed chef-init - root process which can launch and manage multiple processes inside a container. It is custom built by chef. Each OS has an init. docker replaces the init of the OS. chef-init runs as PID1 and delegates managing child processes to runit.
• #44: Bootstrap the chef-client without an SSH connection Manage multiple services inside your container Use the knife container plugin to work with Linux containers; use the docker build and docker init arguments to manage Docker image contexts Use chef-client resources the same way in a container as on any UNIX- or Linux-based platform Consistency across architecgures – you don’t need different CM tools for Physical, virtual, or machines on cloud. If you have invested in Chef already which runs for you on a physical machine you leverage the same config on chef container Mixed architecture. You might be using docker for development or test environments. But not in production. If you use chef containers for your dev & test env but physical machines on production, then it is lot easier to manage the config on each, as the config is idempotent if you use Chef for CM.
• #45: transitioning trad architecture to containers - all you need is the chef run list, it can run on bare metal, VMs or containers. handling last mile config when container boots - registering an agent you need for monitoring maybe,  or some env specific changes. e.g.: in our DEV env we read some reference data which is a truncated version of the real data while in the test env its a full blown test data. and in prod it is the real data & not the synthetic test data. Using chef containers is not complex. You can define container configuration using chef recipies instead of long bash scripts Chef container can be used to manage docker images. You can set certain services to launch when the container launches using the enable action of the chef resource. When chef-init starts, it can launch chef-client and you can configure which service you want to start using start action of the chef resource.
• #46: 1. install knife-container gem knife container is the only command needed to manage the docker lifecycle 2 Knife container docker init creates the docker context which comprises of docker components and chef components To initialize the Docker context, use the init command. The knife container uses a folder called dockerfiles to organize all the Docker contexts that you manage. By default, the dockerfiles folder is created in your chef repo. To initialize the Docker context, type the following command: Pass in your image name (in this example,demo/apache2), a run list, a –zand a –b. The –z is for local mode and the –b says to generate a Berksfile.
• #48: { "run_list": [ "recipe[apache2]" ], "container_service": { "apache2": { "command": "/usr/sbin/apache2 -k start" } } } Resolves chef dependencies Build docker image Cleansup chef artifacts
• #49: Note step 2: chef-init —bootstrap (this runs the chef-client) step 1: adds the chef dir to /etc/chef step 0: & last tag: only 1 copy of the image, see docker images so we created an image from ubuntu 14.04 and created image, the new image will have same tag but diff image id.
• #51: Our product is still under development. It is not yet live. So how easy can you imagine the DevOps work would be?
• #52: 6 man years of code. All developed by pure developers, without any Ops or systems idea. We had to start at: how do we run the application?! We had to figure out how to run using SBT – the scala build tool!  And just while we figured out how to run it, the product architecture gets changed. And the process continues. It was not difficult to get the application working as docker. We used the scala build tool to create docker images that ran as docker containers and that ran rather well. But there were a couple of lessons we learnt…
• #53: We will be using containers, we are planning to use mesos.