Using containers for continuous integration and continuous delivery - Carlos Sanchez
1 like1,125 views
This document discusses using containers for continuous integration and delivery with Jenkins. It covers running Jenkins in Docker containers, using Docker plugins with Jenkins, and running Jenkins pipelines in Kubernetes. Scaling Jenkins with more build agents or masters is discussed. When resource limits are exceeded in containers, the JVM or CPU is restricted instead of the container being killed. The Jenkins Kubernetes plugin allows dynamically running Jenkins agents as Kubernetes pods.
![PERMISSIONS
containers: [...]
securityContext:
fsGroup: 1000
volumes: [...]
Volumes which support ownership
management are modified to be owned
and writable by the GID specified in fsGroup](https://image.slidesharecdn.com/usingcontainersforcontinuousintegrationandcontinuousdelivery-170619081243/85/Using-containers-for-continuous-integration-and-continuous-delivery-Carlos-Sanchez-28-320.jpg)
![JENKINS KUBERNETES PIPELINE
podTemplate(label: 'maven', containers: [
containerTemplate(name: 'maven', image: 'maven:3.3.9-jdk-8-alpine'
ttyEnabled: true, command: 'cat') ]) {
node('maven') {
stage('Get a Maven project') {
git 'https://github.com/jenkinsci/kubernetes-plugin.git'
container('maven') {
stage('Build a Maven project') {
sh 'mvn -B clean package'
}
}
}
}
}](https://image.slidesharecdn.com/usingcontainersforcontinuousintegrationandcontinuousdelivery-170619081243/85/Using-containers-for-continuous-integration-and-continuous-delivery-Carlos-Sanchez-40-320.jpg)
![Multi-language Pipeline
podTemplate(label: 'maven-golang', containers: [
containerTemplate(name: 'maven', image: 'maven:3.3.9-jdk-8-alpine',
ttyEnabled: true, command: 'cat'),
containerTemplate(name: 'golang', image: 'golang:1.8.0',
ttyEnabled: true, command: 'cat')]) {
node('maven-golang') {
stage('Build a Maven project') {
git 'https://github.com/jenkinsci/kubernetes-plugin.git'
container('maven') {
sh 'mvn -B clean package'
}
}
stage('Build a Golang project') {
git url: 'https://github.com/hashicorp/terraform.git'
container('golang') {
sh """
mkdir -p /go/src/github.com/hashicorp
ln -s `pwd` /go/src/github.com/hashicorp/terraform
cd /go/src/github.com/hashicorp/terraform && make core-dev
"""
}
}
}](https://image.slidesharecdn.com/usingcontainersforcontinuousintegrationandcontinuousdelivery-170619081243/85/Using-containers-for-continuous-integration-and-continuous-delivery-Carlos-Sanchez-41-320.jpg)
Using containers for continuous integration and continuous delivery - Carlos Sanchez
- 1. USING CONTAINERS FOR CONTINUOUS INTEGRATION AND CONTINUOUS DELIVERY Carlos Sanchez /csanchez.org @csanchez Watch online at carlossg.github.io/presentations
- 2. ABOUT ME Engineer @ CloudBees, Scaling Jenkins Author of Jenkins Kubernetes plugin Contributor to Jenkins Mesos plugin & Jenkins and Maven official Docker images Long time OSS contributor at Apache Maven, Eclipse, Puppet,…
- 5. USING CONTAINERS IS NOT TRIVIAL
- 7. SCALING JENKINS Two options: More build agents per master More masters
- 8. SCALING JENKINS: MORE BUILD AGENTS Pros Multiple plugins to add more agents, even dynamically Cons The master is still a SPOF Handling multiple configurations, plugin versions,... There is a limit on how many build agents can be attached
- 9. SCALING JENKINS: MORE MASTERS Pros Different sub-organizations can self service and operate independently Cons Single Sign-On Centralized configuration and operation Covered by CloudBees Jenkins Enterprise
- 14. JENKINS DOCKER PLUGINS Dynamic Jenkins agents with Docker plugin or Yet Another Docker Plugin No support yet for Docker Swarm mode Isolated build agents and jobs Agent image needs to include Java, downloads slave jar from Jenkins master
- 15. JENKINS DOCKER PLUGINS Multiple plugins for different tasks Docker build and publish Docker build step plugin CloudBees Docker Hub/Registry Notification CloudBees Docker Traceability Great pipeline support
- 18. JENKINS DOCKER PIPELINE def maven = docker.image('maven:3.3.9-jdk-8'); stage('Mirror') { maven.pull() } docker.withRegistry('https://secure-registry/', 'docker-registry-login') { stage('Build') { maven.inside { sh "mvn -B clean package" } } stage('Bake Docker image') { def pcImg = docker.build( "examplecorp/spring-petclinic:${env.BUILD_TAG}", 'app') pcImg.push(); } }
- 19. WHEN ONE MACHINE IS NO LONGER ENOUGH Running Docker across multiple hosts In public cloud, private cloud, VMs or bare metal HA and fault tolerant
- 21. If you haven't automatically destroyed something by mistake, you are not automating enough
- 24. KUBERNETES Based on Google Borg Run in local machine, virtual, cloud Google provides Google Container Engine (GKE) Other services run by stackpoint.io, CoreOS Tectonic, Azure,... Minikube for local testing
- 25. GROUPING CONTAINERS (PODS) Example: Jenkins agent Maven build Selenium Hub with Firefox Chrome 5 containers
- 26. STORAGE Jenkins masters need persistent storage, agents (maybe) Persistent volumes GCE disks GlusterFS NFS EBS etc
- 27. PERMISSIONS Containers should not run as root Container user id != host user id i.e. jenkins user in container is always 1000 but matches ubuntu user in host
- 28. PERMISSIONS containers: [...] securityContext: fsGroup: 1000 volumes: [...] Volumes which support ownership management are modified to be owned and writable by the GID specified in fsGroup
- 29. NETWORKING Jenkins masters open several ports HTTP JNLP Build agent SSH server (Jenkins CLI type operations) Jenkins agents connect to master: inbound (SSH) outbound (JNLP)
- 30. Multiple :networking options GCE, Flannel, Weave, Calico,... One IP per Pod Containers can find other containers in the same Pod using localhost
- 31. MEMORY LIMITS Scheduler needs to account for container memory requirements and host available memory Prevent containers for using more memory than allowed Memory constraints translate to Docker --memory https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#how- pods-with-resource-limits-are-run
- 32. WHAT DO YOU THINK HAPPENS WHEN? Your container goes over memory quota?
- 34. NEW JVM SUPPORT FOR CONTAINERS JDK 8u131+ and JDK 9 $ docker run -m 1GB openjdk:8u131 java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XshowSettings:vm -version VM settings: Max. Heap Size (Estimated): 228.00M Ergonomics Machine Class: server Using VM: OpenJDK 64-Bit Server VM Running a JVM in a Container Without Getting Killed https://blog.csanchez.org/2017/05/31/running-a-jvm-in-a-container-without-getting-killed
- 35. NEW JVM SUPPORT FOR CONTAINERS $ docker run -m 1GB openjdk:8u131 java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=1 -XshowSettings:vm -version VM settings: Max. Heap Size (Estimated): 910.50M Ergonomics Machine Class: server Using VM: OpenJDK 64-Bit Server VM Running a JVM in a Container Without Getting Killed https://blog.csanchez.org/2017/05/31/running-a-jvm-in-a-container-without-getting-killed
- 36. CPU LIMITS Scheduler needs to account for container CPU requirements and host available CPUs CPU requests translates into Docker --cpu-shares CPU limits translates into Docker --cpu-quota https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#how- pods-with-resource-limits-are-run
- 37. WHAT DO YOU THINK HAPPENS WHEN? Your container tries to access more than one CPU Your container goes over CPU limits
- 38. Totally different from memory
- 39. JENKINS KUBERNETES PLUGIN Dynamic Jenkins agents, running as Pods Multiple container support One jnlp image, others custom Pipeline support for both agent Pod definition and execution Persistent workspace
- 40. JENKINS KUBERNETES PIPELINE podTemplate(label: 'maven', containers: [ containerTemplate(name: 'maven', image: 'maven:3.3.9-jdk-8-alpine' ttyEnabled: true, command: 'cat') ]) { node('maven') { stage('Get a Maven project') { git 'https://github.com/jenkinsci/kubernetes-plugin.git' container('maven') { stage('Build a Maven project') { sh 'mvn -B clean package' } } } } }
- 41. Multi-language Pipeline podTemplate(label: 'maven-golang', containers: [ containerTemplate(name: 'maven', image: 'maven:3.3.9-jdk-8-alpine', ttyEnabled: true, command: 'cat'), containerTemplate(name: 'golang', image: 'golang:1.8.0', ttyEnabled: true, command: 'cat')]) { node('maven-golang') { stage('Build a Maven project') { git 'https://github.com/jenkinsci/kubernetes-plugin.git' container('maven') { sh 'mvn -B clean package' } } stage('Build a Golang project') { git url: 'https://github.com/hashicorp/terraform.git' container('golang') { sh """ mkdir -p /go/src/github.com/hashicorp ln -s `pwd` /go/src/github.com/hashicorp/terraform cd /go/src/github.com/hashicorp/terraform && make core-dev """ } } }
- 42. JENKINS PLUGINS CAVEATS Using the Cloud API Not ideal for containerized workload Agents take > 1 min to start provision and are kept around Agents can provide more than one executor
- 43. JENKINS PLUGINS CAVEATS One Shot Executor Improved API to handle one off agents Optimized for containerized agents Plugins need to support it