DevOpsCon London: How containerized Pipelines can boost your CI/CD
2 likes229 views
The document discusses the advantages of containerized pipelines and Kubernetes for enhancing CI/CD processes, highlighting features like isolation, dependencies, and scalability. It covers various tools including GitLab CI/CD with Kubernetes, Kaniko for building container images, and Tekton for cloud-native pipelines. Additionally, it provides insights into the architecture of pipeline jobs and advanced configurations to optimize CI/CD workflows.
DevOpsCon London: How containerized Pipelines can boost your CI/CD
- 1. How Containerized Pipelines and Kubernetes Can Boost Your CI/CD DevOpsCon London, September 2020
- 2. Nico Meisenzahl • Senior Cloud & DevOps Consultant at white duck • GitLab Hero, Microsoft MVP & Docker Community Leader • loves Kubernetes, Cloud-Native & DevOps © white duck GmbH 2020 Phone: +49 8031 230159 0 Email: nico.meisenzahl@whiteduck.de Twitter: @nmeisenzahl LinkedIn: https://www.linkedin.com/in/nicomeisenzahl Blog: https://meisenzahl.org
- 3. Agenda • containerized pipeline – why? • pipeline workload on Kubernetes with GitLab CI/CD • container image builds on Kubernetes with Kaniko • cloud-native pipelines with Tekton © white duck GmbH 2020
- 4. Containerized Pipelines – Why? • for the same reasons why you should use containers • isolation • dependencies • scalability • immutability • example: your new project needs version X all others still require Y • you can include any kind of build / deploy dependency • NodeJS, .NET Core, Go, Terraform, Ansible… you name it © white duck GmbH 2020
- 5. docker run • running Docker on build hosts is a familiar pattern • but: docker run pipeline jobs have some issues • containers are immutable • you will end up managing caching and artifacts on your own • (and we're not even talking about building containers) © white duck GmbH 2020
- 6. GitLab Runner Kubernetes executor • a Gitlab Runner that is aware of Kubernetes • integrates your CI/CD with Kubernetes • runs itself within your Kubernetes Cluster (as pod) • schedules pipeline jobs as pods • allows you to share your compute and scale your pipelines • easy deployment • one-click • production ready Helm deployment © white duck GmbH 2020
- 7. Cloning, Caching and Artifacts • GitLab Runner Kubernetes executor runs service containers within the pipeline job pod to handle cloning, caching and artifacts • every pipeline job contains out of these stages • prepare → creates pod with build and service containers • pre-build → clones repo, restore cache, download artifacts • build → user build steps • post-build → creates caches and upload artifacts © white duck GmbH 2020
- 8. Advanced topics • supports advanced configurations like • resource limits for build jobs • CI services for build dependencies • node selectors for runners • … © white duck GmbH 2020
- 9. Pipeline job image • contains everything a single pipeline job needs • binaries, libraries, tools, ... • provides all external dependencies • uses a pipeline to build/rebuild it periodically (security fixes!) • you should define fix versions for your dependencies • uses caching to speed up your builds © white duck GmbH 2020
- 10. Pipeline job image © white duck GmbH 2020
- 11. Demo • containerized deploy pipeline • https://gitlab.com/containerized-cicd/containerized-pipelines © white duck GmbH 2020
- 12. Container Image builds on Kubernetes • you may use Docker-in-Docker to build containers in containers • DinD has some issues • exposing Docker socket • mounting /var/lib/docker • privileged mode • those might be acceptable on a build host but not in a Kubernetes cluster © white duck GmbH 2020
- 13. Image builds with Kaniko • image builds without the need of any privileges or dependencies • runs in a container (gcr.io/kaniko-project/executor) • easy to use • you should think about caching to speed up your pipeline • layer caching (layers get pushed to a registry) • base image caching (GitLab cache) © white duck GmbH 2020
- 14. Image builds on Kubernetes with Kaniko © white duck GmbH 2020
- 15. Demo • containerized image builds on Kubernetes with Kaniko • https://gitlab.com/containerized-cicd/containerized-jobimage © white duck GmbH 2020
- 16. Cloud-native pipelines with Tekton • Tekton Pipelines emerged out of the Knative Build project • founding Continuous Delivery Fundation project • „CI/CD framework for Kubernetes” • Jenkins X pipelines are based on Tekton • contributions by Google, IBM, RedHat, Cloudbees, TriggerMesh, … © white duck GmbH 2020
- 17. Tekton Pipelines are … • … Cloud Native • run on Kubernetes • use containers as their building blocks • managed with Controllers • … decoupled • a pipeline can be reused in multiple clusters • tasks can be isolated and are reusable • resources can be swapped between runs © white duck GmbH 2020
- 18. Triggers, CLI & Dashboard • Tekton Triggers can be used to call a pipeline • push, issue, webhook, …. • Tekton also provides a CLI and Dashboard © white duck GmbH 2020
- 19. Demo • cloud-native pipelines with Tekton Pipelines • https://gitlab.com/containerized-cicd/tekton-sample © white duck GmbH 2020
- 20. Questions? Slides: https://www.slideshare.net/nmeisenzahl Demos: https://gitlab.com/containerized-cicd Nico Meisenzahl (Senior Cloud & DevOps Consultant) Phone: +49 8031 230159 0 Email: nico.meisenzahl@whiteduck.de Twitter: @nmeisenzahl LinkedIn: https://www.linkedin.com/in/nicomeisenzahl Blog: https://meisenzahl.org © white duck GmbH 2020