MHA2018 - Docker and Jenkins Pipeline for Continuous integration - Mark Waite
- 1. Docker & Jenkins Pipeline Continuous Integration – Mark Waite Mile High Agile 2018
- 2. The Intersection Accident Problem Accidents frequently happen at intersections
- 3. The Software Integration Problem Merge Team 1 Team 2 Team 3 • Integration frequently fails • Conflicting changes • Misunderstood requirements • Missed capabilities • Larger integrations fail more
- 4. Integration surprises • Frequently surprised • That should have worked • Works on my machine • Why would anyone do that? • That’s a feature, not a bug • …
- 5. The Intersection Accident Solution Roundabouts have fewer accidents and less severe accidents
- 6. The Software Integration Solution • Integrate frequently • Combine changes often • Evaluate changes often • Detect issues early Merge Team 1 Team 2 Team 3 Merge Team 1 Team 2 Team 3 Merge Team 1 Team 2 Team 3 Merge Team 1 Team 2 Team 3 Merge Team 1 Team 2 Team 3
- 7. Implementation: Meet the Red Team • Goal: Deliver great software from a happy team • Integrate • Build • Test • Deliver • Find failures sooner, faster
- 8. Red Team Uses Jenkins Pipeline • Jenkins Continuous Integration • Open source • Full-featured • Scalable • Powerful • Easy to configure
- 9. Red Team Demo – Jenkins Pipeline • Red Team finds team level problems sooner • Conflicts discovered earlier • Missed requirements identified sooner • Integration is less painful because it is more frequent
- 10. Meet Ms. Green and Mrs. Black • Red team members • Ms. Green • Mrs. Black • Find problems sooner Merge Ms. Green Red Team Mrs. Black What if I could find problems before merge?
- 11. Check Proposed Changes • Detect conflicts before merge • Run tests before code review • Try Pipeline improvements Merge Ms. Green Red Team Mrs. Black What if I could find problems before merge?
- 12. Red Team Uses Jenkins Multi-branch Pipeline • Multi-branch Pipeline • Domain specific language (DSL) • Developer defined and modified • Stored with the source code • Job per branch • Jobs created automatically • Jobs destroyed automatically pipeline { agent any stages { stage('Build') { steps { sh 'make package' } } } }
- 13. Multi-branch Jenkins Pipeline Demo • Ms. Green and Mrs. Black finds problems sooner • Private merge of proposed change • Merge conflicts detected • Tests run before code review
- 14. The Unintended Variation Problem Surprises aren’t always happy This Photo by BlarSnow is licensed under CC BY-SA
- 15. Variation in Software Development • Good: Intentional Variation • Platform specific build • Multi-platform tests • Web browser tests • Low bandwidth tests This icon by Sean Martell is licensed under CC BY-SA
- 16. Variation in Software Development • Bad: Unintentional Variation • Wrong build tools • Outdated dependencies • Contaminated tests • Full file systems
- 17. Reducing Unintentional Variation with Docker • Docker Containers • Repeatable • Isolated file system • Isolated process - lightweight • Open source
- 18. Building with Docker – Apache Maven • Java 7, latest Maven 3 • docker run maven:3-jdk-7 mvn –v • Java 8, latest Maven 3 • docker run maven:3-jdk-8 mvn –v • Java 9, latest Maven 3 • docker run maven:3-jdk-9 mvn –v • Java 10, latest Maven 3 • docker run maven:3-jdk-10 mvn -v • Java 11, latest Maven 3 • docker run maven:3-jdk-11 mvn -v See https://hub.docker.com/_/maven/ for more command line details
- 19. Building with Docker – Gradle • Java 8, Gradle 4.7.0 • docker run gradle:4.7.0-jre8 gradle –-version • Java 9, Gradle 4.7.0 • docker run gradle:4.7.0-jre9 gradle –-version • Java 10, Gradle 4.7.0 • docker run gradle:4.7.0-jre10 gradle –-version See https://hub.docker.com/_/gradle/ for more command line details
- 20. Define Your Own Toolchain • Dockerfile specifies • Operating system (CentOS, Debian, Red Hat, SUSE, Ubuntu, …) • Packages • File system contents • Volume mount points (reuse, persistence across run, etc.)
- 21. Build with Docker • Repeatable builds in precisely defined environment • Specify exact tool versions in Dockerfile • Clean file system is easy • Examples • I need gcc 6.1 and go 1.8, they need gcc 4.3 and java 1.8 • Both can be satisfied by each compiling from a purpose-built docker image • I need specific library versions, they need different versions • Both satisfied by their own Dockerfile
- 22. Docker for Unit and Integration Tests Fewer environment surprises
- 23. Docker for Better Unit & Integration Testing • Docker provides more consistent runtime • Reduce differences between test environments • Examples • Unit tests run in Docker start from a clean state • Test environment cleanup is simpler
- 24. Specific Example – Jenkins Test Harness • Jenkins test harness uses a specific named temporary directory • Multiple users on same machine cannot use same directory • Docker file system isolation solves it
- 25. Docker for End to End Tests Rapid configuration of complex scenarios
- 26. Docker for Better End to End Testing • Docker automates complicated configuration • Repeatable test environment creation • Compose multi-component test configurations • Spend less time configuring test environments • Spend more time testing • Examples: • Jenkins LTS with plugins • https://github.com/MarkEWaite/docker-lfs/tree/lts-with-plugins
- 27. Docker for Bug Reports Repeatable environments in the bug report
- 28. Docker for Better Bug Reports • Simplify long sequences of pre-conditions in Dockerfile • Focus the report on specific bug details • Simplify setup for bug fixer • Some examples: • JENKINS-41948 – noSuchMethodError in scanning • JENKINS-42204 – NPE due to GitSCMSource change • JENKINS-46724 – Illegal reflective access from remoting
- 29. Docker & Jenkins Together • Jenkins for Pipeline as code • Docker for infrastructure as code
Editor's Notes
- #3: We’ve found through years of driving automobiles that accidents happen more frequently at intersections than on long straight stretches.
- #5: Tell the story of the wooden trivet. Wooden trivet looks like it should assemble in either direction. Legs appear consistently sized, finish is consistent, and it looks reasonable Inverted parts do not assemble cleanly, no matter how much they look like they will.
- #15: A birthday gift is an intentional surprise. We laugh and smile and enjoy the experience of being surprised by the contents of that beautifully wrapped package. The gift is a source of joy and fun
- #16: Intentional variation in software development happens all the time. We develop for multiple platforms. We test on multiple platforms. We develop for multiple browsers. We test our mobile apps in low bandwidth environments. Those can all be good variations in software development. They help us create great software.
- #17: Unfortunately, some variation can delay or derail software creation. When we build with the wrong tool, we may get the wrong results. When we use outdated dependencies, we may have bugs we did not expect. When unexpected files on a test system alter the behavior of the test, we may miss important bugs. When a build system or a test system runs out of space, we may waste time diagnosing irrelevant failures
- #18: Describe a brief history of containers, Solaris and BSD as jails, then Linux with greater ease and more capabilities.