ContainerSched 2017 "Continuous Delivery with Containers: The Good, the Bad, and the Ugly"
Download as PPTX, PDF1 like631 views
The document discusses continuous delivery with containers. It covers the impact of containers on continuous delivery pipelines, including that container images become the single binary in the pipeline. It also discusses challenges like adding important metadata to container images and validating container constraints. The document provides recommendations for aspects of continuous delivery like developing in container-based environments, testing individual components and integration, and deploying container-based applications.
ContainerSched 2017 "Continuous Delivery with Containers: The Good, the Bad, and the Ugly"
- 1. Continuous Delivery with Containers: The Good, the Bad, and the Ugly Daniel Bryant @danielbryantuk
- 2. Containers: Expectations versus reality 29/09/2017 @danielbryantuk “DevOps”
- 3. Setting the scene… • Continuous delivery is a large topic • No business focus today (value stream etc) • PaaS and Serverless are super interesting… • But I’m assuming you’re all-in on containers • Focusing today on the process and tooling • No live coding today • Mini-book contains more details (thanks nginx!) 29/09/2017 @danielbryantuk bit.ly/2jWDSF7
- 4. TL;DR – Containers and CD • Container image becomes the build pipeline ‘single binary’ • Adding metadata to containers images is vital, but challenging • Must validate container constraints (NFRs) • Cultivate container ‘mechanical sympathy’ 29/09/2017 @danielbryantuk
- 5. @danielbryantuk • Independent Technical Consultant, CTO at SpectoLabs • Architecture, DevOps, Java, microservices, cloud, containers • Continuous Delivery (CI/CD) advocate • Leading change through technology and teams 29/09/2017 @danielbryantuk
- 7. Continuous Delivery • Produce valuable and robust software in short cycles • Optimising for feedback and learning • Not (necessarily) Continuous Deployment 29/09/2017 @danielbryantuk
- 8. Creation of a build pipeline is mandatory for continuous delivery 29/09/2017 @danielbryantuk
- 10. The Impact of containers on CD 29/09/2017 @danielbryantuk
- 11. Container technology (and CD) • OS-level virtualisation • cgroups, namespaces, rootfs • Package and execute software • Container image == ‘single binary’ 29/09/2017 @danielbryantuk
- 14. Creating a pipeline for containers 29/09/2017 @danielbryantuk
- 16. Make your dev environment like production • Develop locally or copy/code in container • Must build/test containers locally • Perform (at least) happy path tests 29/09/2017 @danielbryantuk
- 17. Quick Aside: Running *entire* system locally 29/09/2017 @danielbryantuk https://news.ycombinator.com/item?id=13960107 https://opencredo.com/working-locally-with-microservices/ https://www.datawire.io/telepresence/ | https://hoverfly.io/
- 18. Make your dev environment like production • Develop locally or copy/code in container • Must build/test containers locally • Perform (at least) happy path tests • Use identical base images from production • With same configuration 29/09/2017 @danielbryantuk
- 19. Lesson learned: Dockerfile content is super important • OS choice • Configuration • Build artifacts • Exposing ports • Java • JDK vs JRE and Oracle vs OpenJDK? • Golang • Statically compiled binary in scratch? • Python • Virtualenv? 29/09/2017 @danielbryantuk
- 20. Please talk to the sysadmin people: Their operational knowledge is invaluable 29/09/2017 @danielbryantuk
- 21. Different test and prod containers? • Create “test” version of container • Full OS (e.g. Ubuntu) • Test tools and data • Easy to see app/configuration drift • Use test sidecar containers instead • ONTEST proposal by Alexi Ledenev 29/09/2017 @danielbryantuk http://blog.terranillius.com/post/docker_testing/
- 22. Docker multi-stage builds 29/09/2017 @danielbryantuk http://blog.alexellis.io/mutli-stage-docker-builds/ https://github.com/moby/moby/pull/31257 https://github.com/moby/moby/pull/32063
- 24. Building images with Jenkins • My report covers this • Build as usual… • Build Docker Image • Cloudbees Docker Build and Publish Plugin • Push image to registry 29/09/2017 @danielbryantuk
- 25. Storing in an image registry (DockerHub) 29/09/2017 @danielbryantuk
- 26. Metadata – Beware of “latest” Docker Tag • Beware of the ‘latest’ Docker tag • “Latest” simply means • the last build/tag that ran without a specific tag/version specified • Ignore “latest” tag • Version your tags, every time • danielbryantuk/test:2.4.1 29/09/2017 @danielbryantuk
- 27. Lesson learned: Metadata is valuable • Application metadata • Version / GIT SHA • Build metadata • Build date • Image name • Vendor • Quality metadata • QA control, signed binaries, ephemeral support • Security profiles (AppArmor), Security audited etc 29/09/2017 @danielbryantuk
- 28. Metadata - Adding Labels at build time • Docker Labels • Add key/value data to image 29/09/2017 @danielbryantuk
- 29. Metadata - Adding Labels at build time • Microscaling Systems’ Makefile • Labelling automated builds on DockerHub (h/t Ross Fairbanks) • Create file ‘/hooks/build’ • label-schema.org • microbadger.com 29/09/2017 @danielbryantuk
- 30. Metadata - Adding Labels at runtime 29/09/2017 @danielbryantuk $ docker run -d --label uk.co.danielbryant.lbname=frontdoor nginx • Can ’docker commit’, but creates new image • Not possible to update running container • Docker Proposal: Update labels #21721
- 31. Liz Rice (and Aqua) to the rescue! 29/09/2017 @danielbryantuk github.com/aquasecurity/manifesto
- 32. External registry with metadata support 29/09/2017 @danielbryantuk
- 35. Testing: Jenkins Pipeline (as code) 29/09/2017 @danielbryantuk
- 37. Testing individual containers 29/09/2017 @danielbryantuk
- 39. Introducing Docker Compose 29/09/2017 @danielbryantuk
- 40. Docker Compose & Jenkins Pipeline 29/09/2017 @danielbryantuk
- 41. Ephemeral Kubernetes Clusters • Kubernaut (WIP) • Manages a pool of clusters • ”Claim” a fresh cluster • Use Helm to install dependencies 29/09/2017 @danielbryantuk
- 42. Testing NFRs in the build pipeline • Performance and Load testing • Gatling / jmeter • Flood.io • Security testing • Findsecbugs / OWASP Dependency check • Bdd-security (OWASP ZAP) / Arachni • Gauntlt / Serverspec • Docker Bench for Security / CoreOS Clair 29/09/2017 @danielbryantuk
- 43. Delaying NFRs to the ‘Last Responsible Moment’ • Newsflash! • Sometimes the last responsible moment is up-front! • Containers / microservices don’t make this easier • Sometimes more difficult… 29/09/2017 @danielbryantuk
- 44. Mechanical sympathy: Docker and Java • Watch for JVM cgroup/taskset awareness • getAvailableProcessors() may incorrectly report the number of cpus in Docker (JDK-8140793) • Runtime.availableProcessors() ignores Linux taskset command (JDK-6515172) • Default fork/join thread pool sizes (and others) is based from host CPU count • Set container memory appropriately • JVM requirements = Heap size (Xmx) + Metaspace + JVM overhead • Account for native thread requirements e.g. thread stack size (Xss) • Entropy • Host entropy can soon be exhausted by crypto operations 29/09/2017 @danielbryantuk 44
- 46. Observability is core to continuous delivery 29/09/2017 @danielbryantuk www.infoq.com/articles/monitoring-containers-at-scale
- 47. Containers are not a silver bullet 29/09/2017 @danielbryantuk
- 48. Moving to containers: Going all-in? 29/09/2017 @danielbryantuk OR
- 49. Should I build my own container platform? Probably not (Unless you are Google, AWS or IBM) Whatever you decide… push it through a pipeline ASAP! 29/09/2017 @danielbryantuk
- 50. Using containers does not obviate the need for good architectural practices 29/09/2017 @danielbryantuk
- 53. In summary • Continuous delivery is vitally important in modern architectures/ops • Container images must be the (single) source of truth within pipeline • And metadata added as appropriate… • Mechanical sympathy is important (assert properties in the pipeline) • Not all developers are operationally aware • The tooling is now becoming stable/mature • We need to re-apply existing CD practices with new technologies/tooling 29/09/2017 @danielbryantuk
- 55. Thanks for listening • Any questions? • Feel free to contact me • @danielbryantuk • daniel.bryant@tai-dev.co.uk 29/09/2017 @danielbryantuk bit.ly/2jWDSF7 Coming soon!
- 56. Bonus slides (for extra context) 29/09/2017 @danielbryantuk
- 57. Containerise an existing (monolithic) app? • For • We know the monolith well • Allows homogenization of the pipeline and deployment platform • Can be a demonstrable win for tech and the business • Against • Can be difficult (100+ line scripts) • Often not designed for operation within containers, nor cloud native • Putting lipstick on a pig? 29/09/2017 @danielbryantuk
- 58. Key lessons learned • Conduct an architectural review • Architecture for Developers, by Simon Brown • Architecture Interview, by Susan Fowler • Look for data ingress/egress • File system access • Support resource constraints/transience • Optimise for quick startup and shutdown • Evaluate approach to concurrency • Store configuration (secrets) remotely 29/09/2017 @danielbryantuk
- 59. New design patterns 29/09/2017 @danielbryantuk bit.ly/2efe0TP
- 60. Microservices… Containers and microservices are complementary Testing and deployment change 29/09/2017 @danielbryantuk https://specto.io/blog/recipe-for-designing-building-testing-microservices.html