SATURN 2018 "Continuous Delivery with Containers" Extended 90 version
Download as PPTX, PDF1 like514 views
The document discusses the integration of containers in continuous delivery (CD), highlighting the importance of using container images as the central component of the build pipeline. It emphasizes the need for metadata management, the significance of architecture practices, and the balance of stability and speed for business success. Ultimately, it advocates reapplying established CD methodologies with modern container technologies to improve processes.
SATURN 2018 "Continuous Delivery with Containers" Extended 90 version
- 1. Continuous Delivery with Containers: The Good, the Bad, and the Ugly Daniel Bryant @danielbryantuk
- 2. Containers: Expectations versus reality 09/05/2018 @danielbryantuk “DevOps”
- 3. @danielbryantuk • Independent Technical Consultant, Product Architect at Datawire • Architecture, DevOps, Java, microservices, cloud, containers • Continuous Delivery (CI/CD) advocate • Leading change through technology and teams 09/05/2018 @danielbryantuk
- 4. Setting the scene… • Continuous delivery is a large topic • No business focus today (value stream etc) • PaaS and Serverless are super interesting… • But I’m assuming you’re all-in on containers • Focusing today on the process and tooling • No live coding today • Mini-book contains more details (thanks nginx!) 09/05/2018 @danielbryantuk bit.ly/2jWDSF7
- 5. TL;DR – Containers and CD • Container image becomes the build pipeline ‘single binary’ • Adding metadata to containers images is vital, but challenging • Must validate container constraints on system quality attributes (NFRs) 09/05/2018 @danielbryantuk
- 6. Continuous Delivery 101 09/05/2018 @danielbryantuk
- 7. Continuous Delivery • Produce valuable and robust software in short cycles • Optimising for feedback and learning • Not (necessarily) Continuous Deployment 09/05/2018 @danielbryantuk
- 8. Velocity (with stability) is key to business success “Continuous delivery is achieved when stability and speed can satisfy business demand. Discontinuous delivery occurs when stability and speed are insufficient.” - Steve Smith (@SteveSmithCD) 09/05/2018 @danielbryantuk
- 9. Creation of a build pipeline is mandatory for continuous delivery 09/05/2018 @danielbryantuk
- 10. 09/05/2018 @danielbryantuk Feedback: - Was our initial hypothesis proven? - How can we improve business, architecture and ops?
- 14. 09/05/2018 @danielbryantuk Feedback: - Was our initial hypothesis proven? - How can we improve business, architecture and ops?
- 15. The impact of containers on CD 09/05/2018 @danielbryantuk
- 16. Container technology (and CD) • OS-level virtualisation • cgroups, namespaces, rootfs • Share the OS kernel • Package and execute software • Container image == ‘single binary’ 09/05/2018 @danielbryantuk
- 19. Should I build my own container platform? Probably not (Unless you are Google, AWS or IBM) Whatever you decide… push it through a pipeline ASAP! 09/05/2018 @danielbryantuk
- 20. But what about microservices? 09/05/2018 @danielbryantuk
- 21. 09/05/2018 @danielbryantuk Independent service deployment https://www.slideshare.net/dbryant_uk/deliveragile-2018-continuous- delivery-patterns-for-modern-architectures
- 24. Make your dev environment like production • Develop locally or copy/code in container • Must build/test containers locally • Perform (at least) happy path tests • Use identical base images from production • With same configuration 09/05/2018 @danielbryantuk
- 25. Quick digression: Working remotely, locally… 09/05/2018 @danielbryantuk https://opencredo.com/working-locally-with-microservices/ https://www.telepresence.io/
- 26. Make your dev environment like production • Develop locally or copy/code in container • Must build/test containers locally • Perform (at least) happy path tests • Use identical base images from production • With same configuration 09/05/2018 @danielbryantuk
- 27. Lesson learned: Dockerfile content is super important • OS choice (distroless?) • Configuration • Build artifacts • Exposing ports, user • Java • JDK vs JRE and Oracle vs OpenJDK? • Golang • Statically compiled binary in scratch? • Python • Virtualenv? 09/05/2018 @danielbryantuk
- 28. Please talk to the sysadmin people: Their operational knowledge is invaluable 09/05/2018 @danielbryantuk
- 29. Different test and prod containers? • Create “test” version of container • Full OS (e.g. Ubuntu) • Test tools and data • Easy to see app/configuration drift • Use test sidecar containers instead • ONTEST proposal by Alexi Ledenev 09/05/2018 @danielbryantuk http://blog.terranillius.com/post/docker_testing/
- 30. Docker multi-stage builds 09/05/2018 @danielbryantuk https://docs.docker.com/develop/develop-images/multistage-build/ https://github.com/GoogleContainerTools/distroless
- 33. Building images with Jenkins • My report covers this • Build as usual… • Build Docker Image • Cloudbees Docker Build and Publish Plugin • Push image to registry 09/05/2018 @danielbryantuk
- 34. Building in the Cluster? 09/05/2018 @danielbryantuk https://github.com/GoogleContainerTools/kaniko https://www.infoq.com/news/2018/03/jenkins-x-kubernetes
- 36. Building Artifacts: Metadata 09/05/2018 @danielbryantuk
- 37. Lesson learned: Metadata is valuable • Application metadata • Version / GIT SHA • Build metadata • Build date • Image name • Vendor • Quality metadata • QA control, signed binaries, ephemeral support • Security profiles (AppArmor), Security audited etc 09/05/2018 @danielbryantuk
- 38. Metadata – Beware of “latest” Docker Tag • Beware of the ‘latest’ Docker tag • “Latest” simply means • the last build/tag that ran without a specific tag/version specified • Ignore “latest” tag • Version your tags, every time • danielbryantuk/test:2.4.1 09/05/2018 @danielbryantuk
- 39. Metadata - Adding Labels at build time • Docker Labels • Add key/value data to image 09/05/2018 @danielbryantuk
- 40. Metadata - Adding Labels at build time • Microscaling Systems’ Makefile • Labelling automated builds on DockerHub (h/t Ross Fairbanks) • Create file ‘/hooks/build’ • label-schema.org • microbadger.com 09/05/2018 @danielbryantuk
- 41. Metadata - Adding Labels at runtime 09/05/2018 @danielbryantuk $ docker run -d --label uk.co.danielbryant.lbname=frontdoor nginx • Can ’docker commit’, but creates new image • Not possible to update running container • Docker Proposal: Update labels #21721
- 42. External registry with metadata support 09/05/2018 @danielbryantuk
- 43. New Solution: Grafeas + Kritis 09/05/2018 @danielbryantuk https://github.com/grafeas/grafeas https://www.infoq.com/news/2018/05/grafeas-kritis-security
- 44. Grafeas Metadata “kinds” (Schema) 09/05/2018 @danielbryantuk
- 46. Grafeas Metadata “kinds” (Schema) 09/05/2018 @danielbryantuk
- 50. Validating Container Structure 09/05/2018 @danielbryantuk https://github.com/GoogleContainerTools/container-structure-test https://github.com/GoogleContainerTools/container-diff
- 51. Quality Assurance: Functional 09/05/2018 @danielbryantuk
- 52. Testing (in Production??) 09/05/2018 @danielbryantuk martinfowler.com/bliki/TestPyramid.html https://medium.com/@copyconstruct/testing-microservices-the-sane-way-9bb31d158c16
- 54. Testing: Jenkins Pipeline (as code) 09/05/2018 @danielbryantuk
- 56. Testing individual containers 09/05/2018 @danielbryantuk
- 58. Introducing Docker Compose 09/05/2018 @danielbryantuk
- 59. Quality Assurance: Nonfunctional 09/05/2018 @danielbryantuk
- 60. Testing NFRs in the build pipeline • Architecture • Performance and Load testing • Gatling / jmeter / Flood.io • Security testing • Findsecbugs / OWASP Dependency check • Bdd-security (OWASP ZAP) / Arachni • Gauntlt / Serverspec • Docker Bench for Security / CoreOS Clair 09/05/2018 @danielbryantuk
- 62. Quick digression: Testing Architecture 09/05/2018 @danielbryantuk https://www.archunit.org/
- 64. Mechanical sympathy: Docker and Java • Watch for JVM cgroup/taskset awareness (with JDK <= 8) • getAvailableProcessors() may incorrectly report the number of cpus in Docker (JDK-8140793) • Runtime.availableProcessors() ignores Linux taskset command (JDK-6515172) • Default fork/join thread pool sizes (and others) is based from host CPU count • Set container memory appropriately • JVM requirements = Heap size (Xmx) + Metaspace + JVM overhead • Account for native thread requirements e.g. thread stack size (Xss) • Entropy • Host entropy can soon be exhausted by crypto operations and /dev/random blocks • -Djava.security.egd=file:/dev/./urandom (notes on this) 09/05/2018 @danielbryantuk 64
- 65. Security Visibility: Basic Code Scanning 09/05/2018 @danielbryantuk
- 67. Static Image Scanning 09/05/2018 @danielbryantuk github.com/arminc/clair-scanner
- 68. Verifying Container/Platform Security 09/05/2018 @danielbryantuk https://github.com/docker/docker-bench-security https://github.com/aquasecurity/kube-bench https://github.com/Shopify/kubeaudit
- 69. Delaying NFRs to the ‘Last Responsible Moment’ Newsflash! Sometimes the last responsible moment is up-front Modern platforms/architectures don’t necessarily make this easier 09/05/2018 @danielbryantuk
- 70. Quality Assurance: Post-Deploy 09/05/2018 @danielbryantuk
- 71. Observability is core to continuous delivery 09/05/2018 @danielbryantuk www.infoq.com/articles/monitoring-containers-at-scale
- 72. Canarying and Synthetic Txns 09/05/2018 @danielbryantuk https://blog.getambassador.io/canary-deployments-a-b-testing-and-microservices-with-ambassador-f104d0458736 https://martinfowler.com/bliki/SyntheticMonitoring.html
- 73. Choas Engineering/Testing 09/05/2018 @danielbryantuk https://github.com/asobti/kube-monkey https://www.infoq.com/news/2018/01/powerfulseal-chaos-kubernetes
- 74. When bad things happen, people are always involved… 09/05/2018 @danielbryantuk | @oakinger
- 76. Mikey Dickerson's Hierarchy of Reliability 09/05/2018 @danielbryantuk www.infoq.com/news/2015/06/too-big-to-fail
- 77. Containers are not a silver bullet 09/05/2018 @danielbryantuk
- 78. Moving to containers: Going all-in? 09/05/2018 @danielbryantuk OR
- 79. Containerise an existing (monolithic) app? • For • We know the monolith well • Allows homogenization of the pipeline and deployment platform • Can be a demonstrable win for tech and the business • Against • Can be difficult (100+ line scripts) • Often not designed for operation within containers, nor cloud native • Putting lipstick on a pig? 09/05/2018 @danielbryantuk
- 80. Key lessons learned • Conduct an architectural review • Architecture for Developers, by Simon Brown • Architecture Interview, by Susan Fowler • Look for data ingress/egress • File system access • Support resource constraints/transience • Optimise for quick startup and shutdown • Evaluate approach to concurrency • Store configuration (secrets) remotely 09/05/2018 @danielbryantuk
- 81. Using containers does not get rid of the need for good architectural practices 09/05/2018 @danielbryantuk
- 84. In summary • Continuous delivery is vitally important in modern architectures/ops • Container images must be the (single) source of truth within pipeline • And metadata added as appropriate… • Mechanical sympathy is important (assert properties in the pipeline) • Not all developers are operationally aware • The tooling is now becoming stable/mature • We need to re-apply existing CD practices with new technologies/tooling 09/05/2018 @danielbryantuk
- 85. Thanks for listening… Twitter: @danielbryantuk Email: daniel.bryant@tai-dev.co.uk Writing: https://www.infoq.com/profile/Daniel-Bryant Talks: https://www.youtube.com/playlist?list=PLoVYf_0qOYNeBmrpjuBOOAqJnQb3QAEtM 09/05/2018 @danielbryantuk bit.ly/2jWDSF7 Coming soon!