Continuous Delivery with NetflixOSS
5 likes1,418 views
The document provides an overview of Netflix's approach to continuous delivery using their open source tools. It discusses how Netflix builds immutable infrastructure by baking software packages into pre-configured server images. It also describes how their build system tools like Gradle and Nebula plugins help standardize builds at scale. Finally, it outlines how tools like Eureka, Ribbon, and Asgard help enable ongoing deployment and management of cloud resources through concepts like service discovery and application clusters.
Continuous Delivery with NetflixOSS
- 1. Continuous Delivery with Netflix OSS Dan Woods
- 2. /danveloper Senior Software Engineer: Delivery Engineering danielpwoods@gmail.com Learning Ratpack
- 3. Overview of Netflix OSS • Netflix encourages talking to the world about how we’re solving problems ! • We solve a ton of problems that companies both small and large are faced with ! • Shoot to open source as much as possible
- 4. Overview of Netflix OSS • Netflix is a large consumer of cloud offerings — mostly from AWS ! • We’ve done a ton of work over the years to lift the infrastructure entirely to the cloud ! • Pioneered running at scale on Amazon AWS
- 5. Overview of Netflix OSS • Developed a massive tool suite to operationalize running in the cloud at scale ! • Teams need to be able to quickly get code running in the cloud ! • Teams needs to quickly be able to see metrics and performance
- 6. Overview of Netflix OSS Links: ! http://techblog.netflix.com/ ! http://github.com/netflix ! http://netflix.github.io
- 7. Continuous Delivery Big Picture:! ! What Does Continuous Delivery Mean At Netflix?
- 8. Continuous Delivery Big Picture:! ! • Immutable Infrastructure ! • Tooling the Build System ! • Ongoing and Continuous Deployment
- 9. Immutable Infrastructure • Designing a server to become your unit of deployment ! • “Bake” the software into a “pre-cooked” (known-good configuration) image ! • Allows you to test and certify a server image for distribution ! • Walk that server through the phases of test, qa, and finally to prod
- 10. Immutable Infrastructure • Builds must be designed in a way that produces an os-package ! • This allows the build to control the manner in which the server image will be created ! • Specify OS-level dependencies (Java, Python, etc) ! • Get all the benefits of a version controlled configuration
- 11. Tooling the Build System • Hundreds, sometimes thousand, of builds that run every day at Netflix ! • Builds need to fit into a somewhat conferment structure to garner the support of the tooling ! • A polyglot stack adds a ton of complexity to designing the tooling for the build system ! • Teams are free to use whatever language or framework or stack that they want, and we need to do our best to have a handle on the permutations
- 12. Tooling the Build System • The JVM is the predominant code platform at Netflix ! • Many different languages on the JVM, including: JavaScript, Scala, Groovy, Clojure, Ruby, Python ! • The “runner up” runtime is NodeJS ! • Lots of new JavaScript stuff starting to come out, starting to design scalable tooling around JS
- 13. Tooling the Build System • Netflix has adopted Gradle as its build platform ! • Gradle is a JVM-based build system that is capable of building JVM and non-JVM projects ! • Support for dynamically and programmatically designing builds (loads of flexibility) ! • Great open source community, tons of support from Gradlware
- 14. Tooling the Build System • Can build plugins for Gradle in Groovy (ahh soo nice :-)) ! • Plugins are designed to make it appealing for teams to conform to the tooling infrastructure ! • Custom internal Gradle wrapper applies common conventions and applies hacks that would be unmanageable at scale ! • The goal of all this is to make teams want to use the build tooling, so that we can operationalize and manage it for scale
- 15. Continuous Deployment • Continuous Delivery at Netflix speaks to more than just staging code for deployment ! • The Continuous Delivery story is a follow through, from source to production ! • Continuous Deployment is an integral part of that process (it means the code running in the cloud!) ! • Hands down this is the trickiest and most-fragile part of the whole process…
- 16. Continuous Deployment • By this point in the workflow, the code has already been built and baked… ! • We have an immutable server image, and we’re ready to ship it off to the cloud… ! • The complexity is here: “ship it off to the cloud” is an inherently asynchronous process… ! • There are many failure points.!
- 17. Continuous Deployment What constitutes a successful deployment? ! • Every application has a different definition of “success” ! • Need to provide tooling so that the process is able to identify the vectors of success !
- 18. Continuous Deployment What constitutes a successful deployment? ! • Amazon telling us the server has deployed is basically the equivalent of them saying they pressed the power button ! • Need to consider a successful deployment in terms of “this server is ready to start taking traffic” ! !
- 19. Continuous Deployment What constitutes a successful deployment? ! • “Ready to start taking traffic” means different things to different applications: ! • Tomcat has started, and the app is listening? ! • Tomcat has started, app is listening, caches are primed? ! • Tomcat has started, app is listening, and the server group is in some designated traffic pool (canary)? ! ! !
- 20. Continuous Deployment • Service discovery becomes a very big part of understanding the health of an app ! • Gives the app the responsibility to inform the tool as to its traffic-taking-readiness ! • It would be difficult for the tool to reach out to every instance to ask it for its health, better to have the instance tell us ! • The tooling now only need to query two places: Amazon and the Service Registry
- 21. Continuous Deployment • Teams can choose if “Discovery” health should be incorporated into their continuous deployment workflow ! • This may not be necessary; for strictly IPC stack apps, it’s ok for them to be “up” and to let the IPC client (Ribbon) determine to which instance traffic is routed
- 22. Continuous Deployment What do we do after success?! ! • Once the new version of code is deployed, now what? ! • Netflix lumps packages of software into a “cluster”, within which different versions may run ! • For rapid rollback, we need to keep the ancestor server group around, but take it out of traffic rotation
- 23. Continuous Deployment What do we do after success?! ! • Put the ancestral server group into a “disabled” state ! • Inform the service registry that the instances within this group are no longer accepting traffic ! • Most consuming apps will use the service registry to find their endpoint, so this is sufficient ! • For those that use DNS and go through a load balancer, we remove the instances from associated load balancers as well
- 24. Continuous Deployment Why not just update the existing config and roll the servers (rolling push)?! ! • Rolling push is a bad, bad thing ! • While new instances are launching against a new image, ancestral instances still exist ! • Can leave the server group in a half-done state, which can yield very weird results ! • Tooling is built around the server group being the management target
- 25. Continuous Deployment Incubating Deployment Strategies…! ! • Phased canary • 25%, 50%, 75%, 100% ! • Global push • Deployment windows to different regions ! • Highlander • Don’t keep the ancestor server group around • This is good for test environments that don’t need rollback
- 26. Continuous Deployment Continuous Delivery Tooling! ! • Many CD tools are available today from NetflixOSS! ! • The puzzle pieces are there for the entire problem domain ! • Tooling for build system packaging, baking immutable infrastructure, service discovery, continuous deployment, and cluster management
- 27. Build System Tooling Nebula Gradle Plugins! ! • Nebula (like, “space clouds”) is a collection of Gradle plugins to assist in the continuous delivery workflow ! • Often two parts: Nebula and Gradle — The “Gradle” part is just a Gradle plugin, and you’re on your own to configure it; the “Nebula” part is an opinionated veneer ! • Tons of great plugins, extensive documentation, many, many, many available videos and presentations on Nebula
- 28. Build System Tooling Nebula OS Package Plugins! ! • The Gradle Side • Provides mechanism for producing Debian and RPM artifacts • Very straight-forward integration that uses Gradle’s well-known CopySpec for getting files into an OS structure • Nice DSL for describing OS-level dependencies ! • The Nebula Side • Derives configuration in a “best fit” kind of way • Provides integration with Gradle’s application plugin to package a runnable distribution into an OS artifact • Provides ability to produce an OS daemon for your service ! https://github.com/nebula-plugins/nebula-ospackage-plugin
- 30. The Bakery Baking a Server Image! ! • Aminator • Provides easy creation of package-specific AMIs • Attaches a “Base Image” volume, installs your software package • Takes a snapshot of the volume, resulting in an AMI • This AMI is the immutable infrastructure • AMI will act as our unit of deployment going forward ! ! https://github.com/netflix/aminator
- 31. Service Discovery Service Registry for Apps! ! • Eureka • Applications can register their own health ! • Integrates tightly with Ribbon to provide inter-app service discovery, load balancing, and fault tolerance ! • Able to be leveraged during the continuous deployment process to inform as to successful deployments ! ! https://github.com/netflix/eureka https://github.com/netflix/ribbon
- 32. Continuous Deployment and Cluster Management Managing Deployments! ! • Asgard • Provides a UI for managing AWS cloud resources • RESTful API for consumers to be able to script against • Decorates AWS with concepts that are relevant to Netflix’s continuous delivery infrastructure • This includes the concept of applications and clusters, which is something that AWS does not have • Standalone, runnable JAR or WAR deployment options ! ! https://github.com/netflix/asgard
- 33. Continuous Deployment and Cluster Management
- 34. Some Harsh Realities… • All of this stuff is difficult to get up-and-running ! • Every tool makes assumptions about account structure, available resources, naming conventions, etc ! • Non-native concepts, like applications and clusters, are difficult to understand from an outsider’s perspective ! • Cost-to-benefit may be low if you’re not adopting the entire stack
- 35. Getting better… • Many initiatives underway currently to engage the open source community more directly ! • The goal is to make the barrier for entry very low on getting up-and-running with NetflixOSS ! • Andrew Spyker (@aspyker) is leading the charge for making NetflixOSS plug-and-play… ! • Although, not very much (right now) speaks directly to gluing tools together for continuous delivery
- 36. Some Resources • Zero to Cloud: • http://www.oscon.com/oscon2014/public/schedule/detail/34252 • Walks you through a document that shows how to setup your AWS account • Shows you how to leverage CloudFormation to configure a NetflixOSS runtime ! • Zero to Docker: • http://techblog.netflix.com/2014/11/zerotodocker-easy-way-to-evaluate. html • Pre-built Docker images for NetflixOSS components • Provides a quick way to get up-and-running • Not for production use; not in-use at Netflix
- 37. Trying to make this easy on you… Introducing the Zero to Cloud Gradle Plugin! ! https://github.com/Netflix-Skunkworks/zerotocloud-gradle ! • “Netflix Skunkworks”, so not officially NetflixOSS at this point ! • A single command can initialize a continuous delivery infrastructure built on NetflixOSS technologies ! • Plugin can be utilizes by builds to be the “glue” between the OS packaging, the Bakery, and Asgard