Beyond Continuous Delivery

Editor's Notes

• #2: There are a lot of technology and process innovations that got me interested in Continuous Delivery, but they also had implications for me far beyond just CD. This presentation explores some of those ideas.I’ll start with some basic concepts rooted in the here and now, then build on top of those to farther out concepts. This sort of builds up to a far future “grand vision”, if you will, but I think it’s more important just to explore the concepts along the way.I have a lot of high-level stuff to get through so I’m not going to get too bogged down in details. Lots of this stuff hasn’t been done yet so the devil is in the details, but those could be long discussions themselves. I don’t want to discourage questions, but just want to keep it at a level of solvable problems.
• #3: These are concepts I assumeeveryone here is pretty familiar with.For the purposes of this discussion, I’ll be assuming cloud resources are cheap and unlimited. I’m going to push these assumptions to ridiculous limits, but I’m going to go with it for the sake of these thought experiments
• #4: As an alternative to monolithic applications, this is an example of a simple modular application wired together with a dependency management system. Each module can build and unit test independently and pulls in needed dependencies through the dependency management system.A more realistic example can look like this…
• #5: This is a moderately complex application at big retailer. There are more complicated ones.But let’s keep it simple…
• #6: …and stick to discussing this instead.So what can you do with dependency management and modular applications that you can’t do with a monolithic build?Chained builds- each module kicks off build of downstream modulesParallel building- instead of monolithic sequential build, build modules in parallelMinimal builds, impact zone- build only what’s been updatedMaven and Ivy for dependency management, Jenkins plugins for chained buildsTim Brown did some of this at GAP and I did all of this at my previous job.
• #7: Now with infrastructure as code, we can add infrastructure modules and include those in our dependency tree, ideally with tests. They can also build and test independently.So we have a base VM module, on top of that we add some basic IT setup (monitoring, security, etc.), then add some environment scripts. The isolation tests module here does the deploy and isolated tests like functional tests, acceptance tests, etc. This could be multiple modules, I just grouped them for simplicity.
• #8: Building on that, we can integrate with another app and run integration tests.
• #9: And even extend our dependency tree all the way to production.So I’m going to say a few possibly controversial things here.1. Point out the pipelines in the graph. Pipelines are abstractions of dependency graphs. Useful concept and I’ll keep using the term, but kind of bullshit. You can talk about pipelines of pipelines, but that’s pretty much just another way of saying dependencies. Can encourage monoliths.2. Everything is a unit.3. Building on that, all tests are unit tests for the right size of unit.
• #10: Next, I want to talk about semi-fluid dependencies and how they can keep every module up-to-date but always in a working state.Semi-fluid dependencies are a combination of static and fluid dependencies.Static dependencies are stable but hard to maintain manuallyFluid dependencies keep things up-to-date easily, but also break things easilyPerversely, fluid dependencies also make it easier for third parties to commit code to your project than your own developersAt Expedia, fluid dependencies were causing outages for the web team several times a week. With a team size of around 150 developers, that was $25-50K in lost productivity every time.Semi-fluid dependencies try to get the best of both worlds. Every dependency has both a static and (possibly) fluid dependency. Developers use the static dependencies. An automated system uses the fluid dependencies, runs tests, and updates the static dependency.So when a new version of common jar is published…
• #11: The automated system buildsA.jar with the new version, it passes, so the static dependency is updated. The build for B.jar also runs, but it doesn’t pass, so the static dependency remains. I’m assuming a Jenkins-like build here, so App.war also builds attempting to incorporate the new version of A.jar, but fails with a dependency version conflict.At this point, developers can still build and work on every module because they each have static dependencies on known good versions, though work on A.jar and B.jar are depending on different versions of common.jar and work on A.jar and common.jar won’t be incorporated into App.war. Most times, like 95%, new dependencies at Expedia were incorporated automatically, but the rest have to be resolved manually. This could be resolve two ways. Update B.jar to work with the new version of common.jar or publish a new version of common.jar that works with B.jar.
• #12: Here, we published a new version of common.jar with a fix. The A.jar build passes and updates its static dependency, B.jar does the same, and App.war updates the static version for both of its dependencies. I implemented something like this at Expedia, saving them a couple of million in lost productivity per year.There’s still some confusion and back and forth that has to go on to fix things, so I’ll talk more about eliminating the introduction of errors later, but first we need to talk about…
• #13: Cloneable pipelinesWith all of the pipeline infrastructure well-defined in source code, it should be trivial to create a copy of of the release pipeline. Clone the source code repository, as well, such as with git, and you have a personal pipeline that is an exact copy of the official pipeline. Commit to your repo, let your pipeline run, push to master when it succeeds. No more it works on my machine. And your code will be better tested.rPath does some of this for infrastructure already
• #14: With most pre-flight build concepts, you run just the module build and tests before the change is accepted for commit. Using a cloneable pipeline, we can accept only changes that work for the entire pipeline.Unlike before, when common.jar breaks the build for B.jar, this time it happens in a separate pipeline instance and the commit is never pushed into the trunk.
• #15: When a change is committed that does work all the way through the pipeline, then it is committed.This is something we are working towards using Electric Commander at GAP. It’s also been wanted in one form or another at 4 of the last 5 places I’ve worked. The fifth one was Shaw.
• #16: We’re getting closer to a pipeline that can never be “red”. I don’t really like the name “pristine trunks”, but I haven’t come up with better wording yet. Not a great diagram, but the black line is trunk and the brown and blue lines are pre-flight pipelines for different developers. The first brown pipeline runs and successfully commits. The blue pipeline runs with a later change and also successfully commits. While that pipeline was running, another brown pipeline runs and fails, so that change is not committed. Another blue pipeline also runs and commits. And the brown pipeline is corrected and eventually commits.There’s still some opportunity for redness here as testing is running while new commits are being made (a common problem fro developers). We need a way for ordered entry of changes.
• #17: Another not so great diagram. Here, I’m saying trunk is already at change n and we know it’s green. Change n+1 comes in and a pre-flight pipeline kicks off with that change.What happens when the next change comes in? We could wait for this pipeline to complete, but that won’t scale with many changes coming in and/or a long-running pipeline.
• #18: With cloneable pipelines, we can do better. So instead, we kick off a pipeline both with and without the in-flight change to start testing the n+2 change, a so-called quantum pipeline.
• #19: When the n+1 pipeline completes, we can “collapse the wave” and abort the unneeded pipeline that assumed the n+1 pipeline would fail.
• #20: And here’s a more complicated example with 3 changes in-flight at once. When n+1 completes, abort half of the pipelines. When n+2 fails, abort the associated n+3 pipeline. N+3 completes and successfully commits.We now have only changes that are successful within the entire pipeline being committed to trunk. It should be impossible for trunk to ever be “red”, at least within the length of the cloned pipeline. What’s more, these successful pipeline are exact copies of the trunk pipeline, so the artifacts from these builds can be used directly without being rebuilt in a trunk pipeline.
• #21: With cloneable pipelines, I can recreate pipelines for dependent pipelines and prevent breaks to downstream projects.Say I’m log4j and want to make sure my changes don’t break tomcat. Log4j could recreate the tomcat pipeline as part of its pipeline and reject changes based results including the tomcat pipeline.Not a lot to say here, but I think there’s opportunity to communicate better across these types of boundaries with some of these concepts.
• #22: With so many pipelines running, won’t this take a lot of time? One way to cut down on the build times would be to have a central build service and use the “power of the swarm”. The bottom build of common.jar refers to the build service and builds normally, returning the built jar. The middle build runs the same build of common.jar, but this time the build service returns the previously built jar as the code and dependencies are exactly the same. The top build runs with a change of one new file and the build service compiles the one file, adds it to the previously built jar to make a new jar artifact, then returns that jar to the running build.Basically, all those pipelines are doing very similar work and could benefit from some level of sharing their work. A build service could do detailed analysis of individual compilable units and their dependencies and optimize the required work. And the build service infrastructure should be well-defined and reproducible for developers to support ‘airplane mode’.
• #23: Next, I want to talk about extreme integration. Not a great term, but it’s an extension of extreme continuous integration. If you aren’t familiar with extreme continuous integration, it’s a concept where as you are programming, a background process is continuously running your unit tests as you pause typing so you get instant feedback without you even needing to explicitly run them.Similarly, extreme integration applies the same automatic feedback on a pipeline scale. - Continuously run tests in personal pipeline- Continuously integrate from trunk into integration pipeline- Auto-commit to trunk when all is greenWhen all of this works and multiple developers are working together in trunk, it’s almost as if the code is a Google WaveBreak this up into multiple slides laterPersonal pipelineIntegrating into extreme branchIntegrating into trunk
• #24: Why develop locally at all? Why not have a cloud IDE?Everyone works on modules/code/infrastructure remotely through a web-based front end.Here’s a really simple mock-up of what I mean.
• #25: Create project, personal pipeline red
• #26: Create test-driven infrastructure test, personal pipeline red
• #27: Create platform, personal pipeline redAdd tomcat package, personal pipeline green, auto-merged to project pipeline
• #28: Create jar projectCreate classEnter test, personal pipeline red
• #29: Enter codeAdd dependency with dynamic range, personal pipeline green, auto-merge to project pipeline
• #30: Create war projectCreate cucumber test, personal pipeline red
• #31: Create JSP fileAdd dependency on jar (automatically adds dynamic range), personal pipeline green, auto-merge to project
• #32: Add test-driven infrastructure test for war, personal pipeline red
• #33: Create deployment script, personal pipeline green, auto-merge to project pipelineUse it to wire dependencies together.It has automatic setup and provisioning for all the environments and pipelines needed.It has continuous delivery built-in; not something users even need to think about.Some object to working remotely, but I think these are solvable problems. Such as having push button access to hotspots and saving VMs or even whole environments for investigating problems. Also, when I say cloud IDE, this doesn’t preclude the system actually being based locally.Kodingen.comis doing a little bit of this around code, but I don’t know too much about it and think there’s a lot more to be done as far as controlling infrastructure and pipelines with something like this.
• #34: These topics are a bit tacked on, but they are somewhat related things I have been thinking about.Somewhat like public bug tracking, customers could write public BDD tests. Developers could flesh out the steps and a personal pipeline would let the reporter and product team know when the test is passing.Compare functionality across products by having public BDD tests run against multiple products.Public development- give public access to code, but not immune system
• #35: There are a lot of technology and process innovations that got me interested in Continuous Delivery, but they also had implications for me far beyond just CD. This presentation explores some of those ideas.I’ll start with some basic concepts rooted in the here and now, then build on top of those to farther out concepts. This sort of builds up to a far future “grand vision”, if you will, but I think it’s more important just to explore the concepts along the way.I have a lot of high-level stuff to get through so I’m not going to get too bogged down in details. Lots of this stuff hasn’t been done yet so the devil is in the details, but those could be long discussions themselves. I don’t want to discourage questions, but just want to keep it at a level of solvable problems.