DevOps 101

Editor's Notes

• #3: Who are you? Manager? Developer? Sysadmin? Something else? Are you interested in the answer to these questions?
• #4: Hi, I’m Ernest. I am not a mall cop. I’ve been in IT since 1993. I’ve been a developer, a sysadmin, an architect, an IT manager, a development manager, a DevOps manager, and a product manager. I’ve worked in huge IT organizations and small SaaS startups. I’ve led three DevOps transformations so far, and have seen compelling results each time.
• #5: I asked Google what DevOps was. Apparently it’s pretty complicated. (And utterly lacking in graphic design capabilities.) I’m not going to tell you anything someone else hasn’t said in this presentation, but what I am going to do is synthesize it from a hundred blogs and conference proceedings and tweets and IRC discussions into something normal folks can understand.
• #6: I know this sounds pretty basic, but bear with me so we’re all on the same page. Nowadays, everything you want to do is either software, or requires software to do. You want to do sales bundling a different way, you want to gather usage data on your trucks, you want to optimize your manufacturing line. It all requires software, running on servers – in ITSM-speak we call these “services.” Email is a service, your e-commerce Web site is a service, Twitter is a service.
• #7: Here’s some hints that you have this root-cause problem.
• #8: Remember the year 2000? When CIO magazine was still in paper form, many inkwells were spilled about this topic.
• #9: But of course, competitive pressure made this untenable. So the business started pushing on the intake side of IT – development. Dev groups frequently reorganized to match LOBs and focused on innovation, and did things like uptake agile software development. But this just created the same problem in miniature inside IT, with the responsibility for reliability and cost siloed into Operations. A lot of the extant DevOps writing focuses on this impedance mismatch, with Dev wanting to push shaky code ever faster and Ops wanting more guarantees of stability and more process checks, since they are responsible for the stability and cost. This results in both a flow problem and a quality problem, which anyone with a background in manufacturing will understand immediately, it’s a basic “The Goal” theory of constraints problem.
• #10: DevOps definition – part one.
• #11: When I was a systems engineering manager at NI in Austin, we had great staff and did as much as we could, but were still “the bottleneck.” Every month we ran these Web software releases where 30-50 people would be on line from 7 PM Friday well through Saturday morning. We had hundreds of on-call pages a week some weeks. Finally one day I came to the realization that there was something fundamentally wrong with our approach. The overall way the software pipeline was constructed was designed in a way to never flow smoothly. Our agile, line-of-business-aligned development teams were crashing against a big, monolithic, infrastructure organization horizontally striped by technology specialty. And rather than understanding the theory of constraints, our organization – like most organizations – was just pushing harder into the hopper and hoping more sausage would come out the end. But the sausage that emerged… Was not fit for consumption. I went to our business owner and explained the problem to him. We then got Dev and Ops together and we put together a five-point set of goals that he would hold both dev and ops responsible for.
• #12: DevOps definition – part two.
• #13: The problem’s not just “outside” ops. To be honest, I found that in ops we had let a lot of computing best practices pass us by. Use of source control, testing, and the like are just plain better, and most system administrators didn’t do any of that. I came to this realization when we got a new VMWare farm in at NI. It took my team 6 weeks minimum to get in a new server – for we had to navigate all the other horizontal IT Infrastructure teams as well. Data center, networking, UNIX or Windows, everyone had their own processes and checklists. The sales people had gleefully demonstrated how “you can have a new server in 15 minutes!” This got my time down to getting a new server to 4 weeks. All the same delays, just with the Dell procurement turnaround removed from the process. “We’re turning 15 minutes into 4 weeks.” That doesn’t pass the smell test. Blah blah standardization blah blah cost blah blah specialization – WHERE THE HELL IS MY SERVER?
• #14: And now you’re caught up. I had been going to Velocity and watching the Agile Infrastructure list, and then got involved when Damon Edwards and John Willis put on OpsCamp Austin in 2010 and DevOps was the hot topic.
• #15: Culture. You may have heard this already. Sounds hippy-dippy. What is “culture?” Culture (/ˈkʌltʃər/) is, in the words of E.B. Tylor, "that complex whole which includes knowledge, belief, art, morals, law, custom, and any other capabilities and habits acquired by man as a member of society." (Tylor 1871:1)
• #16: One of the things that makes learning about DevOps difficult is that different discussions slot in at different parts in the conceptual hierarchy around DevOps. I’m not going to go through these in depth, they are just examples – but when you hear someone talking about something that “is DevOps” or “is not DevOps” it can bring clarity to understand what conceptual level they are talking about. You may also see this stated as People/Process/Tools.
• #17: At the top level you have the cultural and conceptual change required to set the vision and values for DevOps. The key is a high trust, shared risk culture with a win-win relationship between dev and ops (and everyone else). The Three Ways are Gene Kim’s way of communicating DevOps principles; John Willis coined CAMS.
• #18: Then you get into specific DevOps practices. No specific ones of them are necessary or sufficient per se, it just depends which you will realize value from. Our first DevOps implementation omitted all CI/CD because that was not a problem we had. This is by no means a complete list. These need to flow from the principles – I’ve been on the receiving end of very non-DevOps CM implementations, for example.
• #19: DevOps Tools, also known as “Chef or Puppet?” I am not covering tools in this talk, except to say: There are a lot of great tools that can help you implement some of the DevOps practices Putting tools first as the solution to a pervasive problem will work about as well as it ever has.
• #20: ITSM, back from the dead to our rescue. ITIL had a good conceptual model, but most proposed implementations approached the problem from the wrong mindset. A service can be described has having three phases in one big feedback loop – Design, Transition, and Operation. When we talk in DevOps we often shorthand these as “Dev,” “Release,” and “Ops” even though that’s usually not strictly correct. The phases don’t map one-to-one with skill sets, and there are other disciplines like QA, security, network engineers, etc. that participate in these. But Service DesTranOps isn’t as catchy.
• #21: These are sometimes ironically called nonfunctional requirements. Because if you neglect them, your service is nonfunctional. There are many ways to do this. You can embed operations experts into development teams, agile style. I have done that several times. You can make developers support their own app until it’s stable enough, Google SRE style. The most important thing is to work with people, not against them.
• #22: When managing a systems team, the developers rolled out a new version that needed 10x the amount of system resources it was using previously. “Can’t you just add more hardware,” they asked? “Not ten times as much hardware.” They then fixed the issue in the next release. Decisions made in the design phase impact performance, reliability, and security by orders of magnitude and is obviously the highest leverage point to make improvements. This can be hard, when we had joint dev and ops spec review meetings the devs were initially put off by the “weird ops questions.” But if you power through that till everyone’s speaking the same language, you can start to solve those problems.
• #23: Automate your change control process.
• #24: Even if you can’t get to continuous deployment, getting the increment down from typical quarterly, six week, or monthly releases reduces the number of things to go wrong. I was brought in to one SaaS company as a release manager to get their release cadence from an ever-slipping six weeks down to two weeks. This required test automation, deployment automation, and process discipline. We got on a regular cadence and it went so well that we then moved to weekly releases with no fanfare later. There was a smaller payload to test, so it was tested more thoroughly. If something went wrong, there was less payload to troubleshoot. If there was a bug, there was another release right around the corner. Dates didn’t slip, items just made a release or didn’t. Order came from chaos.
• #25: In other words, enhance your service operation phase with an understanding of development techniques.
• #26: Are you implementing this “bottom up?” Don’t go tool crazy, think about what you’re doing, and be able to track and prove the improvement to get more support.
• #27: Or are you looking to implement “top down?” Here’s some steps to take. Remember, you are looking for the highest overall flow of successfully delivered services for your limited resources. One specific thing that works here is reorganizing so necessary operations expertise is embedded in or aligned with the business/dev flow – basic queueing theory (hint: M/M/1 Markov queues) will show that having multiple consuming groups engaged with one centralized group to get what they need is suboptimal and as the load increases on that group, the wait time approaches infinity.
• #28: DevOps can fail for the reason most other things fail – because you take the easy way out and give it lip service instead of doing the hard work. Cultural transformation is hard, and shared responsibility takes commitment
• #30: I personally like The Principles of Product Development Flow but it’s pretty hardcore. Many more specialized conferences have flourished in the last couple years – SREConf, Monitorama, SCALE, SURGE, re:invent…
• #31: Questions?
• #32: Come to DevOpsDays Austin! And look up San Antonio DevOps on Meetup.com!