Fifteen Years of DevOps -- LISA 2012 keynote
4 likes902 views
The document discusses the evolution of software development and system administration over the past 15 years, highlighting the shift from formal methodologies like waterfall to more agile practices. It introduces DevOps as a cultural and operational adaptation that integrates developers and operations teams to improve collaboration and efficiency in software delivery and infrastructure management. Key concepts include automation, serviceability criteria, and the importance of treating networked systems as a single entity, emphasizing the need for continuous improvement in practices and tools within the context of changing technology landscapes.
![Test-Driven System Administration
Babushka
dep ‘ruby 1.9.2 in use’ do {!
!
met? {!
shell(‘ruby -version’)[‘ruby 1.9.2 p0’]!
}!
meet {!
shell(‘rvm use 1.9.2’)!
}!
!
end!
!
!
!
Idempotent Scripting!
49](https://image.slidesharecdn.com/fifteenyearsofdevopsv2-140809205104-phpapp02/85/Fifteen-Years-of-DevOps-LISA-2012-keynote-49-320.jpg)
Fifteen Years of DevOps -- LISA 2012 keynote
- 1. 15 Years of DevOps 1 Geoff Halprin The SysAdmin Group geoff@sysadmin.com.au LISA 2012 Conference San Diego, December 2012 (C) Copyright 2012, Geoff Halprin.
- 2. We will all look a lot more like Google over the next decade. ! Software Development has changed forever. ! Now it’s System Administration’s turn. 2
- 3. <1>
- 4. A Short History of Software Development
- 5. 5 The Age of the Dinosaurs Software Development in the 80s http://www.flickr.com/photos/cblue98/6679374097/
- 6. Highly Formalised, Unidirectional Workflow The Waterfall Model 6 Requirements Architecture Design Code Test Release ‣ Input: ‣ Requirements from user ‣Output: ‣Requirements Definition Document (RDD) ‣ Input: ‣ RDD ‣Output: ‣System Architecture Description (SAD) ‣ Input: ‣ SAD ‣Output: ‣Detailed Design Description (DDD) ‣ Input: ‣ DDD ‣Output: ‣Code ‣ Input: ‣ RDD, SAD, DDD, Code ‣Output: ‣Test Reports ‣ Input: ‣ Code, Test Reports ‣Output: ‣Product ‣ Key Attributes: • Left-to-right progress. Each phase completed before next commenced (in theory). • Right-to-left traceability. Code can be traced back to requirements.
- 7. Lie #1: It was always a bad guess 7
- 8. Lie #2: The world does not stand still 8
- 9. Lie #3: Not all developers are equal 9
- 10. Lie #4: The A-team versus the B-team 10
- 11. The Root Causes The Big M Methodologies ‣ Human Resources • They eschewed the difference in productivity of programmers. ‣ Wrong Model • All the creativity is at the beginning, the rest is just building. ‣ High Risk, Hence High Formality • Such large investments require significant controls to (attempt to) ensure that the value is delivered. 11
- 12. Software was no longer delivered monolithically The Environment Was Changing, Too ‣ Smaller, Interconnected Systems • 2-Tier (Client/Server) networking model (RPC) - Thick clients • Enterprise Service Bus ‣ Software Design Patterns (1994) ‣ The Web • No longer creating long distribution chains for physical media. • 3-tier architecture (Model, View, Controller) 12
- 13. Putting the SDLC on its head 13 The Agile Manifesto (2001) We Favour: Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan
- 14. We follow these principles: Agile Principles ‣ Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. ‣ Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage. ‣ Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. ‣ Business people and developers must work together daily throughout the project. ‣ Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done. ‣ The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. ‣ Working software is the primary measure of progress. ‣ Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely. ‣ Continuous attention to technical excellence and good design enhances agility. ‣ Simplicity--the art of maximising the amount of work not done--is essential. ‣ The best architectures, requirements, and designs emerge from self-organising teams. ‣ At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behaviour accordingly. 14
- 15. New words, but simple concepts Project Cadence 15 Release (8w) Iteration (2w) Showcase Retrospective Planning Iteration (2w) Iteration (2w) Iteration (2w) Unplanned Features, Stories Project Backlog Release Planning Execution Iteration Backlog
- 16. Business Value Faster and More Reliably Agile Versus Waterfall 16 Project Release 1 Release 3 Iteration 1 Iteration 2 Iteration 3 Iteration 4 Iteration 5 Release 2 Agile Project Value realised over time 2 weeks Project Solution Definition Design & Build RDD Waterfall-based Project SAD DDD Code Test Release Value realised over time 2 years?
- 17. A rivalry with much history ‣ The First Wrong Assumption • That Production looks just like Development. ‣ The Second Wrong Assumption • The Production looks just like a vanilla vendor install. ‣ The Third Wrong Assumption • That the Production environment is static. 17 Dev vs Ops, Part 1
- 18. The Methodologies Assumed We Would Talk How Did it Get This Way? ‣ ISO/IEC-12207:1995 SDLC • Does not mention system operations at all ‣ IEEE 1074-1991 • Does not mention system operations at all ‣ Rational Unified Process (2003) • 3½ whole pages on deployment! ‣ They all assume that the programmers will deal with the operational aspects of the system as part of their normal SDLC activities. • This has been proven to be a bad assumption! 18
- 19. All These Methodologies Assume We Are “In The Room” ‣ This same mistake in thinking can derail Agile software development projects • If your company is doing Agile software development, are Operations participating? • Are you placing “non-functional” stories in the backlog, or is the only ‘customer’ the one wanting new features? ‣ DevOps puts an Ops person in the room • That doesn’t guarantee the right stories are placed on the project backlog. 19 The Great Assumption
- 20. Making a System Administrator’s Life Less Stressful 20 Serviceability Criteria ‣ Each production environment is unique. • Applications must adapt to their environment. ‣ The environment that an application is deployed into is never static across its life. • Applications must cope with the world changing around them. ‣ These goals are achieved by the application exposing the necessary controls to the system administrator to provide for maintenance. ‣ This is actually a definable set of criteria that, when met, make a system maintainable across its life. • We will call these the serviceability criteria.
- 21. aka “Non-Functional Requirements” 21 Operational Requirements ‣ Serviceability Criteria (Scope: Global) • Wouldn’t it be nice if all applications exposed these controls in a uniform manner? ‣ Standard Operating Environment (Scope: Local) • All applications within a particular environment share these. e.g. naming conventions, filesystem layout standards. ‣ Application Non-Functional Requirements (Scope: Application) • These are unique to each application. e.g. Application performance targets. Data storage requirements.
- 22. As an industry, we have left each individual sysadmin to fight for themselves (and repeat the same mistakes). ! We need to do better. 22
- 23. One Last Thing... ‣ All programming projects require the basics to be correct: • Software Change Management - aka Change Control • Software Configuration Management - aka Version Control • Software Build and Release Management 23 The Grammar of Programming
- 24. </1>
- 25. <2>
- 26. A Short History of System Administration
- 27. An Apology The following history is highly subjective and superficial. It ignores by omission the incredibly important contributions of a great many people. This was not intentional. 27
- 28. 28 Firefighters First and Foremost
- 29. Hardware was expensive People were cheap 29
- 30. You are in a maze of twisty passages, all different The Age of the Pioneer 30 ‣ Large Systems ‣ Many Networking Variants • TCP/IP, SNA, X.25, Token Ring, DECnet, Netware ‣ Bespoke Configurations ‣ Vertical Scaling ‣ Everyone had a Unix variant
- 31. A well-kept secret! What’s a System Administrator? 31 ‣ LISA Conference: 1986 • 100 users or over 1GB of storage ‣ SAGE: 1992 (SAGE-AU: 1993) • Earlier: Sun User Group, etc. ‣ What’s in a name? • System Programmer, System Administrator, Network Administrator, Network Engineer • Officially: Programmer, Research Scientist, ...
- 32. LISA plays a key role in bringing thought leaders together The Profession is Developing ‣ SA-CMM (1993) ‣ SAGE Job Descriptions Booklet (1993) ‣ Formal courses in SysAdmin at CQU (Aus) and U. Oslo 32
- 33. But TCP/IP is here to stay The Network Isn’t The Computer 33 ‣ The Internet • TCP/IP becomes the standard network. ‣ Custom IT Solutions • Everyone is special. Every solution unique. ‣ Tools begin to emerge: • sudo (1980), top (1984), rdist (1986), perl 4.036 (1993), cfengine (1993), mrtg (1995), ssh (1995), rsync (1996)
- 34. Before ITIL, there was SA-BOK 34 The SA-BOK Network Management Facilities Management Server Management Software Management Data Management To Organise Data Security Business Continuity Planning To Protect Performance Management Process Automation To Optimise Capacity Planning Technology Planning To Plan Service Management Problem Management Production Management Asset Management Change Management To Control
- 35. With great power comes great responsibility The Rise of The Web 35 ‣ The Emergence of The Web • Thin clients! ‣ Three Tier Infrastructure Model (Web-App-DB) • Physical Servers Rule ‣ Linux at the Edge; Solaris at the Core
- 36. Penguin rising The Commoditisation of IT 36 ‣ Commodity Hardware • Sun, IBM, Intel still battling for the enterprise. • Intel/Linux has won the startups. ‣ The Rise of Virtualisation • VMware and Linux drive adoption of x86 architecture.
- 37. Nothing will ever be the same again The Emergence of the Cloud ‣ The Wars are over: Linux, x86, VMs. • Amazon Web Services. Infrastructure on demand. ‣ The Rise of the API • Infrastructure as code. ‣ Automation driven by scale • The WebOps Movement. • The DevOps Movement. ‣ What’s in a name (part 2)? • SRE, Web Ops, DevOps... 37
- 38. </2>
- 39. <3>
- 40. The Rise of DevOps 40
- 41. The same way that failure of Waterfall drove developers to change the way we developed software, Cloud-based sites and the need for rapidly provisioned, highly scalable infrastructure drove the way we managed infrastructure. 41
- 42. DevOps is to System Administration what Agile is to Software Development ‣ Culture and Attitude • First and foremost, DevOps is about creating a collaborative environment between Dev and Ops, integrating the teams as much as possible. ‣ Practices and Processes • Automate as much as possible. ‣ Technology and Tools • Old and New tools to support the effort. 42 What is DevOps?
- 43. It all starts by sitting on the same side of the table DevOps Culture ‣ The simple act of colocating Dev and Ops people is the single, largest step. • Visibility of each others work through Kanban walls. • Shared food and discussions. • Cross-over of teams. 43
- 44. Some common DevOps practices DevOps Practices ‣ Developer Behaviour Changes • Infrastructure as Code • Continuous Integration • Production Support (carrying a pager!) ‣ SysAdmin Behaviour Changes • Embedded with Dev teams • Kanban walls for Ops 44
- 45. Visibility is the first step to collaboration Kanban Walls for Ops 45 Backlog Ready Active Review Done Express
- 46. So many (these are just a few) DevOps Tools ‣ Configuration Management • Puppet, Chef, Babushka, cfengine ‣ Software Change Management (Version Control) • Github ‣ Continuous Integration • Jenkins, Capistrano ‣ Testing • Rails, Cucumber, Vagrant 46
- 47. Infrastructure in the Cloud And Frameworks ‣ Commercial • Amazon Web Services • VMware Dynamic Ops, BMC CLM ‣ Open Source • Open Stack, CloudForms, Eucalyptus, Open Nebula, Open Cloud Framework, • Cloud Foundry, AppFog, Heroku, OpenShift ‣ Standards • ??? 47
- 48. Adapting Agile tools to DevOps Cucumber # language: en! Feature: Addition! In order to avoid silly mistakes! As a math idiot ! I want to be told the sum of two numbers! ! Scenario Outline: Add two numbers! Given I have entered <input_1> into the calculator! And I have entered <input_2> into the calculator! When I press <button>! Then the result should be <output> on the screen! ! Examples:! | input_1 | input_2 | button | output |! | 20 | 30 | add | 50 |! | 2 | 5 | add | 7 |! | 0 | 40 | add | 40 | 48
- 49. Test-Driven System Administration Babushka dep ‘ruby 1.9.2 in use’ do {! ! met? {! shell(‘ruby -version’)[‘ruby 1.9.2 p0’]! }! meet {! shell(‘rvm use 1.9.2’)! }! ! end! ! ! ! Idempotent Scripting! 49
- 50. Breaks down the silos What DevOps Does ‣ Ops are now “in the room” with the Dev team. • Dev do after hours support! • Dev now build more maintainable code. ‣ An end-to-end system view • Dev now see (and respond to) the problems their code causes across the entire lifecycle. ‣ Configuration Management is automated. ‣ Change Management is automated. ‣ Continuous Integration and Release 50
- 51. Keeping the Code Clean Continuous Integration and Release ‣ Continuous Integration is a Development Team Practice • It requires well-behaved developers and a clean code base. ‣ Continuous Release is a DevOps Practice • It requires a well-defined (automated) release procedure. • It requires well-defined target infrastructure. 51
- 52. 52
- 53. Fully cooked... What DevOps Isn’t ‣ A silver bullet • Systems still break. Patches still need to be applied. • Not all products fit a pure DevOps model. ‣ Complete or holistic • ITIL? SA-BOK? Normal lifecycle? ‣ Highly scalable into traditional enterprise environments • Agile started in small teams. It took a while to learn how to scale it. • DevOps has started with highly specialised applications, developed to leverage the DevOps model. 53
- 54. DevOps sees the vertical world of an application Problems DevOps Doesn’t Solve ‣ Serviceability Criteria ‣ Technical Debt / Shoemaker Projects ‣ Service Monitoring ‣ The Rest of the Ops Lifecycle ‣ Professional Ethics, Training, etc. 54
- 55. Does DevOps Fit the Enterprise? Web Ops versus Enterprise Ops ‣ You can’t just throw away infrastructure ‣ Vertically scaled apps ‣ Enterprise Change Management ‣ Poorly behaving vendors 55
- 56. Vendors have been writing bad software for a very long time... Badly Written Software ‣ 1994: No, you can't have root (Craig Bishop) ‣ 1995: Guidelines for software developers (Geoff Halprin) ‣ 2001: The problem with developers (Geoff Halprin) ‣ 2005: Software release engineering (Geoff Halprin and Lee Damon) 56
- 57. 57
- 58. 58
- 59. 59
- 60. </3>
- 61. <4>
- 62. DevOps Isn’t New
- 63. A definition System Administration ‣ To maintain the availability and integrity of computing resources ‣ To support and encourage the effective use of those resources 63
- 64. To make ourselves redundant Our Motivation ‣ People don't like doing work ‣ Lazy people invent ways to avoid work ‣ Therefore, all progress is made by lazy people 64
- 65. Making Ourselves Redundant ‣ Strategy 1: Consistency • More consistency = reduced effort across a fleet ‣ Strategy 2: Autonomous Systems • The more they look after themselves, the less for us to do! ‣ Strategy 3: Working on the Right Problems! • Automate the tedium. • Capture your expertise. 65
- 66. Working on the right problem Changing System Files ‣ Our job should not be a memory test ‣ We should concentrate on where the best payoff is ‣ Why do I have to remember which signal to send to which process for each configuration file? • Or which ones are binary and need a different editor? 66
- 67. Why not edit all files the same way? Change 67 Checko ut Edit Validate CheckinCommit # change /etc/passwd ! /etc/.change: Default: checkout rcs co %% checkin rcs ci %% edit vi %% end ! passwd: edit: vipw end ! inetd.conf: validate: v_etc_inetd commit: kill -HUP `...`
- 68. Moving from managing a single host to managing a fleet Structured Systems Administration ‣ Treat the whole network as a single system; not as a large number of independent hosts. ‣ Consistency ‣ Scalability 68
- 69. 69
- 70. 70
- 71. 71
- 72. 72
- 73. 73
- 74. 74
- 75. 1995 75
- 76. Spinning up a Point-of-Presence in 30 minutes (after the physical rack-and-stack) 76 ISP in a Box ‣ Components • Configurator. Build configuration files from a central master properties file. • Jumpstart with a whole bunch of post-provisioning, to install patches (with reboots), software, configuration, etc. • Configuration of Cisco routers. • Configuration of central servers to talk to the new POP. ‣ This was all done in 1997
- 77. </4>
- 78. <5>
- 79. 79 Let’s look a little deeper at some System Administration practices...
- 81. The Life of a System Patches System Faults User Problems Software Upgrades New Products Entropy The Perfect Stable System
- 83. The System as a State Machine • Configuration Management • Management of State Definition and Integrity • Change Management • Management of State Transition Integrity
- 84. Configuration Management • Managing State Integrity • Ability to accurately define system components (CMDB) • Ability to accurately define system state (configuration files, etc) • Ability to recover system state (backups)
- 85. Change Management • Managing State Transition Integrity • Correctness of target state • Correctness of work instructions
- 86. Change Management = Risk Management 86
- 87. The Purpose of Change Management is to manage the risks associate with state transition The Risks Being Managed ‣ Process Failures • Incorrect Change Procedure • Procedure Not Followed Correctly • Unexpected Side-Effect ‣ End State Failures • Toxic End State 87
- 88. All Changes share the same risk mitigation strategies Risk Mitigation ‣ Backups • The point in time directly before the change is the last known stable point, hence the baseline for all other mitigation plans. ‣ Other techniques • Copy files sideways. Split mirrors. Snapshots. • Change qualification. Progressive deployment. • Copilots. Contingency time. • Automation. 88
- 89. DevOps sees everything as a code push DevOps and Change Management ‣ DevOps is highly optimised for a single type of Change -- the code push. • This often requires a full build from scratch. • They also deal with complex changes, like database schema changes with minimal service disruption. • Service availability is paramount in DevOps change philosophy. 89
- 91. System Administration has been talking about Configuration Management for a long time! ‣ Many Tools • rdist, cfengine, bcfg2, satan, Netomata, puppet, chef, ... ‣ Many Approaches • push/pull, centralised/distributed, various abstraction layers and models. ‣ Many Papers • Dozens of papers over 26 years. 91 Traditional Configuration Management
- 92. DevOps is highly reliant on automated Configuration Management ‣ DevOps solves this for individual applications. • The vertical stack. • Often by blowing a VM away and starting again. ‣ Not the complexity of the entire infrastructure. • Assumes shared infrastructure (physical compute, store, network) are all in place (and configured!). 92 DevOps and Configuration Management
- 93. DevOps doesn’t solve these problems, but it does establish a better culture for solving them! 93
- 94. </5>
- 95. <6>
- 96. DevOps is the continuation of the path we have been on since the beginning. 96
- 97. DevOps is filling the void that our lack of progress as a profession has left unanswered. 97
- 98. Where does that path lead? 98
- 99. Everything is software now The Software Defined Data Centre ‣ Virtualisation is only the first step • It is necessary, but not sufficient • 20K machines virtualised are still 20K machines individually managed. ‣ Automation is critical to that journey • Standardisation is the basis for automation. ‣ Cloud standards are in their infancy • Expect a lot of change over the coming years. 99
- 100. Realising the Cloud 100 The Automation Manifesto We Favour: VM templates and golden images over Standards and documents Automated processes over Documented procedures and build instructions Self-service of standardised offerings over Custom designs Consistency of hosts over Custom configurations Machine parseable hand-off artefacts over Human written and processed documents Low risk “standard” change activities over Complex, scheduled change windows Centralised management platforms over Individual host and component management Industry standards over Proprietary protocols Consistency of toolset over Consolidation of products © Geoff Halprin
- 101. DevOps is the harbinger of the next wave DevOps and Automated Infrastructure Innovators Early Adopters Early Majority Late Majority Laggards TheChasm Early Market WebOps Crossing the Chasm DevOps Mainstream Market Enterprise Hybrid Clouds Late Market Mainstream Market Enterprise Private Clouds
- 102. Infrastructure delivered via API will be the basis for large scale enterprise outsourcing of infrastructure and associated support teams. 102
- 103. Infrastructure delivered via API will be the basis for large scale enterprise outsourcing of system teams The Next Outsourcing Wave? ‣ Rob Thomsett: Process versus Project Work • First generation of outsourcing was just replacing staff. • Infrastructure was bespoke; each machine managed as an individual, customised solution. ‣ Standardised infrastructure delivered by API • You can shop around; you can use multiple providers simultaneously. 103
- 104. A market place for services The Infrastructure Service Bus 104 Service Bus Service Domains Compute Storage Network PortalConsumer Service Catalogue
- 105. This really will change the industry The Infrastructure Service Bus ‣ Enterprise (Infrastructure) Service Bus • Service Provider Model • Being embraced by large enterprises now! • The outsource providers are becoming cloud providers! ‣ Our Cloud APIs are evolving into Middleware (message bus) APIs • System Administrations looks even more like enterprise software systems! 105
- 106. The irony here is that as infrastructure is treated more like code, the very frameworks and patterns used for software development become applicable to infrastructure. 106
- 107. </6>
- 108. <7>
- 109. ‣ The cost models of Enterprise IT are broken • If you don’t move to a cloud model for IT service delivery, you will be outsourced. 109
- 110. ‣ We are now in the decade of the API • Everything is code. 110
- 111. ‣ We are at the beginning of this transition • There’s plenty for everyone to do! 111
- 112. What you do next is up to you...
- 114. Some of the source data used in this talk References ‣ The Agile Manifesto • http://www.agilemanifesto.org ‣ The Facts and Fallacies of Software Engineering • Robert L. Glass, ISBN 0-321-11742-5. ‣ System Administrator’s Body of Knowledge (SA-BOK) • http://www.sysadmin.com.au/sa-bok.html ‣ Serviceability Criteria • https://www.sysadmin.com.au/whitepapers/developer_guidelines-0212.pdf 114