Gov 2.0: Scaling, Automation, & Management in the Cloud
11 likes9,014 views
The document discusses scaling infrastructure in the cloud. It provides summaries aimed at different audiences including developers, systems administrators, and executives. For developers, it notes that infrastructure is now the application and they need tools to manage it. For systems administrators, it encourages embracing new approaches. For executives, it notes that benefits come from efficiency rather than raw spending and there are cultural implications. It also discusses how infrastructure management has evolved and become easier over time but remains a challenge without the right tools.
Gov 2.0: Scaling, Automation, & Management in the Cloud
- 1. Scaling in the Cloud Speaker: Jesse Robbins CEO ‣ jesse@opscode.com ‣ @jesserobbins ‣ www.opscode.com Copyright © 2010 Opscode, Inc - All Rights Reserved 1
- 2. Opscode makes a new kind of Infrastructure Automation, offered as a hosted Service. Copyright © 2010 Opscode, Inc. – Confidential – Do Not Redistribute 2
- 4. • Developers? http://www.flickr.com/photos/ timyates/2854357446/sizes/l/
- 5. • Developers? • Systems Administrators? http://www.flickr.com/photos/ timyates/2854357446/sizes/l/
- 6. • Developers? • Systems Administrators? • Executives/Leaders? http://www.flickr.com/photos/ timyates/2854357446/sizes/l/
- 8. For Developers... • Do it yourself.
- 9. For Developers... • Do it yourself. • The infrastructure is the application (and vice versa).
- 10. For Developers... • Do it yourself. • The infrastructure is the application (and vice versa). • You are not a Systems Administrator.
- 11. For Developers... • Do it yourself. • The infrastructure is the application (and vice versa). • You are not a Systems Administrator. • You need tools.
- 12. Sysadmins.. http://covers.oreilly.com/images/9780596007836/lrg.jpg Lean into it appears courtesy of Cliff Moon, of Dynomite fame: http://twitter.com/moonpolysoft
- 13. Sysadmins.. • Say “Yes”. http://covers.oreilly.com/images/9780596007836/lrg.jpg Lean into it appears courtesy of Cliff Moon, of Dynomite fame: http://twitter.com/moonpolysoft
- 14. Sysadmins.. • Say “Yes”. • You never liked rack and stack that much anyway. http://covers.oreilly.com/images/9780596007836/lrg.jpg Lean into it appears courtesy of Cliff Moon, of Dynomite fame: http://twitter.com/moonpolysoft
- 15. Sysadmins.. • Say “Yes”. • You never liked rack and stack that much anyway. • You have never been more critical. http://covers.oreilly.com/images/9780596007836/lrg.jpg Lean into it appears courtesy of Cliff Moon, of Dynomite fame: http://twitter.com/moonpolysoft
- 16. Sysadmins.. • Say “Yes”. • You never liked rack and stack that much anyway. • You have never been more critical. • Lean into it. http://covers.oreilly.com/images/9780596007836/lrg.jpg Lean into it appears courtesy of Cliff Moon, of Dynomite fame: http://twitter.com/moonpolysoft
- 17. Executives...
- 18. Executives... • Not a magic unicorn
- 19. Executives... • Not a magic unicorn • Benefits come from efficiency, not raw Capex
- 20. Executives... • Not a magic unicorn • Benefits come from efficiency, not raw Capex • Has real cultural implications at every level
- 21. Executives... • Not a magic unicorn • Benefits come from efficiency, not raw Capex • Has real cultural implications at every level • You are the biggest asset to success
- 22. “Traditional” Operations Operations - The “Secret Sauce” 50 50 40 40 # of Hours 30 30 20 20 Hardware OS Install 10 10 Config Upkeep 20 20 15 15 Servers 10 10 New 5 5 Existing 0 0 1 2 3 4 5 6 7 9 10 11 12 1 2 3 4 5 6 7 9 10 11 12 Week # Week # (http://radar.oreilly.com/archives/2007/10/operations-advantage.html) Copyright © 2010 Opscode, Inc - All Rights Reserved 7
- 23. “Traditional” Operations Operations - The “Secret Sauce” 50 50 This is the secret of 40 40 Cloud Computing. Every other virtue stems from # of Hours 30 30 here. 20 20 Hardware OS Install 10 10 Config Upkeep 20 20 15 15 Servers 10 10 New 5 5 Existing 0 0 1 2 3 4 5 6 7 9 10 11 12 1 2 3 4 5 6 7 9 10 11 12 Week # Week # (http://radar.oreilly.com/archives/2007/10/operations-advantage.html) Copyright © 2010 Opscode, Inc - All Rights Reserved 7
- 25. You are 10% Unique
- 26. You are 10% Unique And itʼs probably the things you did wrong
- 27. Infrastructure is Hard Copyright © 2010 Opscode, Inc - All Rights Reserved 9
- 28. Infrastructure is Hard 1999 Inventory, packaged file transers and desktops Copyright © 2010 Opscode, Inc - All Rights Reserved 9
- 29. Infrastructure is Hard 1999 Inventory, packaged file transers and desktops 2005 Unattended bare metal servers “very very” hard 7k Nodes took 5 days w/90 success Copyright © 2010 Opscode, Inc - All Rights Reserved 9
- 30. Infrastructure is Hard 1999 Inventory, packaged file transers and desktops 2005 Unattended bare metal servers “very very” hard 7k Nodes took 5 days w/90 success 2007 Unattended bare metal in under 10 minutes Fully configured in under 3 mins Copyright © 2010 Opscode, Inc - All Rights Reserved 9
- 31. Infrastructure is Hard 1999 Inventory, packaged file transers and desktops 2005 Unattended bare metal servers “very very” hard 7k Nodes took 5 days w/90 success 2007 Unattended bare metal in under 10 minutes Fully configured in under 3 mins 2008 Unattended server in 2 minutes 5000 servers in a week Copyright © 2010 Opscode, Inc - All Rights Reserved 9
- 32. Infrastructure is Hard 1999 Inventory, packaged file transers and desktops 2005 Unattended bare metal servers “very very” hard 7k Nodes took 5 days w/90 success 2007 Unattended bare metal in under 10 minutes Fully configured in under 3 mins 2008 Unattended server in 2 minutes 5000 servers in a week 2010 10k Nodes in under 5 minutes Copyright © 2010 Opscode, Inc - All Rights Reserved 9
- 33. Infrastructure is changing Copyright © 2010 Opscode, Inc - All Rights Reserved 10
- 34. Infrastructure is changing ‣ Easier to get (good!) ...but harder to manage (bad!) Copyright © 2010 Opscode, Inc - All Rights Reserved 10
- 35. Infrastructure is changing ‣ Easier to get (good!) ...but harder to manage (bad!) ‣ Demand is dynamic Copyright © 2010 Opscode, Inc - All Rights Reserved 10
- 36. Infrastructure is changing ‣ Easier to get (good!) ...but harder to manage (bad!) ‣ Demand is dynamic ‣ Developers are crucial to Operations Copyright © 2010 Opscode, Inc - All Rights Reserved 10
- 37. Infrastructure is changing ‣ Easier to get (good!) ...but harder to manage (bad!) ‣ Demand is dynamic ‣ Developers are crucial to Operations ‣ Web / Cloud services are proliferating ...and Enterprise is following along. Copyright © 2010 Opscode, Inc - All Rights Reserved 10
- 38. Infrastructure is changing ‣ Easier to get (good!) ...but harder to manage (bad!) ‣ Demand is dynamic ‣ Developers are crucial to Operations ‣ Web / Cloud services are proliferating ...and Enterprise is following along. ‣ Manual configuration no longer a crutch Copyright © 2010 Opscode, Inc - All Rights Reserved 10
- 39. Infrastructure is changing ‣ Easier to get (good!) ...but harder to manage (bad!) ‣ Demand is dynamic ‣ Developers are crucial to Operations ‣ Web / Cloud services are proliferating ...and Enterprise is following along. ‣ Manual configuration no longer a crutch ‣ Few tools to solve a ubiquitous problem Copyright © 2010 Opscode, Inc - All Rights Reserved 10
- 40. Managing Infrastructure Is Hard Has Always Been Proprietary Solutions Previous Attempts Typically... 1980 • Solve very little of the problem... 1989 • Reach just a handful of large, enterprise customers 1999 • Require custom implementations with large professional services bills • Deployed exclusively on-premise 2001 • Acquired by companies with large consulting organizations (IBM, HP, CA) Copyright © 2010 Opscode, Inc - All Rights Reserved
- 41. Google, Amazon, Microsoft built their own tools Copyright © 2010 Opscode, Inc - All Rights Reserved 12
- 42. but it’s “secret sauce” Copyright © 2010 Opscode, Inc - All Rights Reserved 13
- 43. P everyone else is here ... inexperienced & poorly equipped for the world they must now operate in. Copyright © 2010 Opscode, Inc - All Rights Reserved 14
- 44. “Cloud”
- 45. Alistair’s mom’s definition Cloud
- 46. Alistair’s mom’s definition Cloud = Web
- 47. Alistair’s mom’s definition Cloud = Web = Internet
- 48. Alistair’s mom’s definition Cloud = Web = Internet = Useless
- 49. Slide courtesy Alistair Croll - alistair@rednod.com
- 50. Private Public Slide courtesy Alistair Croll - alistair@rednod.com
- 51. Private Public Managed Virtualization hosting Slide courtesy Alistair Croll - alistair@rednod.com
- 52. Private Public IaaS IaaS Managed Virtualization hosting Slide courtesy Alistair Croll - alistair@rednod.com
- 53. Private Public PaaS PaaS IaaS IaaS Managed Virtualization hosting Slide courtesy Alistair Croll - alistair@rednod.com
- 54. Private Public SaaS PaaS PaaS IaaS IaaS Managed Virtualization hosting Slide courtesy Alistair Croll - alistair@rednod.com
- 55. Private Public nt t o SaaS a w d s, o u lo u y c If l k PaaS rs t.PaaS t a ne f i i ck o IaaS p IaaS Managed Virtualization hosting Slide courtesy Alistair Croll - alistair@rednod.com
- 56. Infrastructure as a Service (IaaS) Amazon EC2, Rackspace Cloud, Terremark, Gogrid, Joyent (and nearly every private cloud built on Zenserver or VMWare.) Slide courtesy Alistair Croll - alistair@rednod.com
- 57. Dedicated On-premise Virtual Third-party hardware private clouds private clouds public clouds Slide courtesy Alistair Croll - alistair@rednod.com
- 58. Slide courtesy Alistair Croll - alistair@rednod.com
- 59. Always on premise Private Compliance- enforced Need to track and audit Legislative Data near local computation Slide courtesy Alistair Croll - alistair@rednod.com
- 60. Always on Can be done premise anywhere Private Compliance- Testing enforced Training Need to track and Prototyping audit Batch processing Legislative Seasonal load Data near local computation Slide courtesy Alistair Croll - alistair@rednod.com
- 61. Always on Can be done Always in premise anywhere cloud Private Partner access Compliance- Testing enforced Proximity to cloud Training services (storage, Need to track and Prototyping CDN, etc.) audit Batch processing Massively grid/ Legislative Seasonal load parallel (genomic, Data near local modelling) computation Slide courtesy Alistair Croll - alistair@rednod.com
- 62. Always on Can be done Always in premise anywhere cloud Load/pricing engine Private Partner access Compliance- Testing enforced Proximity to cloud Training services (storage, Need to track and Prototyping CDN, etc.) audit Batch processing Massively grid/ Legislative Seasonal load parallel (genomic, Data near local modelling) computation Slide courtesy Alistair Croll - alistair@rednod.com
- 63. Always on Can be done Always in premise anywhere cloud Load/pricing engine Private Partner access Compliance- Testing enforced Proximity to cloud Training services (storage, Policy engine Need to track and Prototyping CDN, etc.) audit Batch processing Massively grid/ Legislative Seasonal load parallel (genomic, Data near local modelling) computation Slide courtesy Alistair Croll - alistair@rednod.com
- 64. Virtual machine (infrastructure cloud) Always on Can be done Always in premise anywhere cloud Load/pricing engine Private Partner access Compliance- Testing enforced Proximity to cloud Training services (storage, Policy engine Need to track and Prototyping CDN, etc.) audit Batch processing Massively grid/ Legislative Seasonal load parallel (genomic, Data near local modelling) computation Slide courtesy Alistair Croll - alistair@rednod.com
- 65. Compute task (service cloud) Always on Can be done Always in premise anywhere cloud Load/pricing engine Private Partner access Compliance- Testing enforced Proximity to cloud Training services (storage, Policy engine Need to track and Prototyping CDN, etc.) audit Batch processing Massively grid/ Legislative Seasonal load parallel (genomic, Data near local modelling) computation Slide courtesy Alistair Croll - alistair@rednod.com
- 66. Automation
- 67. Bootstrapping
- 68. Bootstrapping Approaches Good Bad Time Known Costs, No High Waste (Hoarding) Variation. Red Tape Corp Approvals Anything you want, as long Expensive ($/Time) 6-8w as IT pre-approved it. Long lead time Lower Waste Agile Corp Known Costs. Less Red Tape Total Hardware Control. Still slow 2-4w Approvals Trivial Approvals. Expensive ($/Time) Shorter lead time Variable Costs. Highly Adaptable. Variable Costs. Cloud Minimal lead time. Trivial approvals. No control over hardware. Must re-train. 5-10m No humans needed.
- 69. Configuration curl -O http://brainspl.at/velocity.sh && sh velocity.sh
- 70. Configuration Approaches Good Bad Slow. You can do anything. Error Prone (Bus Error!) Manual Results in an intimate knowledge of the details. Non-repeatable. Difficult knowledge transfer. Rarely idempotent. More repeatable. Hard to collaborate. Ad-Hoc Knowledge is dispersed. Built your way, with your model. Brittle. No API. Repeatable. Infrastructure Idempotent. Agile. Have to learn how to use it. Hard things remain hard. as Code Sharable. Self documenting. Not magic. (Yet!)
- 71. Command and Control
- 72. Command and Control Good Bad Super flexible. Error Prone. Can do almost anything. Slow. Meatcloud* Always easy to find someone to blame. Expensive to Scale. Not repeatable. Free will. Free will. One-off by neccessity. More repeatable. Tooling sprawl. Ad-Hoc Easier to scale. Less error prone (hopefully!) Hard to share solutions. Much higher learning curve. One system to learn. Scales well. Not everything maps cleanly. Framework Paint by numbers. Repeatable. Trades depth of knowledge for ease of use. Two-Way. *Meatcloud appears in this presentation courtesy of Andrew Shafer - http://is.gd/Ega
- 73. Lightning Strikes! DOOM Webservers Database Servers Webservers
- 74. Lightning Strikes! DOOM X Webservers XX Database Servers Webservers
- 75. Lightning Strikes! DOOM X 1 1 2 1 Signals Moar! XX Monitoring Command & Bootstrapping System Webservers Updates Control Provisions 2 1 3 1 5 1 3 1 Database Servers 4 1 4 1 Configuration Webservers
- 76. Lightning Strikes! Monitoring Signals Nanite /node/down Service DOOM X 1 1 2 1 Signals Moar! XX Monitoring Command & Bootstrapping System Webservers Updates Control Provisions 2 1 3 1 5 1 3 1 Database Servers 4 1 4 1 Configuration Webservers
- 77. Lightning Strikes! Nanite boots new EC2 Nanite removes DOOM Instances, with Chef Role + nodes in Chef Attribute X 1 1 2 1 Signals Moar! XX Monitoring Command & Bootstrapping System Webservers Updates Control Provisions 2 1 3 1 5 1 3 1 Database Servers 4 1 4 1 Configuration Webservers
- 78. Lightning Strikes! DOOM X 1 1 2 1 Provisions Instances, EBS, Signals Moar! Elastic IPs XX Monitoring Command & Bootstrapping System Webservers Updates Control Provisions 2 1 3 1 5 1 3 1 Database Servers 4 1 4 1 Configuration Webservers
- 79. Lightning Strikes! DOOM X 1 1 2 1 Signals Moar! XX Monitoring Command & Bootstrapping System Webservers Updates Control Provisions 2 1 3 1 5 1 3 1 Chef Database Servers configures nodes 4 1 4 1 according to Configuration assigned Webservers
- 80. Lightning Strikes! DOOM Chef X 1 1 2 1 Signals Moar! updates the XX Monitoring Command & Bootstrapping monitoring System Webservers Updates Control Provisions system 5 1 2 1 3 1 3 1 Database Servers 4 1 4 1 Configuration Webservers
- 81. A word about Scaling...
- 82. Typical Peak Load 1.Bring on capacity as traffic ramps up 2.Take down capacity as it ramps down 3.10-15 Minutes on either side, fully unattended Graphs in this portion of the presentation taken from Theo Schlossnagle http://omniti.com/seeds/dissecting-todays-internet-traffic-spikes
- 83. Atypical Load No way However, around you are Capacity still better Planning off! 1.Hope you know it is coming. 2.Increase capacity in advance. 3.Take down capacity as it ramps down. Graphs in this portion of the presentation taken from Theo Schlossnagle http://omniti.com/seeds/dissecting-todays-internet-traffic-spikes
- 84. Capacity Planning is king. http://www.flickr.com/photos/allspaw/2095439645/sizes/l/
- 86. Have a queue?
- 87. Have a queue? Does it scale linearly with more resources?
- 88. Have a queue? Does it scale linearly with more resources? Congratulations - you can auto-scale!
- 90. CAP Theorem • Consistency Pick Two • Availability • Partition Tolerance
- 91. Most SQL Databases • Choose Consistency over all • Availability comes distant second
- 92. Web Applications need... • Availability • Partition Tolerance
- 93. “Global temporal consistency is a fiction” Christopher Brown
- 94. Choosing Consistency for your Web App... Means failure is global
- 95. When you choose Partition Tolerance and Availability... You fail or succeed for a subset of users
- 96. Apologies • Apologize after the fact for failures • Better than nothing at all
- 97. NoSQL • Many different tools • They tweak CAP differently • CouchDB • Cassandra • Redis • MongoDB
- 98. Scaling in the Cloud Speaker: Jesse Robbins CEO ‣ jesse@opscode.com ‣ @jesserobbins ‣ www.opscode.com Copyright © 2010 Opscode, Inc - All Rights Reserved 43