3 Years of Puppet at Cisco: The Secrets to Our Success - PuppetConf 2013
1 like2,376 views
The document outlines Cisco's success with Puppet over three years, detailing team members' roles and experiences. It describes various phases of development, highlighting the evolution of technologies used, deployment processes, and improvements made over time. Key tools mentioned include Director, RPMs, monitoring solutions, and the open-source community's role in enhancing operational efficiency.
3 Years of Puppet at Cisco: The Secrets to Our Success - PuppetConf 2013
- 1. Cisco Confidential© 2010 Cisco and/or its affiliates. All rights reserved. 1 3 Years of Puppet at Cisco: The Secrets to Our Success PuppetConf 2013 Keith Chambers & Ryan Uber
- 2. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2 • Worked at Cisco for 13 years - 7 years in Technical Assistance Center (support) - 2 years in Product Marketing - 4 years as WebEx Social Platform Architect • Passionate about technology • Passionate about people • Always looking to raise the bar
- 3. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 3 • Worked at Cisco for 3 years - Software Engineer - Has played a critical role in our success • Work at a managed host provider for 5 years - Designed and implemented most of the infrastructure - Supported everything he built • Puppet user for 6 years
- 4. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 4 • Overview • Experience • Plumbing • History & Evolution • Secrets to our success
- 5. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 5
- 6. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 6 • Enterprise Social Media - Create content - Communities - Follow - Like - Activity feed - Recommendations • On-premise deployment model • Later added cloud hosted
- 7. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 7 Persistence • Traditional tiered web service • Large number of services - App Server - Worker - OpenFire - RabbitMQ - Memcached - MongoDB - Oracle - Solr Caching Application Web Server Worker
- 8. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 8 • Virtual appliance delivery model • Roles based architecture - 1 service per VM - 25 VM deployments are common • Director provides system wide: - Configuration - Software maintenance - Diagnostics and logging - Health monitoring • Everything works out of the box Oracle OS MongoDB OS Solr OS Worker OS Memcached OS App OS RabbitMQ OS OpenFire OS Virtualized Compute & Storage Director OS
- 9. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 9
- 10. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 10 1. Customer downloads a single OVA from Cisco
- 11. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 11 1. Customer downloads a single OVA from Cisco 2. Customer deploys the Director VM from the OVA by selecting the Director configuration option
- 12. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 12
- 13. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 13 1. Customer downloads a single OVA from Cisco 2. Customer deploys the Director VM from the OVA by selecting the Director configuration option 3. Customer configures Director VM network settings
- 14. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 14 1. Customer downloads a single OVA from Cisco 2. Customer deploys the Director VM from the OVA by selecting the Director configuration option 3. Customer configures Director VM network settings 4. Customer browses to the Director UI and configures system settings and defines the topology
- 15. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 15
- 16. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 16
- 17. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 17 1. Customer downloads a single OVA from Cisco 2. Customer deploys the Director VM from the OVA by selecting the Director configuration option 3. Customer configures Director VM network settings 4. Customer browses to the Director UI and configures system settings and defines the topology 5. Customer deploy all VMs defined in the topology from the OVA by selecting the appropriate configuration
- 18. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 18 1. Customer downloads a single OVA from Cisco 2. Customer deploys the Director VM from the OVA by selecting the Director configuration option 3. Customer configures Director VM network settings 4. Customer browses to the Director UI and configures system settings and defines the topology 5. Customer deploy all VMs defined in the topology from the OVA by selecting the appropriate configuration • System handles all other setup and configuration
- 19. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 19 1. Customer download a single release image from file from Cisco
- 20. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 20 1. Customer download a single release image from file from Cisco 2. Customer uploads the release image to the Director
- 21. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 21
- 22. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 22 1. Customer download a single release image from file from Cisco 2. Customer uploads the release image to the Director 3. Customer clicks upgrade
- 23. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 23 1. Customer download a single release image from file from Cisco 2. Customer uploads the release image to the Director 3. Customer clicks upgrade • System handles the upgrade
- 24. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 24
- 25. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 25
- 26. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 26
- 27. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 27 OVF OS Bootstrap Config Network Settings Director FQDN Glue Code
- 28. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 28
- 29. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 29 • Package everything as an RPM • RPMs distributed from Director via Yum • Controller is a Puppet Master Director OS Analytics Store OS Puppet Master Yum
- 30. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 30 • Versioned build artifact containing: - All RPMs for all roles - All Puppet modules • Upgrade mechanics: - New image uploaded to Director - Old image unmounted and new image mounted - Director calls Puppet run for all hosts - On success the updated version is stored in on VM file system - Salt used to fetch version number from all VMs
- 31. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 31 Director OS Analytics Store QUAD OS JSON Store QUAD OS RDBMS Store QUAD OS Graph Store QUAD OS Rsyslog • Rsyslog - High performance - Reliable - Part of CentOS 6 Rsyslogx Rsyslogx Rsyslogx Rsyslogx
- 32. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 32 Director OS Analytics Store OS BSON Store OS RDBMS Store OS Social Graph OS monit monit monit monit Nagios • Nagios: - Reliable - Extremely flexible - Available in EPEL • Monit - Reliable - Flexible and extensible - Enables self healing - HTTP API - Available in EPEL
- 33. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 33 Director OS Analytics Store OS BSON Store OS RDBMS Store OS Social Graph OS Graphite collectd collectd collectd collectd • Graphite: - Reliable - Extremely flexible - Available in EPEL • Collectd - Reliable - 100+ plugins - Extensible - In EPEL
- 34. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 34
- 35. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 35 Phase 1 Mid 2010 • 20GB VM • CentOS 5 • deployer.zip • Monitoring 1.0 • 100% proprietary • Time to market focused • No DevOps tooling experience in the team
- 36. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 36 Phase 1 Mid 2010 • 20GB VM • CentOS 5 • deployer.zip • Monitoring 1.0 • 100% proprietary • Time to market focused • No DevOps tooling experience in the team Phase 2 Early 2011 • RPM + Yum • Puppet • Release Image • Reduce VM footprint • Major pushback on Puppet • Your team matters
- 37. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 37 Phase 1 Mid 2010 • 20GB VM • CentOS 5 • deployer.zip • Monitoring 1.0 • 100% proprietary • Time to market focused • No DevOps tooling experience in the team Phase 2 Early 2011 • RPM + Yum • Puppet • Release Image • Reduce VM footprint • Major pushback on Puppet • Your team matters Phase 3 Mid 2011 • Director REST API • OVF Properties • Unified Logging w/ Rsyslog + Scribe • Rsyslog was resisted due to name association with syslog
- 38. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 38 Phase 1 Mid 2010 • 20GB VM • CentOS 5 • deployer.zip • Monitoring 1.0 • 100% proprietary • Time to market focused • No DevOps tooling experience in the team Phase 2 Early 2011 • RPM + Yum • Puppet • Release Image • Reduce VM footprint • Major pushback on Puppet • Your team matters Phase 3 Mid 2011 • Director REST API • OVF Properties • Unified Logging w/ Rsyslog + Scribe • Rsyslog was resisted due to name association with syslog Phase 4 Early 2012 • CentOS 6 upgrade • Monit • Salt • Remove Scribe • Maintain fewer libraries • Further reduce VM footprint • Started replacing Monitoring 1.0 • Salt proved buggy
- 39. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 39 Phase 1 Mid 2010 • 20GB VM • CentOS 5 • deployer.zip • Monitoring 1.0 • 100% proprietary • Time to market focused • No DevOps tooling experience in the team Phase 2 Early 2011 • RPM + Yum • Puppet • Release Image • Reduce VM footprint • Major pushback on Puppet • Your team matters Phase 3 Mid 2011 • Director REST API • OVF Properties • Unified Logging w/ Rsyslog + Scribe • Rsyslog was resisted due to name association with syslog Phase 4 Early 2012 • CentOS 6 upgrade • Monit • Salt • Remove Scribe • Maintain fewer libraries • Further reduce VM footprint • Started replacing Monitoring 1.0 • Salt proved buggy Phase 5 Late 2012 • Replaced Monitoring 1.0 with Nagios, Collectd, Graphite • Last minute decision to go with Graphite • Puppet code spaghetti • VM tampering undercutting upgrade reliability • Other Cisco groups want to reuse our code
- 40. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 40 Phase 1 Mid 2010 • 20GB VM • CentOS 5 • deployer.zip • Monitoring 1.0 • 100% proprietary • Time to market focused • No DevOps tooling experience in the team Phase 2 Early 2011 • RPM + Yum • Puppet • Release Image • Reduce VM footprint • Major pushback on Puppet • Your team matters Phase 3 Mid 2011 • Director REST API • OVF Properties • Unified Logging w/ Rsyslog + Scribe • Rsyslog was resisted due to name association with syslog Phase 4 Early 2012 • CentOS 6 upgrade • Monit • Salt • Remove Scribe • Maintain fewer libraries • Further reduce VM footprint • Started replacing Monitoring 1.0 • Salt proved buggy Phase 5 Late 2012 • Replaced Monitoring 1.0 with Nagios, Collectd, Graphite • Last minute decision to go with Graphite • Puppet code spaghetti • VM tampering undercutting upgrade reliability • Other Cisco groups want to reuse our code Phase 6 Started 2012 • Generic programmable Controller • Deploy VMs via IaaS API • Orchestrate deploy and upgrade with Fabric • Baked VM images • Hiera for module parameters
- 41. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 41
- 42. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 42 Keep it simple • 80/20 rule • Build only what you need • What works is boring Learn and adapt • Don’t fall in love with what you’ve built Unix tool chain pattern • Loosely couple with best of bread tools • End-to-end solutions lock you in
- 43. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 43 • Excellent at configuration management • Bulletproof reliability • Long term investment protection • Vibrant community and ecosystem • Commitment to open source
- 44. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 44 • Alex Honor of DTO Solutions and Simplify Ops - Insightful program leadership - Walking encyclopedia of proven patterns - Always there when I’m in a jam
- 45. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 45