Managing RightScale on RightScale
Download as PPTX, PDF2 likes524 views
RightScale uses RightScale to manage its own infrastructure including production, staging, development, and meta systems. The production environment has hundreds of servers running in various clouds to provide high availability. Upgrades are done by launching new servers with the latest code while keeping existing servers available as a rollback option. Monitoring is extensive and includes monitoring the monitoring systems. Key RightScale features like server templates and the library are very useful but some areas could be improved.
Managing RightScale on RightScale
- 1. Managing RightScale on RightScaleRafael H. SaavedraVP of Engineering
- 2. TopicsRightScale managed by RightScaleMeta, production, staging & developmentAn overview of the production systemQuisCustodietIpsosCustodesDeploying RightScale – best practices What we love about using RightScale Features that are difficult to use
- 3. RightScale: Cloud Management PlatformRightScale ProductionCustomer ACustomer DCustomer BCustomer C
- 4. RightScale: Cloud Management PlatformRightScale Meta ProductionRightScaleProductionRightScaleStagingRightScaleDevelopmentCustomer ACustomer DRightScaleDevelopment
- 5. A multitude of RightScale systemsMeta Production currently lives outside the cloud Use only to manage the production systemOnly RightScale ops accountsProduction: my.rightscale.comReaching 200 servers, a large fraction in EC2 us-eastServers in every cloud to achieve high availabilityServers allocated in well defined availability zonesA few staging systems used for integration and QAAd hoc systems for performance testing, demos, betasMany development systems with simplified configurationsA development system at the click of a button
- 6. Significant increase in cloud usage
- 7. Some interesting RightScale numbers1.65M servers launched by RightScaleRightScale continuously monitors more than 60k serversEvery day at RightScale:2,000 array resize actions are executed35,000 alert escalations are triggered20,000 escalation emails are sent to users9.0TB of monitoring data is exchange with our servers1.6TB of logging data is sent to our servers
- 8. RightScale production – simplifiedothersMain AppFront EndsloggingAPIdashboarddatabasesdaemonsDB MastermonitoringDB Slavemirrors
- 9. What is that our users do?Dashboard, API, monitoring graphs & event notificationsMost of the requests are monitoring updates 85% (70%)Dashboard and API represent 7% of requests but 26% of traffic
- 10. We eat our own dog foodProduction servers organized into independent deploymentsCore servers: frontends, core/api servers, databases, daemons
- 11. We eat our own dog foodExtensive use of security groups to isolate serversServerTemplates are maintained for each major releaseAbility to launch exact configurations of past versions
- 12. Monitoring, alerts & escalationsMonitor as much as possible, what is relevant and display it in insightful waysThe need to quickly detect patterns and abnormalitiesProactively eliminate the conditions that raise critical alertsNo broken windows policy APIs Cores
- 13. QuisCustodietIpsosCustodes?*The need to monitor the monitoring and alerting systemsExtensive use of alerts to monitor the responsiveness of all the RightScale serversInstance and EBS failures gives us headachesDecoupling the meta & production monitoring and alerting systems* Who watches the watchmen?
- 14. How to Monitor hundreds of servers?Starting to use stacked graphs & heat mapsThe need to quickly detect patterns and abnormalities
- 15. Our favorite RightScale featuresRightImages: never again the need to build custom imagesInput inheritance: makes it easy to keep the configurations of dozens of servers in sync ServerTemplates: very easy to reproduce configurations in production, staging and developmentThe Library: there is always an example of something new that can be adapted to our needsMonitoring: easy to make a collectdplugins to monitor just about anything
- 16. Our not so favorite featuresServerTemplate inputs: powerful but too many of them make templates difficult to useRevision management: a way to go to make users aware of new revisions and version and how to updateThe Library: checking out new resources from library is not easyAlerts: they work pretty well but they are not easy to configure, in particular, custom ones
- 17. Best practices: upgrading RightScaleAvoid upgrading existing servers; instead launch fresh ones with new software (fail forward)Not possible on some components, e.g. monitoring servers, which are in the hundredsThe cost of duplicating servers is minimal Old servers can take over in case something goes wrong Launch additional slaves to capture recovery pointsOne slave continues to replicate in case of master failureAnother slave is frozen at upgrade point – can rollback by failing overDon’t forget to take snapshots in case of major failure
- 18. Upgrading RightScale: Step by StepFront EndsMain AppMain Appservers with new codetake snapshot at cutoffDatabasesDB MasterDB SlaveDB Slavestop replicationcut access to sitestop all access to DBservers with old code
- 19. Upgrading RightScale: Step by StepFront EndsMain AppMain Appservers with new codesnapshot at cutoffreconnect all serversDatabasesDB MasterDB SlaveDB Slavestop replicationservers with old code
- 20. Upgrading RightScale: Step by StepFront EndsMain AppMain Appservers with new codereconnect all serversopen access to siteDatabasesDB MasterDB SlaveDB Slaveservers with old code
Editor's Notes
- #3: RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure. In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others. Specifically, we'll walk through the steps to share and update ServerTemplates across your organization. We'll also show you how to publish ServerTemplates publicly for the whole world to use. This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
- #4: RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure. In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others. Specifically, we'll walk through the steps to share and update ServerTemplates across your organization. We'll also show you how to publish ServerTemplates publicly for the whole world to use. This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
- #5: RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure. In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others. Specifically, we'll walk through the steps to share and update ServerTemplates across your organization. We'll also show you how to publish ServerTemplates publicly for the whole world to use. This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
- #6: RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure. In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others. Specifically, we'll walk through the steps to share and update ServerTemplates across your organization. We'll also show you how to publish ServerTemplates publicly for the whole world to use. This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
- #7: RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure. In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others. Specifically, we'll walk through the steps to share and update ServerTemplates across your organization. We'll also show you how to publish ServerTemplates publicly for the whole world to use. This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
- #8: RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure. In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others. Specifically, we'll walk through the steps to share and update ServerTemplates across your organization. We'll also show you how to publish ServerTemplates publicly for the whole world to use. This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
- #9: RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure. In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others. Specifically, we'll walk through the steps to share and update ServerTemplates across your organization. We'll also show you how to publish ServerTemplates publicly for the whole world to use. This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
- #10: RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure. In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others. Specifically, we'll walk through the steps to share and update ServerTemplates across your organization. We'll also show you how to publish ServerTemplates publicly for the whole world to use. This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
- #11: RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure. In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others. Specifically, we'll walk through the steps to share and update ServerTemplates across your organization. We'll also show you how to publish ServerTemplates publicly for the whole world to use. This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
- #12: RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure. In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others. Specifically, we'll walk through the steps to share and update ServerTemplates across your organization. We'll also show you how to publish ServerTemplates publicly for the whole world to use. This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
- #13: RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure. In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others. Specifically, we'll walk through the steps to share and update ServerTemplates across your organization. We'll also show you how to publish ServerTemplates publicly for the whole world to use. This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
- #14: RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure. In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others. Specifically, we'll walk through the steps to share and update ServerTemplates across your organization. We'll also show you how to publish ServerTemplates publicly for the whole world to use. This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
- #15: RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure. In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others. Specifically, we'll walk through the steps to share and update ServerTemplates across your organization. We'll also show you how to publish ServerTemplates publicly for the whole world to use. This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
- #16: RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure. In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others. Specifically, we'll walk through the steps to share and update ServerTemplates across your organization. We'll also show you how to publish ServerTemplates publicly for the whole world to use. This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
- #17: RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure. In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others. Specifically, we'll walk through the steps to share and update ServerTemplates across your organization. We'll also show you how to publish ServerTemplates publicly for the whole world to use. This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
- #18: RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure. In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others. Specifically, we'll walk through the steps to share and update ServerTemplates across your organization. We'll also show you how to publish ServerTemplates publicly for the whole world to use. This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
- #19: RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure. In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others. Specifically, we'll walk through the steps to share and update ServerTemplates across your organization. We'll also show you how to publish ServerTemplates publicly for the whole world to use. This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
- #20: RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure. In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others. Specifically, we'll walk through the steps to share and update ServerTemplates across your organization. We'll also show you how to publish ServerTemplates publicly for the whole world to use. This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
- #21: RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure. In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others. Specifically, we'll walk through the steps to share and update ServerTemplates across your organization. We'll also show you how to publish ServerTemplates publicly for the whole world to use. This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.