Building RightScale's Globally Distributed Datastore - RightScale Compute 2013
Download as PPTX, PDF0 likes521 views
The document discusses RightScale's design and implementation of their globally distributed datastore, focusing on data taxonomy, storage design, and considerations for scalability, high availability, and disaster recovery. It contrasts the use of RDBMS (MySQL) and NoSQL (Cassandra), outlining their respective benefits in terms of transactionality, availability, and queryability. The presentation also covers how data is structured, accessed, and the strategies employed for maintaining availability and fault isolation across different cloud environments.
Building RightScale's Globally Distributed Datastore - RightScale Compute 2013
- 1. april25-26 sanfrancisco cloud success starts here Building RightScale’s Globally Distributed Datastore Josep M. Blanquer, Chief Architect
- 2. #2#2 #RightscaleCompute In this talk… • Intro • Data Taxonomy • Data Storage Design • Scale, HA and DR considerations • Conclusion
- 3. #3#3 #RightscaleCompute Intro: Expectations and scope What this is and what is not • IS a talk about: • how RightScale has designed and implemented its backing datastores • …for a few of the most representative internal systems • …with the rationale behind it • Is NOT a talk about • RightScale’s overall architecture • Nodes or hosts, it’s about Systems • RightScale’s data modeling
- 4. #4#4 #RightscaleCompute Intro: Tools and Technologies • RightScale uses a mix of RDBMS and NoSQL technologies: • MySQL , Cassandra and S3 (for backups and archiving) • Transactionality: • MySQL: strong ACID properties • Cassandra: no Atomicity, eventually Consistent, some Isolation, Durable • Availability: • MySQL: async replication. Master-SlaveN or Master-Master • Cassandra: Distributed, master-less, highly-replicated (multi-DC) • Queryability: • MySQL: Extremely flexible at adding indexes and changing data model • Cassandra: More difficult to change the querying patterns
- 5. #5#5 #RightscaleCompute Taxonomy of RightScale’s Data Representative systems with different data semantics: Global Objects Marketplace Assets Dashboard Objects Audits Tags Recent Events Cloud Polling Data Routing Data Monitoring/Syslog
- 6. #6#6 #RightscaleCompute Taxonomy of RightScale’s Data Representative systems with different data semantics: Global Objects Marketplace Assets Dashboard Objects Audits Tags Recent Events Cloud Polling Data Routing Data Monitoring/Syslog Common across accounts: Users Plans Settings MultiCloud Marketplace: Published Assets Sharing Groups …
- 7. #7#7 #RightscaleCompute Taxonomy of RightScale’s Data Representative systems with different data semantics: Global Objects Marketplace Assets Dashboard Objects Audits Tags Recent Events Cloud Polling Data Routing Data Monitoring/Syslog Private to each account: Deployments Imported assets Alert Specifications Server Inputs Audit Tags User Events …
- 8. #8#8 #RightscaleCompute Taxonomy of RightScale’s Data Representative systems with different data semantics: Global Objects Marketplace Assets Dashboard Objects Audits Tags Recent Events Cloud Polling Data Routing Data Monitoring/Syslog Private to each account: Cloud resource states (cache) Cloud credentials
- 9. #9#9 #RightscaleCompute Taxonomy of RightScale’s Data Representative systems with different data semantics: Global Objects Marketplace Assets Dashboard Objects Audits Tags Recent Events Cloud Polling Data Routing Data Monitoring/Syslog Private to each account: Instance agents location Core agents location Agent action registry …
- 10. #10#10 #RightscaleCompute Taxonomy of RightScale’s Data Representative systems with different data semantics: Global Objects Marketplace Assets Dashboard Objects Audits Tags Recent Events Cloud Polling Data Routing Data Monitoring/Syslog Private to each account: Collected metric data Collected syslog data …
- 11. #11#11 #RightscaleCompute Taxonomy of RightScale’s Data UsersInstances Global Objects Marketplace Assets Dashboard Objects Audits Tags Recent Events Cloud Polling Data Routing Data Monitoring/Syslog Who uses the data? • Users through the Dash/API • Instances from the Cloud Data close to the Users Data close to the Cloud Data Placement
- 12. #12#12 #RightscaleCompute Taxonomy of RightScale’s DataX-acctAccount Global Objects Marketplace Assets Dashboard Objects Audits Tags Recent Events Cloud Polling Data Routing Data Monitoring/Syslog Which data do we need? • Data for all accounts • Data for a single account Data shared between accounts Data required within scope of a single account Data scope and containment
- 13. #13#13 Talk with the Experts. Users Taxonomy of RightScale’s Data Instances X-acctAccount Global Objects Marketplace Assets Dashboard Objects Audits Tags Recent Events Cloud Polling Data Routing Data Monitoring/Syslog Who uses the data? Proximity to User vs. Cloud Which data do we need? Scope of data available Close to cloud resources Account-shardable* data Close to user Account-shardable data Close to user Globally accessible data
- 15. #15#15 #RightscaleCompute UsersInstances global X-Account Custom replication Why custom? More control • Multiple sources • Individual columns • Apply transformations • Smart re-sync features Global: MySQL • ACID semantics • Master-Slave replication
- 16. #16#16 #RightscaleCompute UsersInstances Account global dash S3 events tags audit X-Account Dashboard: MySQL • ACID semantics • Master-SlaveN replication • Slave reads • Rows tagged by account Other systems: Cassandra • Simpler Key-Value access • Great scalability • Great replica control • High write availability • Time-to-live expiration as cache • Rows tagged by account Data archive: S3 • Low read rate • Globally accessible
- 17. #17#17 #RightscaleCompute UsersInstances Account global dash S3 events tags audit X-Account dash events tags audit So we can horizontally scale our dashboard by partitioning objects based on account groups: Clusters
- 18. #18#18 #RightscaleCompute Users AccountCluster1 dash S3 events tags audit ClusterN dash S3 events tags audit Account Set 1 Account Set 2 RightScale Accounts Cluster3 dash S3 events tags audit … Features: • 1 cluster: N accounts • 1 account: 1 home • Migratable accounts Benefits: • Great horizontal growth • Better failure isolation • Independent scale • Load rebalancing • Versionable code • Differentiated service
- 20. #20#20 #RightscaleCompute routing polling monitor routing polling monitor UsersInstances Account global dash S3 events tags audit X-Account And partition our cloud objects based on the cloud the instances of an account run on: Islands
- 21. #21#21 #RightscaleCompute Account Instances Services co-located with resources Services co-located with resources Services co-located with resources routing polling monitor Island1 Island2 IslandN routing polling monitor routing polling monitor Cloud 1 Cloud 2 Cloud N
- 22. #22#22 #RightscaleCompute Account Instances Features: • 1 instance: 1 home island • 1 Island can serve N clouds • Core Agents: global data Benefits: • Close to cloud resources • Good failure isolation • As good as cloud • Good scale: global replicas across cassandra DCs routing polling monitor Island1 Island2 IslandN routing polling monitor routing polling monitor routing polling monitor routing polling monitor routing polling monitor Island1 Island2 IslandN Polling Clouds: MySQL • Master-Slave replication • Can port to NoSQL easily • Mostly a resource cache • But cloud partitionable Monitoring: Custom • Replicated files • Backup to S3 • Archive to S3 Routing: Cassandra • Simpler Key-Value access • Very high availability • Great scalability • Great replica control • Plus cross DC replication*
- 23. #23#23 #RightscaleCompute Users AccountCluster1 dash S3 events tags audit ClusterN dash S3 events tags audit Cluster3 dash S3 events tags audit … routing polling monitor routing polling monitor routing polling monitor Island1 Island2 IslandN Instances Different Geographies Different Clouds What if the cloud where the cluster is deployed on… Fails? What if the cloud where the island is deployed on… Fails?
- 24. #24#24 #RightscaleCompute Users AccountCluster1 dash S3 events tags audit ClusterN dash S3 events tags audit Cluster3 dash S3 events tags audit … routing polling monitor routing polling monitor routing polling monitor Island1 Island2 IslandN Instances Sister Clusters Full replica Features: • Each master has an extra remote slave • Each cluster in a pair is a DC replica of the other’s localring At Disaster Recovery time: • Apps are told to start serving an extra shard • No need to provision more infrastructure to recover (try to avoid since everybody is on the same boat) • New resources can be allocated over time to help offload existing ones
- 25. #25#25 #RightscaleCompute Conclusions • Shown that RightScale uses multiple database technologies: • RDBMS – MySQL for the ACID semantics and ‘queryability’ • Using a Master to N-Slaves for RO scale, and quick failure recovery • And ReadOnly Provisioning – To increase RO availability and scale remote systems • NoSQL: Cassandra for Availability and Scalability • for higher Read/Write availability within a cluster • For fully replicated regions across the globe (for Read/Write!) • Shown how RightScale uses them in different techniques • It partitions resource data into Islands based on cloud proximity • Can achieve in-cloud polling,and keep monitoring/syslog data storage next to instances • Can provide routing availability, colocated with instances for any world region • It partitions core data into Clusters based on account groups • To scale the core horizontally, and independently and achieve account isolation/differentiation • Enhances fault isolation: Assigning accounts to Clusters deployed away their cloud resources • It maintains cluster pairs (sister sites) • To recover from full cloud region failures • It doesn’t require massive amounts of new resources to recover
- 26. april25-26 sanfrancisco cloud success starts here Questions?