Gossip & Key Value Store
6 likes876 views
The document discusses the gossip protocol and key-value store, detailing their definitions, applications, and specific implementations like Cassandra. It explains how gossip protocols facilitate communication in distributed systems, while key-value stores manage data, highlighting Cassandra's architecture and adherence to the CAP theorem. The document also emphasizes the significance of consistency, availability, and partition tolerance in distributed databases, along with further learning resources.
![Cassandra Architecture ..
Replication Strategy
Simple Strategy: uses the Partitioner, of which there are two kinds
RandomPartitioner: Chord-like hash partitioning
ByteOrderedPartitioner: Assigns ranges of keys to servers.
Easier for range queries (e.g., get me all twitter users starting with
[a-b])
NetworkTopologyStrategy NetworkTopologyStrategy: for multi-DC
deployments
Two replicas per DC
Three replicas per DC
Per DC
First replica placed according to Partitioner
Then go clockwise around ring until you hit a dierent rack
Dr. SAJEEV G P Gossip Protocol Key-Value Store 20 / 1](https://image.slidesharecdn.com/facultyseminar-160416080533/85/Gossip-Key-Value-Store-20-320.jpg)
![References I
Atikoglu, B., Xu, Y., Frachtenberg, E., Jiang, S., and Paleczny, M.
Workload analysis of a large-scale key-value store.
In ACM SIGMETRICS Performance Evaluation Review (2012),
vol. 40, ACM, pp. 5364.
Datastax.
Python cassandra-driver[onlne] available:.
https://pypi.python.org/pypi/cassandra-driver (2015).
Gupta, I., and Meseguer, J.
Quantitative analysis of consistency in nosql key-value stores.
In Quantitative Evaluation of Systems: 12th International
Conference, QEST 2015, Madrid, Spain, September 1-3, 2015,
Proceedings (2015), vol. 9259, Springer, p. 228.
Dr. SAJEEV G P Gossip Protocol Key-Value Store 28 / 1](https://image.slidesharecdn.com/facultyseminar-160416080533/85/Gossip-Key-Value-Store-28-320.jpg)
Gossip & Key Value Store
- 1. Gossip Protocol & Key-Value Store Theory and Practice Dr. SAJEEV G P April 16, 2016 Dr. SAJEEV G P Gossip Protocol & Key-Value Store 1 / 1
- 2. Contents Outline Background Introduction to Gossip Gossip Model Key-Value Store Cassandra CAP Theorem Further study Dr. SAJEEV G P Gossip Protocol & Key-Value Store 2 / 1
- 3. Background Dr. SAJEEV G P Gossip Protocol & Key-Value Store 3 / 1
- 4. Real World Applications Resume Youtube Video Last watched status.. Already Watched? Youtube Dr. SAJEEV G P Gossip Protocol & Key-Value Store 4 / 1
- 5. Some Real World Applications Facebook Search Term Search Amazon Search Amazon Recommendations Dr. SAJEEV G P Gossip Protocol & Key-Value Store 5 / 1
- 6. Cloud Platform App Engine Compute Engine Cloud storage Cloud BigTable Google Dataow Google Translate API Google BigQuery Cloud Prediction API Gossip Key-Value Store Gossip is for communication Key-Value store is the database Dr. SAJEEV G P Gossip Protocol Key-Value Store 6 / 1
- 7. Gossip Protocol Multicast Multicast is group communication where information is addressed to a group of destination computers simultaneously. Types of Casting Unicast Multicast Broadcast Multicast in application level Multicast in Network level Dr. SAJEEV G P Gossip Protocol Key-Value Store 7 / 1
- 8. Multicast Protocol: Centralized Tree Based Centralized Tree based Dr. SAJEEV G P Gossip Protocol Key-Value Store 8 / 1
- 9. Gossip Protocol: Epedemic multicast Dr. SAJEEV G P Gossip Protocol Key-Value Store 9 / 1
- 10. Gossip Analysis Basics Population of (n+1) individuals mixing homogeneously Contact rate between any individual pair is β At any time, each individual is either uninfected (numbering x) or infected (numbering y) Then, x0 = n , y0 = 1 and at all times x + y = n + 1 Infected-uninfected contact turns latter infected, and it stays infected Dr. SAJEEV G P Gossip Protocol Key-Value Store 10 / 1
- 11. Gossip Analysis Gossip Properties Lightweight in large groups Spreads quickly Fault-tolerant Terms n + 1 nodes x: # of uninfected nodes y: # of infected nodes x + y = n + 1 Continuous time .. dx dt = −βxy β contact rate = b n Solution: x = n(n+1) n+eβ(n+1)t y = (n+1) 1+ne−β(n+1)t Dr. SAJEEV G P Gossip Protocol Key-Value Store 11 / 1
- 12. Gossip Analysis .. No. of infected nodes .. At t = c log (n), y ≈ (n + 1) − 1 ncb−2 Low Latency Set c, b to be small numbers Within t = c log (n) rounds: all will receive the multicast except 1 ncb−2 Lightweight Each node has transmitted no more than: cb log (n) gossip messages. Fault tolerance With 50% of packet drop b ← b 2 : Takes twice as many rounds With 50% node failures n ← n 2 : Same as above Dr. SAJEEV G P Gossip Protocol Key-Value Store 12 / 1
- 13. Key-Value Stores Simplest form of database management systems. They store pairs of keys and values as well as retrieve values when a key is known. Examples twitter.com: Tweet id ⇒ information about tweet amazon.com: Item number ⇒ information about it kayak.com: Flight number ⇒ information about ight, e.g., availability yourbank.com: Ac number ⇒ information about it Dr. SAJEEV G P Gossip Protocol Key-Value Store 13 / 1
- 14. Key-Value Stores.. It's a dictionary data structure. NoSQL database Insert, lookup, and delete by key E.g., hash table, binary tree But distributed Key-Value stores reuse many techniques from DHTs NoSQL Databases Traditional RDBMS.. Schema-based, i.e., structured tables Primary key that is unique within that table Queried using SQL , Supports joins Dr. SAJEEV G P Gossip Protocol Key-Value Store 14 / 1
- 15. Todays Workload Big Data Era .. Data: Large and unstructured Lots of random reads and writes Sometimes write-heavy Foreign keys rarely needed Joins infrequent Need of todays Workload Speed Avoid Single Point of Failure (SPOF) Low TCO (Total cost of operation) Fewer system administrators Incremental scalability, Scale out, not up Dr. SAJEEV G P Gossip Protocol Key-Value Store 15 / 1
- 16. Key-Value or NoSQL NoSQL systems often use column-oriented storage RDBMSs store an entire row together (on disk or at a server) NoSQL systems typically store a column together (or a group of columns). Entries within a column are indexed and easy to locate, given a key (and vice-versa) Why useful? Range searches within a column are fast since you don't need to fetch the entire database E.g., get me all the blog-ids from the blog table that were updated within the past month Search in the the last-updated column, fetch corresponding blog-id column Don't need to fetch the other columns Dr. SAJEEV G P Gossip Protocol Key-Value Store 16 / 1
- 17. CASSANDRA What is CASSANDRA A distributed key-value store Intended to run in a data-center (and also across DCs) Originally designed at Facebook Open-sourced later, today an Apache project In use Some of the companies that use Cassandra in their production clusters IBM, Adobe, HP, eBay, Ericsson, Symantec Twitter, Spotify PBS Kids Netix: uses Cassandra to keep track of your current position in the video youâre watching Dr. SAJEEV G P Gossip Protocol Key-Value Store 17 / 1
- 18. CASSANDRA Objectives and Functions Performance Availability Scalability Fault-Tolerance P2P Cluster Decentralized design Each node has the same role No single point of failure Avoids issues of master-slave DBMS's No bottlenecking Dr. SAJEEV G P Gossip Protocol Key-Value Store 18 / 1
- 19. Cassandra Architecture DHT Like Dr. SAJEEV G P Gossip Protocol Key-Value Store 19 / 1
- 20. Cassandra Architecture .. Replication Strategy Simple Strategy: uses the Partitioner, of which there are two kinds RandomPartitioner: Chord-like hash partitioning ByteOrderedPartitioner: Assigns ranges of keys to servers. Easier for range queries (e.g., get me all twitter users starting with [a-b]) NetworkTopologyStrategy NetworkTopologyStrategy: for multi-DC deployments Two replicas per DC Three replicas per DC Per DC First replica placed according to Partitioner Then go clockwise around ring until you hit a dierent rack Dr. SAJEEV G P Gossip Protocol Key-Value Store 20 / 1
- 21. Cassandra Architecture .. At node On receiving a write Log it in disk commit log (for failure recovery) Make changes to appropriate memtables Memtable = In-memory representation of multiple key- value pairs Later, when memtable is full or old, ush to disk Data le: An SSTable (Sorted String Table) - list of key-value pairs, sorted by key Index le: An SSTable of (key, position in data sstable) pairs And a Bloom lter (for ecient search) Dr. SAJEEV G P Gossip Protocol Key-Value Store 21 / 1
- 22. Cassandra Architecture .. CAP C consistency: all nodes see same data at any time, or reads return latest written value by any client A vailability: the system allows operations all the time, and operations return quickly P artition-tolerance: the system continues to work in spite of network partitions CAP Theorem In a distributed system you can satisfy at most 2 out of the 3 guarantees. Eventual consistency : weak consistency model Dr. SAJEEV G P Gossip Protocol Key-Value Store 22 / 1
- 23. Cassandra Architecture .. CAP Theorem CAP Cassandra Consistency Cassandra chooses Consistency and Availability Cassandra has consistency levels Dr. SAJEEV G P Gossip Protocol Key-Value Store 23 / 1
- 24. CAP Theorem .. Consistency: No. of Replicas .. Client is allowed to choose a consistency level for each operation (read/write) ANY: any server, Fastest ALL: all replicas, Ensures strong consistency, but slowest ONE: at least one replica, Faster than ALL QUORUM: quorum across all replicas in all datacenters (DCs) Dr. SAJEEV G P Gossip Protocol Key-Value Store 24 / 1
- 25. Cassandra Architecture .. Cassandra Gossip Dr. SAJEEV G P Gossip Protocol Key-Value Store 25 / 1
- 26. For further learning Gossip Applications Distributed Computing/Networking Information Dissemination Gossip Learning Distributed Data Research/Projects Gossip Protocol: Anti-entropy (simple epidemics) Rumor mongering (complex epidemics) Eager epidemic dissemination Key-Value Store: Cassandra Redis python-Cassandra Dr. SAJEEV G P Gossip Protocol Key-Value Store 26 / 1
- 27. Further Learning .. Cassandra Resources http://cassandra.apache.org/ http://academy.datastax.com/ http://www.planetcassandra.org/ Use locally installed apache-cassandra-3.4 Use Cassandra Cluster service Python-Cassandra Cassndra Cluster Setup cqlengine: Cassandra CQL object mapper for Python Python Application Dr. SAJEEV G P Gossip Protocol Key-Value Store 27 / 1
- 28. References I Atikoglu, B., Xu, Y., Frachtenberg, E., Jiang, S., and Paleczny, M. Workload analysis of a large-scale key-value store. In ACM SIGMETRICS Performance Evaluation Review (2012), vol. 40, ACM, pp. 5364. Datastax. Python cassandra-driver[onlne] available:. https://pypi.python.org/pypi/cassandra-driver (2015). Gupta, I., and Meseguer, J. Quantitative analysis of consistency in nosql key-value stores. In Quantitative Evaluation of Systems: 12th International Conference, QEST 2015, Madrid, Spain, September 1-3, 2015, Proceedings (2015), vol. 9259, Springer, p. 228. Dr. SAJEEV G P Gossip Protocol Key-Value Store 28 / 1
- 29. References II Jenkins, K., Hopkinson, K., and Birman, K. A gossip protocol for subgroup multicast. In Distributed Computing Systems Workshop, 2001 International Conference on (2001), IEEE, pp. 2530. Lakshman, A., and Malik, P. Cassandra: A decentralized structured storage system. SIGOPS Oper. Syst. Rev. 44, 2 (Apr. 2010), 3540. van der Hoek, W. A framework for epistemic gossip protocols. In Multi-Agent Systems: 12th European Conference, EUMAS 2014, Prague, Czech Republic, December 18-19, 2014, Revised Selected Papers (2015), vol. 8953, Springer, p. 193. Dr. SAJEEV G P Gossip Protocol Key-Value Store 29 / 1
- 30. Thank You.. Dr. SAJEEV G P Gossip Protocol Key-Value Store 30 / 1