The computer science behind a modern disributed data store
0 likes1,639 views
The document discusses the computer science principles behind modern distributed data stores, highlighting the importance of resilience, consensus protocols like Paxos and Raft, and mechanisms for handling distributed ACID transactions. It emphasizes the challenges of managing consistency and operation in distributed systems, particularly with respect to network failures and clock synchronization. Additionally, it covers sorting algorithms and the implementation of log-structured merge trees for efficient data management.
The computer science behind a modern disributed data store
- 1. The computer science behind a modern distributed data store Max Neunhöffer Malaga, 19 May 2017 www.arangodb.com
- 2. Overview Topics Resilience and Consensus Sorting Log-structured Merge Trees Hybrid Logical Clocks Distributed ACID Transactions
- 3. Overview Topics Resilience and Consensus Sorting Log-structured Merge Trees Hybrid Logical Clocks Distributed ACID Transactions Bottom line: You need CompSci to implement a modern data store
- 4. Resilience and Consensus The Problem A modern data store is distributed,
- 5. Resilience and Consensus The Problem A modern data store is distributed, because it needs to scale out and/or be resilient.
- 6. Resilience and Consensus The Problem A modern data store is distributed, because it needs to scale out and/or be resilient. Different parts of the system need to agree on things.
- 7. Resilience and Consensus The Problem A modern data store is distributed, because it needs to scale out and/or be resilient. Different parts of the system need to agree on things. Consensus is the art to achieve this as well as possible in software. This is relatively easy, if things are good, but very hard, if:
- 8. Resilience and Consensus The Problem A modern data store is distributed, because it needs to scale out and/or be resilient. Different parts of the system need to agree on things. Consensus is the art to achieve this as well as possible in software. This is relatively easy, if things are good, but very hard, if: the network has outages, the network has dropped or delayed or duplicated packets, disks fail (and come back with corrupt data), machines fail (and come back with old data), racks fail (and come back with or without data).
- 9. Resilience and Consensus The Problem A modern data store is distributed, because it needs to scale out and/or be resilient. Different parts of the system need to agree on things. Consensus is the art to achieve this as well as possible in software. This is relatively easy, if things are good, but very hard, if: the network has outages, the network has dropped or delayed or duplicated packets, disks fail (and come back with corrupt data), machines fail (and come back with old data), racks fail (and come back with or without data). (And we have not even talked about malicious attacks and enemy action.)
- 10. Paxos and Raft Traditionally, one uses the Paxos Consensus Protocol (1998). More recently, Raft (2013) has been proposed. Paxos is a challenge to understand and to implement efficiently. Various variants exist. Raft is designed to be understandable.
- 11. Paxos and Raft Traditionally, one uses the Paxos Consensus Protocol (1998). More recently, Raft (2013) has been proposed. Paxos is a challenge to understand and to implement efficiently. Various variants exist. Raft is designed to be understandable. My advice: First try to understand Paxos for some time (do not implement it!), then enjoy the beauty of Raft,
- 12. Paxos and Raft Traditionally, one uses the Paxos Consensus Protocol (1998). More recently, Raft (2013) has been proposed. Paxos is a challenge to understand and to implement efficiently. Various variants exist. Raft is designed to be understandable. My advice: First try to understand Paxos for some time (do not implement it!), then enjoy the beauty of Raft, but do not implement it either!
- 13. Paxos and Raft Traditionally, one uses the Paxos Consensus Protocol (1998). More recently, Raft (2013) has been proposed. Paxos is a challenge to understand and to implement efficiently. Various variants exist. Raft is designed to be understandable. My advice: First try to understand Paxos for some time (do not implement it!), then enjoy the beauty of Raft, but do not implement it either! Use some battle-tested implementation you trust!
- 14. Paxos and Raft Traditionally, one uses the Paxos Consensus Protocol (1998). More recently, Raft (2013) has been proposed. Paxos is a challenge to understand and to implement efficiently. Various variants exist. Raft is designed to be understandable. My advice: First try to understand Paxos for some time (do not implement it!), then enjoy the beauty of Raft, but do not implement it either! Use some battle-tested implementation you trust! But most importantly: DO NOT TRY TO INVENT YOUR OWN!
- 15. Raft in a slide An odd number of servers each keep a persisted log of events.
- 16. Raft in a slide An odd number of servers each keep a persisted log of events. Everything is replicated to everybody.
- 17. Raft in a slide An odd number of servers each keep a persisted log of events. Everything is replicated to everybody. They democratically elect a leader with absolute majority.
- 18. Raft in a slide An odd number of servers each keep a persisted log of events. Everything is replicated to everybody. They democratically elect a leader with absolute majority. Only the leader may append to the replicated log.
- 19. Raft in a slide An odd number of servers each keep a persisted log of events. Everything is replicated to everybody. They democratically elect a leader with absolute majority. Only the leader may append to the replicated log. An append only counts when a majority has persisted and confirmed it.
- 20. Raft in a slide An odd number of servers each keep a persisted log of events. Everything is replicated to everybody. They democratically elect a leader with absolute majority. Only the leader may append to the replicated log. An append only counts when a majority has persisted and confirmed it. Very smart logic to ensure a unique leader and automatic recovery from failure.
- 21. Raft in a slide An odd number of servers each keep a persisted log of events. Everything is replicated to everybody. They democratically elect a leader with absolute majority. Only the leader may append to the replicated log. An append only counts when a majority has persisted and confirmed it. Very smart logic to ensure a unique leader and automatic recovery from failure. It is all a lot of fun to get right, but it is proven to work.
- 22. Raft in a slide An odd number of servers each keep a persisted log of events. Everything is replicated to everybody. They democratically elect a leader with absolute majority. Only the leader may append to the replicated log. An append only counts when a majority has persisted and confirmed it. Very smart logic to ensure a unique leader and automatic recovery from failure. It is all a lot of fun to get right, but it is proven to work. One puts a key/value store on top, the log contains the changes.
- 24. Sorting The Problem Data stores need indexes. In practice, we need to sort things.
- 25. Sorting The Problem Data stores need indexes. In practice, we need to sort things. Most published algorithms are rubbish on modern hardware.
- 26. Sorting The Problem Data stores need indexes. In practice, we need to sort things. Most published algorithms are rubbish on modern hardware. The problem is no longer the comparison computations but the data movement .
- 27. Sorting The Problem Data stores need indexes. In practice, we need to sort things. Most published algorithms are rubbish on modern hardware. The problem is no longer the comparison computations but the data movement . Since the time when I was a kid and have played with an Apple IIe, compute power in one core has increased by ×20000 a single memory access by ×40 and now we have 32 cores in a CPU this means computation has outpaced memory access by ×1280!
- 28. Idea for a parallel sorting algorithm: Merge Sort Min−Heap: sorted merged
- 29. Idea for a parallel sorting algorithm: Merge Sort Min−Heap: sorted merged Nearly all comparisons hit the L2 cache!
- 30. Log structured merge trees (LSM-trees) The Problem People rightfully expect from a data store, that it can hold more data than the available RAM, works well with SSDs and spinning rust, allows fast bulk inserts into large data sets, and provides fast reads in a hot set that fits into RAM.
- 31. Log structured merge trees (LSM-trees) The Problem People rightfully expect from a data store, that it can hold more data than the available RAM, works well with SSDs and spinning rust, allows fast bulk inserts into large data sets, and provides fast reads in a hot set that fits into RAM. Traditional B-tree based structures often fail to deliver with the last 2.
- 32. Log structured merge trees (LSM-trees) (Source: http://www.benstopford.com/2015/02/14/log-structured-merge-trees/, Author: Ben Stopford, License: Creative Commons)
- 33. Log structured merge trees (LSM-trees) LSM-trees — summary writes first go into memtables, all files are sorted and immutable, compaction happens in the background, merge sort can be used, all writes use sequential I/O, Bloom filters or Cuckoo filters for fast reads, =⇒ good write throughput and reasonable read performance, used in ArangoDB, BigTable, Cassandra, HBase, InfluxDB, LevelDB, MongoDB, RocksDB, SQLite4 and WiredTiger, etc.
- 34. Hybrid Logical Clocks (HLC) The Problem Clocks in different nodes of distributed systems are not in sync.
- 35. Hybrid Logical Clocks (HLC) The Problem Clocks in different nodes of distributed systems are not in sync. general relativity poses fundamental obstructions to synchronizity, in practice, clock skew happens, Google can use atomic clocks, even with NTP (network time protocol) we have to live with ≈ 20ms.
- 36. Hybrid Logical Clocks (HLC) The Problem Clocks in different nodes of distributed systems are not in sync. general relativity poses fundamental obstructions to synchronizity, in practice, clock skew happens, Google can use atomic clocks, even with NTP (network time protocol) we have to live with ≈ 20ms. Therefore, we cannot compare time stamps from different nodes!
- 37. Hybrid Logical Clocks (HLC) The Problem Clocks in different nodes of distributed systems are not in sync. general relativity poses fundamental obstructions to synchronizity, in practice, clock skew happens, Google can use atomic clocks, even with NTP (network time protocol) we have to live with ≈ 20ms. Therefore, we cannot compare time stamps from different nodes! Why would this help? establish “happened after” relationship between events, e.g. for conflict resolution, log sorting, detecting network delays, time to live could be implemented easily.
- 38. Hybrid Logical Clocks (HLC) The Idea Every computer has a local clock, and we use NTP to synchronize.
- 39. Hybrid Logical Clocks (HLC) The Idea Every computer has a local clock, and we use NTP to synchronize. If two events on different machines are linked by causality, the cause should have a smaller time stamp than the effect.
- 40. Hybrid Logical Clocks (HLC) The Idea Every computer has a local clock, and we use NTP to synchronize. If two events on different machines are linked by causality, the cause should have a smaller time stamp than the effect. causality ⇐⇒ a message is sent Send a time stamp with every message. The HLC always returns a value > max(local clock, largest time stamp ever seen).
- 41. Hybrid Logical Clocks (HLC) The Idea Every computer has a local clock, and we use NTP to synchronize. If two events on different machines are linked by causality, the cause should have a smaller time stamp than the effect. causality ⇐⇒ a message is sent Send a time stamp with every message. The HLC always returns a value > max(local clock, largest time stamp ever seen). Causality is preserved, time can “catch up” with logical time eventually. http://muratbuffalo.blogspot.com.es/2014/07/hybrid-logical-clocks.html
- 42. Distributed ACID Transactions Atomic either happens in its entirety or not at all Consistent reading sees a consistent state, writing preser- vers consistency Isolated concurrent transactions do not see each other Durable committed writes are preserved after shut- down and crashes
- 43. Distributed ACID Transactions Atomic either happens in its entirety or not at all Consistent reading sees a consistent state, writing preser- vers consistency Isolated concurrent transactions do not see each other Durable committed writes are preserved after shut- down and crashes (All relatively doable when transactions happen one after another!)
- 44. Distributed ACID Transactions The Problem In a distributed system: How to make sure, that all nodes agree on whether the transaction has happened? (Atomicity)
- 45. Distributed ACID Transactions The Problem In a distributed system: How to make sure, that all nodes agree on whether the transaction has happened? (Atomicity) How to create a consistent snapshot across nodes? (Consistency)
- 46. Distributed ACID Transactions The Problem In a distributed system: How to make sure, that all nodes agree on whether the transaction has happened? (Atomicity) How to create a consistent snapshot across nodes? (Consistency) How to hide ongoing activities until commit? (Isolation)
- 47. Distributed ACID Transactions The Problem In a distributed system: How to make sure, that all nodes agree on whether the transaction has happened? (Atomicity) How to create a consistent snapshot across nodes? (Consistency) How to hide ongoing activities until commit? (Isolation) How to handle lost nodes? (Durability)
- 48. Distributed ACID Transactions The Problem In a distributed system: How to make sure, that all nodes agree on whether the transaction has happened? (Atomicity) How to create a consistent snapshot across nodes? (Consistency) How to hide ongoing activities until commit? (Isolation) How to handle lost nodes? (Durability) We have to take replication, resilience and failover into account.
- 49. Distributed ACID Transactions WITHOUT Distributed databases without ACID transactions: ArangoDB, BigTable, Couchbase, Datastax, Dynamo, Elastic, HBase, MongoDB, RethinkDB, Riak, and lots more . . . WITH Distributed databases with ACID transactions: (ArangoDB,) CockroachDB, FoundationDB, Spanner
- 50. Distributed ACID Transactions WITHOUT Distributed databases without ACID transactions: ArangoDB, BigTable, Couchbase, Datastax, Dynamo, Elastic, HBase, MongoDB, RethinkDB, Riak, and lots more . . . WITH Distributed databases with ACID transactions: (ArangoDB,) CockroachDB, FoundationDB, Spanner =⇒ Very few distributed engines promise ACID, because this is hard!
- 51. Distributed ACID Transactions Basic Idea Use Multi Version Concurrency Control (MVCC), i.e. multiple revisions of a data item are kept.
- 52. Distributed ACID Transactions Basic Idea Use Multi Version Concurrency Control (MVCC), i.e. multiple revisions of a data item are kept. Do writes and replication decentrally and distributed, without them becoming visible from other transactions.
- 53. Distributed ACID Transactions Basic Idea Use Multi Version Concurrency Control (MVCC), i.e. multiple revisions of a data item are kept. Do writes and replication decentrally and distributed, without them becoming visible from other transactions. Then have some place, where there is a switch, which decides when the transaction becomes visible.
- 54. Distributed ACID Transactions Basic Idea Use Multi Version Concurrency Control (MVCC), i.e. multiple revisions of a data item are kept. Do writes and replication decentrally and distributed, without them becoming visible from other transactions. Then have some place, where there is a switch, which decides when the transaction becomes visible. These “switches” need to be persisted somewhere (durability), scale out (no bottleneck for commit/abort), be replicated (no single point of failure), be resilient in case of fail-over (fault-tolerance).
- 55. Distributed ACID Transactions Basic Idea Use Multi Version Concurrency Control (MVCC), i.e. multiple revisions of a data item are kept. Do writes and replication decentrally and distributed, without them becoming visible from other transactions. Then have some place, where there is a switch, which decides when the transaction becomes visible. These “switches” need to be persisted somewhere (durability), scale out (no bottleneck for commit/abort), be replicated (no single point of failure), be resilient in case of fail-over (fault-tolerance). Transaction visibility needs to be implemented (MVCC), time stamps play a crucial role.