CAP Theorem and Split Brain Syndrome
Download as PPTX, PDF2 likes1,200 views
The CAP theorem, formulated by Eric Brewer, states that in distributed systems, it is impossible to achieve consistency, availability, and partition tolerance simultaneously, forcing designers to prioritize two out of the three. Misunderstanding the theorem can lead to poor design choices, especially as the future direction of databases leans heavily towards distributed architectures. Additionally, the document explores eventual consistency, real-world examples like Dropbox, and issues related to split-brain syndrome in distributed systems.
CAP Theorem and Split Brain Syndrome
- 1. CAP Theorem & Split Brain Syndrome CS5242 Software Development on Cloud Platforms Dilum Bandara, PhD Slides adapted from CSE 40822-Cloud Computing-Fall 2014 by Prof. Dong Wang and Reliable Distributed Systems by Ken Birman
- 2. CAP Theorem • Conjectured by Prof. Eric Brewer at PODC (Principle of Distributed Computing) 2000 keynote talk • Described the trade-offs involved in distributed system • It is impossible for a web service to provide following 3 guarantees at the same time: • Consistency • Availability • Partition-tolerance 2
- 3. CAP Theorem • Consistency: – All nodes should see the same data at the same time • Availability: – Node failures do not prevent survivors from continuing to operate • Partition-tolerance: – System continues to operate despite network partitions • A distributed system can satisfy any 2 of these guarantees at the same time but not all 3 3
- 5. CAP Theorem • A simple example: Hotel Booking: are we double-booking same room? Bob Dong 5
- 6. CAP Theorem: Proof • 2002: Proven by Nancy Lynch & Seth Gilbert at MIT Gilbert, Seth, and Nancy Lynch. "Brewer's conjecture and the feasibility of consistent, available, partition-tolerant web services." ACM SIGACT News 33.2 (2002): 51-59. 6
- 7. CAP Theorem: Proof • A simple proof using 2 nodes: A B 7
- 8. CAP Theorem: Proof • A simple proof using 2 nodes: A B Not Consistent! Respond to client 8
- 9. CAP Theorem: Proof • A simple proof using 2 nodes: A B Not Partition Tolerant! A gets updated from B 9
- 10. Why this is important? • Future of databases is distributed (Big Data Trend, etc.) • CAP theorem describes the trade-offs involved in distributed systems • Proper understanding of CAP theorem is essential to making decisions about distributed database/system design • Misunderstanding can lead to erroneous or inappropriate design choices 10
- 11. Problem for Relational Database to Scale • RDMS is built on the principle of ACID – Atomicity, Consistency, Isolation, Durability • It implies that a truly distributed RDMSs should have availability, consistency & partition tolerance • Which unfortunately is impossible … 11
- 12. Revisit CAP Theorem C A P • Pick 2 out of 3 suggests there are 3 kinds of distributed systems: • CP • AP • CA Any problems? 12
- 13. A popular misconception: 2 out 3 • How about CA? • Can a distributed system (with unreliable network) really be not tolerant of partitions? C A 13
- 14. A few witnesses • Coda Hale, Yammer software engineer: – “Of the CAP theorem’s Consistency, Availability, and Partition Tolerance, Partition Tolerance is mandatory in distributed systems. You cannot not choose it.” • Werner Vogels, Amazon CTO – “An important observation is that in larger distributed-scale systems, network partitions are a given; therefore, consistency and availability cannot be achieved at the same time.” • Daneil Abadi, Co-founder of Hadapt – So in reality, there are only two types of systems ... I.e., if there is a partition, does the system give up availability or consistency? 14
- 15. CAP Theorem 12 year later • Prof. Eric Brewer: father of CAP theorem • “The “2 of 3” formulation was always misleading because it tended to oversimplify the tensions among properties. ... • CAP prohibits only a tiny part of the design space: perfect availability & consistency in the presence of partitions, which are rare.” http://www.infoq.com/articles/cap-twelve-years-later-how-the-rules-have-changed15
- 16. Consistency or Availability C A P • Consistency & Availability is not “binary” decision • AP systems relax consistency in favor of availability – but are not inconsistent • CP systems sacrifice availability for consistency- but are not unavailable • This suggests both AP & CP systems can offer a degree of consistency, & availability, as well as partition tolerance 16
- 17. AP: Best Effort Consistency • Example: – Web Caching – DNS • Trait: – Optimistic – Expiration/Time-to-live – Conflict resolution 17
- 18. CP: Best Effort Availability • Example: – Majority protocols – Distributed Locking (Google Chubby Lock service) • Trait: – Pessimistic locking – Make minority partition unavailable 18
- 19. Types of Consistency • Strong Consistency – After the update completes, any subsequent access will return the same updated value. • Weak Consistency – It is not guaranteed that subsequent accesses will return the updated value. • Eventual Consistency – Specific form of weak consistency – It is guaranteed that if no new updates are made to object, eventually all accesses will return the last updated value (e.g., propagate updates to replicas in a lazy fashion) 19
- 20. Eventual Consistency - A Dropbox Example • Dropbox enabled immediate consistency via synchronization in many cases. • However, what happens in case of a network partition? 20
- 21. Eventual Consistency - A Dropbox Example • Let’s do a simple experiment here: – Open a file in your drop box – Disable your network connection (e.g., WiFi, 4G) – Try to edit the file in the drop box: can you do that? – Re-enable your network connection: what happens to your dropbox folder? 21
- 22. Eventual Consistency - A Dropbox Example • Dropbox embraces eventual consistency: – Immediate consistency is impossible in case of a network partition – Users will feel bad if their word documents freeze each time they hit Ctrl+S , simply due to the large latency to update all devices across WAN – Dropbox is oriented to personal syncing, not on collaboration, so it is not a real limitation. 22
- 23. Dynamic Tradeoff between C and A • An airline reservation system: – When most of seats are available: it is ok to rely on somewhat out-of-date data, availability is more critical – When the plane is close to be filled: it needs more accurate data to ensure the plane is not overbooked, consistency is more critical • Neither strong consistency nor guaranteed availability, but it may significantly increase the tolerance of network disruption 23
- 24. Split brain Syndrome… primary backup Clients initially connected to primary, which keeps backup up to date. Backup collects the log log
- 25. Split brain Syndrome… Transient problem causes some links to break but not all. Backup thinks it is now primary, primary thinks backup is down primary backup
- 26. Split brain Syndrome Some clients still connected to primary, but one has switched to backup and one is completely disconnected from both primary backup
- 27. Implication? • Air Traffic System with a split brain could malfunction disastrously! – For example, suppose the service is used to answer the question “is anyone flying in such-and- such a sector of the sky” – With the split-brain version, each half might say “nope”… in response to different queries!
- 28. Can we fix this problem? • No, if we insist on an end-to-end solution – Essential insight is that we need some form of “agreement” on which machines are up and which have crashed – Can’t implement “agreement” on a purely 1-to-1 (hence, end-to-end) basis. • Separate decisions can always lead to inconsistency • So we need a “membership service”… and this is fundamentally not an end-to-end concept!
- 29. Can we fix this problem? • Yes, many options, once we accept this – Just use a single server and wait for it to restart • This common today, but too slow for ATC – Give backup a way to physically “kill” the primary, e.g. unplug it • If backup takes over… primary shuts down – Or require some form of “majority vote” • Ad mentioned, maintains agreement on system status • Bottom line? You need to anticipate the issue… and to implement a solution.
Editor's Notes
- #12: Atomicity requires that each transaction be "all or nothing" The consistency property ensures that any transaction will bring the database from one valid state to another. The isolation property ensures that the concurrent execution of transactions results in a system state that would be obtained if transactions were executed serially, The durability property ensures that once a transaction has been committed, it will remain so, even in the event of power loss, crashes, or errors.