U1-T9,T5-Distributed Vs Parallel Processing and Typical Hadoop Environment.pptx
- 7. Distributed: Why? • Simple, cheaper components • Easy to add capability incrementally • Let multiple users cooperate (maybe) • Physical components owned by different users • Enable collaboration between diverse users 9
- 8. The Promise of Dist. Systems • Availability: One machine goes down, overall system stays up • Durability: One machine loses data, but system does not lose anything • Security: Easier to secure each component of the system individually? 10
- 9. Distributed: Worst-Case Reality • Availability: Failure in one machine brings down entire system • Durability: Any machine can lose your data • Security: More components means more points of attack 11
- 10. Distributed Systems Goal • Transparency: Hide "distributed-ness" from any external observer, make system simpler • Types • Location: Location of resources is invisible • Migration: Resources can move without user knowing • Replication: Invisible extra copies of resources (for reliability, performance) • Parallelism: Job split into multiple pieces, but looks like a single task • Fault Tolerance: Components fail without users knowing 12
- 11. Challenge of Coordination • Components communicate over the network • Send messages between machines • Need to use messages to agree on system state • This issue does not exist in a centralized system 13
- 12. CAP Theorem • Originally proposed by Eric Brewer (Berkeley) 1. Consistency – changes appear to everyone in same sequential order 2. Availability – can get a result at any time 3. Partition Tolerance – system continues to work even when one part of network can't communicate with the other • Impossible to achieve all 3 at the same time (pick two) 14
- 13. CAP Theorem Example • What do we do if a network partition occurs? • Prefer Availability: Allow the state at some nodes to disagree with the state at other nodes (AP) • Prefer Consistency: Reject requests until the partition is resolved (CP) 15 Partition A Partition B
- 14. Consistency Preferred • Block writes until all nodes able to agree • Consistent: Reads never return wrong values • Not Available: Writes block until partition is resolved and unanimous approval is possible 16
- 15. What about AP Systems? • Partition occurs, but both groups of nodes continue to accept requests • Consequence: State might diverge between the two groups (e.g., different updates are executed) • When communication is restored, there needs to be an explicit recovery process • Resolve conflicting updates so everyone agrees on system state once again 17