java.util.concurrent for Distributed Coordination - Berlin Expert Days 2019
0 likes91 views
The document outlines a presentation at Berlin Expert Days 2019 about distributed coordination using Java's concurrency utilities and Hazelcast IMDG. It discusses the evolution of coordination APIs, challenges in concurrency, consensus algorithms, and includes demos on leader election and distributed locking. The presentation emphasizes using high-level APIs to reduce guesswork in development and ensure operational simplicity.
java.util.concurrent for Distributed Coordination - Berlin Expert Days 2019
- 1. java.util.concurrent for distributed coordination Berlin Expert Days 2019
- 2. @metanet Hazelcast The leading open source Java IMDG Distributed Java collections, concurrency primitives, messaging Caching, application scaling, distributed coordination 🎉 Hazelcast Cloud https://hazelcast.cloud Hazelcast Jet: In-memory stream and fast batch processing #bedcon
- 3. @metanet Agenda What is distributed coordination? How distributed coordination APIs evolved over time? Why to use java.util.concurrent.* Demo on Hazelcast IMDG 3.12 #bedcon
- 4. # #bedcon
- 5. @metanet Distributed Coordination Leader election Synchronization Group membership Configuration and metadata management #bedcon
- 6. @metanet Challenges Concurrency Strong consistency Partial failures Crashes Network failures Timing failures #bedcon
- 8. @metanet Consensus algorithms under the hood CP with respect to CAP Deployed as a central repository APIs for coordination tasks Distributed Coordination Systems #bedcon
- 10. Google Chubby (Paxos) Apache ZooKeeper (ZAB) @metanet #bedcon
- 11. Google Chubby (Paxos) Apache ZooKeeper (ZAB) etcd (Raft) @metanet #bedcon
- 12. @metanet /services /payment /product /photo /services /services/payment /services/product /services/product/photo Chubby & ZooKeeper etcd #bedcon
- 13. @metanet Chubby Locking APIs Recipes ZooKeeper etcd #bedcon
- 14. @metanet A Simple Locking Recipe for ZooKeeper 1. create an ephemeral znode “/lock” 2. if success, enter to the critical section 3. else, register a watch on “/lock” 4. when the watch is notified, i.e., the lock is released, retry step #1 #bedcon
- 15. @metanet Chubby Locking APIs Recipes “Friends don't let friends write ZK recipes.” Apache Curator Tech Notes #6 Leader election and distributed lock primitives ZooKeeper etcd #bedcon
- 16. @metanet High-level APIs A low-level file-system / KV store API is - easy to misuse, - not suitable for all coordination tasks. “Consensus in the Cloud: Paxos Systems Demystified” High-level APIs minimise guesswork and development effort. java.util.concurrent.* in JDK #bedcon
- 17. Google Chubby (Paxos) Apache ZooKeeper (ZAB) etcd (Raft) Hazelcast IMDG 3.12 java.util.concurrent on top of Raft @metanet #bedcon
- 18. @metanet An Opinionated & High-Level Framework IAtomicLong, IAtomicReference, ICountDownLatch, ISemaphore, FencedLock Well-defined failure semantics CP with respect to CAP DIY-style tested with Jepsen #bedcon
- 19. @metanet Strong consistency Used for building RSM. Handles crashes and network failures. Operational as long as the majority is up. Consensus #bedcon Node Count Majority Degree of Fault Tolerance 2 2 0 3 2 1 4 3 1 5 3 2 6 4 2 7 4 3
- 20. @metanet Understandability as a primary goal. Runtime concerns (snapshotting, dynamic membership) Performance optimizations (fast reads, batching) https://raft.github.io Why Raft? #bedcon
- 21. @metanet A leader is elected among the nodes. The leader replicates ops to the followers. All nodes run the ops in the same order. Result of a commit is never lost. Replicated State Machines #bedcon
- 23. @metanet https://github.com/metanet/juc-talk DEMO #1: Configuration management #bedcon
- 24. @metanet FencedLock Linearizable distributed impl of java.util.concurrent.locks.Lock Suitable for both fine-grained and coarse-grained locking #bedcon
- 25. @metanet CP Sessions A session starts on the first lock / semaphore request. Session heartbeats are periodically committed in the background. If no heartbeat for some time (session TTL), the session is closed. Auto-release mechanism for FencedLock and ISemaphore #bedcon
- 26. @metanet DEMO #2: Adding Redundancy We use FencedLock for leader election. #bedcon
- 27. @metanet CP sessions offer a trade-off between safety and liveness. #bedcon
- 28. @metanet DEMO #3: Fencing-off Stale Lock Holders “How to do distributed locking” “Distributed locks are dead; long live distributed locks!” #bedcon
- 29. @metanet Recap Avoid writing your own implementations for coordination. High-level APIs minimise guesswork and development effort. java.util.concurrent.* FTW! Operational simplicity matters. Dynamic clustering Horizontal scalability #bedcon
- 30. @metanet Current Status Disk persistence (WIP - 95% done) Tooling & Metrics (WIP - 95% done) KV Store (in roadmap) Event Listeners (in roadmap) #bedcon
- 31. @metanet Resources https://github.com/metanet/juc-talk (demos) Hazelcast IMDG Docs CP Subsystem Code Samples https://hazelcast.com/blog/author/ensarbasri Hazelcast IMDG 3.12 #bedcon