Replication Distilled: Hazelcast Deep Dive @ In-Memory Computing Summit San Francisco 2018
1 like205 views
This document summarizes Hazelcast's replication protocol. It uses primary copy replication where operations are sent to primary replicas for strong consistency when reachable. It partitions data to scale primaries, updates replicas asynchronously, and uses anti-entropy to sync backups. During network partitions, consistency can be lost and divergence must be resolved upon reconnection using merge policies. The protocol is classified as PA/EC under the PACELC principle, favoring availability over consistency, though failure detectors and CP data structures allow it to behave more consistently in some cases.
Replication Distilled: Hazelcast Deep Dive @ In-Memory Computing Summit San Francisco 2018
- 1. Replication Distilled: Hazelcast Deep Dive Ensar Basri Kahveci Hazelcast
- 2. Hazelcast ▪ The leading open source Java IMDG ▪ Distributed Java collections, concurrency primitives, ... ▪ Distributed computations, messaging, ...
- 3. In-Memory Data Grids ▪ Distributed caching ▪ Keeping data in local JVM for fast access & processing ▪ Elasticity, availability, high throughput, and low latency ▪ Multiple copies of data to tolerate failures
- 4. Replication ▪ Putting a data set into multiple nodes ▪ Fault tolerance ▪ Latency ▪ Throughput
- 5. Challenges ▪ Where to perform reads & writes? ▪ How to keep replicas sync? ▪ How to handle concurrent reads & writes? ▪ How to handle failures?
- 6. CAP Principle ▪ Pick two of C, A, and P ▪ CP versus AP
- 7. CP
- 8. AP
- 11. PACELC Principle ▪ If there is a network partition (P), we have to choose between availability and consistency (AC). ▪ Else (E), during normal operation, we can choose between latency and consistency (LC).
- 12. Let’s build the core replication protocol of Hazelcast
- 13. Primary Copy ▪ Operations are sent to primary replicas. ▪ Strong consistency when the primary is reachable.
- 14. Partitioning (Sharding) ▪ Partitioning helps to scale primaries. ▪ A primary replica is elected for each partition.
- 16. Updating Replicas partition id = hash(serialize(key)) % partition count
- 19. Async Replication ▪ Each replica is updated separately. ▪ High throughput and availability
- 20. Anti-Entropy ▪ Backup replicas can fall behind the primary. ▪ Non-sync backups are fixed with an active anti-entropy mechanism.
- 21. Replicas are not sync ▪ The client reads a key from the current primary replica.
- 22. Network Partitioning ▪ The client reads the same key.
- 23. Split-Brain ▪ Strong consistency is lost.
- 24. Resolving the Divergence ▪ Merge policies: higher hits, latest update / access, … ▪ Merging may cause lost updates.
- 25. Let’s classify this protocol with PACELC
- 26. Hazelcast is PA/EC ▪ Consistency is usually traded to availability and latency together. ▪ Hazelcast works in memory and mostly used in a single computing cluster. ▪ Consistency - latency trade-off is minimal. ▪ PA/EC works fine for distributed caching.
- 28. Scaling Reads ▪ Reads can be served locally from near caches and backup replicas.
- 30. Failure Detectors ▪ Local failure detectors rely on timeouts. ▪ Operations are blocked after the cluster size falls below a threshold.
- 31. Failure Detectors ▪ It takes some time to detect an unresponsive node. ▪ Minimizes divergence and maintains the baseline consistency.
- 32. Isolated Failure Detectors ▪ Configure failure detectors independently for data structures ▪ Phi-Accrual Failure Detector
- 33. CP Data Structures ▪ IDGenerator ▪ Distributed impls of java.util.concurrent.* ▪ PA/EC is not the perfect fit for CP data structures.
- 34. Flake IDs ▪ Local unique id generation ▪ Nodes get a unique node id during join. ▪ K-ordered IDs
- 35. CRDTs ▪ CRDTs: Conflict-free Replicated Data Types ▪ Replicas are updated concurrently without coordination. ▪ Strong eventual consistency ▪ Counters, sets, maps, graphs, ...
- 36. PN-Counter
- 37. PN-Counter
- 38. Sync Replication ▪ Concurrency primitives imply the true CP behavior. ▪ Paxos, Raft, ZAB, VR ▪ Re-implementing Hazelcast concurrency primitives with Raft
- 39. Recap ▪ http://bit.ly/hazelcast-replication-consistency ▪ http://bit.ly/hazelcast-network-partitions ▪ http://dbmsmusings.blogspot.com/2017/10/hazelcast-an d-mythical-paec-system.html
- 40. Thanks! You can find me at ▪ @metanet ▪ ebkahveci@gmail.com