Data has a better idea the in-memory data grid
- 1. The In-Memory Data Grid Photo by Franki Chamaki on Unsplash
- 2. About me ... - Passionate software engineer - Focused mainly on JVM - Interested in all software development phases - Having erethic opinions - Non politically correct, but just correct … - Personal belief: “living in a distributed and reactive full of actors system” Posting on:
- 3. Agenda 1. What is In-Memory Data Grid (IMDG) ? 2. Hazelcast IMDG 3. Cluster Discovery 4. Partitioning and Replication 5. Data Structure Overview 6. User-Code Deployment & Hazelcast-Spring 7. Demo time!
- 4. What is an In-Memory Data Grid (IMDG) ? A Data Grid is a system of multiple servers that work together to manage information and related operations in a distributed environment.
- 5. What is an In-Memory Data Grid (IMDG) ? A Data Grid is a system of multiple servers that work together to manage information and related operations in a distributed environment. The servers from the grid can be located in the same location or distributed across multiple data centers.
- 6. What is an In-Memory Data Grid (IMDG) ? A Data Grid is a system of multiple servers that work together to manage information and related operations in a distributed environment. The servers from the grid can be located in the same location or distributed across multiple data centers. An In-Memory Data Grid is a grid that stores data entirely into RAM.
- 7. What is an In-Memory Data Grid (IMDG) ?
- 8. What is an In-Memory Data Grid (IMDG) ?
- 9. Why to use an In-Memory Data Grid? Performance ● Access data 1000x faster than a database ● Low latency for batch and stream processing
- 10. Why to use an In-Memory Data Grid? Performance Data structure/Handling ● Access data 1000x faster than a database ● Low latency for batch and stream processing ● Non-relational key-value ● ACID compliance
- 11. Why to use an In-Memory Data Grid? Performance Data structure/Handling Operations ● Access data 1000x faster than a database ● Low latency for batch and stream processing ● Non-relational key-value ● ACID compliance ● Scalability ● Redundancy for HA
- 12. When to use an In-Memory Data Grid? Data Cache ● Eliminates data store bottlenecks ● Eliminates slow network connections ● Long-running blocking calculations
- 13. When to use an In-Memory Data Grid? Data Cache Data Service Fabric ● Eliminates data store bottlenecks ● Eliminates slow network connections ● Long-running blocking calculations ● Real-time integration ● Compute grid ● Message broker
- 14. When to use an In-Memory Data Grid? Data Cache Data Service Fabric Examples ● Eliminates data store bottlenecks ● Eliminates slow network connections ● Long-running blocking calculations ● Real-time integration ● Compute grid ● Message broker ● Analytics (Risk, Fraud-detection) ● Trading Systems (FX Trading, Stock Exchange) ● eCommerce ● Online Gaming
- 15. Basic operations of an In-Memory Data Grid Cluster ● Distributed data ● Highly scalable ● Fault tolerance
- 16. Basic operations of an In-Memory Data Grid Cluster Discovery ● Distributed data ● Highly scalable ● Fault tolerance ● Form ● Join ● Find
- 17. Basic operations of an In-Memory Data Grid Cluster Discovery Data Distribution ● Distributed data ● Highly scalable ● Fault tolerance ● Form ● Join ● Find ● Replication/Mirroring ● Partitioning/Sharding
- 18. Replication and Partitioning Replication - all the data is replicated (synchronously or asynchronously) to every node in the cluster.
- 19. Replication and Partitioning Replication - all the data is replicated (synchronously or asynchronously) to every node in the cluster. examples: a cluster of a relational database (MySQL, Oracle, etc.) leveraging the master/leader-slave/follower model
- 20. Replication and Partitioning Replication - all the data is replicated (synchronously or asynchronously) to every node in the cluster. examples: a cluster of a relational database (MySQL, Oracle, etc.) leveraging the master/leader-slave/follower model - Synchronous replication: writes need to be confirmed by a configurable number of slaves/followers before the master/leader reports success (ACKs). - Asynchronous replication: the master/leader reports success immediately after a write was committed to its own disk/memory; followers apply the changes in their own pace.
- 21. Replication and Partitioning Partitioning/Sharding - is the most scalable distributed mode and relies on keeping data in multiple disjoint distributed collections on different nodes, leveraging 3 partitioning strategies: range, hash and custom partitioning.
- 22. Replication and Partitioning Partitioning/Sharding - is the most scalable distributed mode and relies on keeping data in multiple disjoint distributed collections on different nodes, leveraging 3 partitioning strategies: range, hash and custom partitioning. examples: most NoSQL databases (Cassandra, MongoDB, etc.), Kafka uses partitions
- 23. Replication and Partitioning Partitioning/Sharding - is the most scalable distributed mode and relies on keeping data in multiple disjoint distributed collections on different nodes, leveraging 3 partitioning strategies: range, hash and custom partitioning. examples: most NoSQL databases (Cassandra, MongoDB, etc.), Kafka uses partitions - Range partitioning: takes into account the natural order of keys to split the dataset in the required number of partitions; ex: MySQL partitions. - Hash partitioning: calculates a hash over the each item key and then produces the modulo of this hash to determine the new partition; ex: Cassandra consistent hashing algorithm. - Custom partitioning: exploits locally or uniqueness properties of the data to calculate the appropriate partition to store the data to; ex: pre-hashed data like git commits or location specific data like all records from Europe
- 24. Replication vs Partitioning Cons Cons Pros Pros Pros Pros Workload scalability High availability Failure recovery
- 25. Replication vs Partitioning Replication Cons Cons Pros Pros Pros Pros Workload scalability High availability Failure recovery Cons Replication negative performance impact Data inconsistency Memory scalability issues
- 26. Replication vs Partitioning Replication Cons Cons Pros Pros Pros Pros Workload scalability High availability Failure recovery Cons Replication negative performance impact Data inconsistency Memory scalability issues Pros Workload scalability Failure recovery Memory scalability Good synchronization performance
- 27. Replication vs Partitioning Replication Cons Cons Pros Pros Pros Pros Workload scalability High availability Failure recovery Cons Replication negative performance impact Data inconsistency Memory scalability issues Pros Workload scalability Failure recovery Memory scalability Good synchronization performance Cons Migration negative performance impact Large memory requirements
- 28. Deployment options Embedded IMDG Client-Server
- 30. Why to choose Hazelcast IMDG? Market Leader ● Hazelcast IMDG is market leader among In-Memory Data Grid solutions
- 31. Why to choose Hazelcast IMDG? Market Leader Rich API ● Hazelcast IMDG is market leader among In-Memory Data Grid solutions ● APIs in various programming languages: Java, C#.NET, Python, etc. ● Powerful features ● Huge user base - open source project
- 32. Why to choose Hazelcast IMDG? Market Leader Rich API Ease of use ● Hazelcast IMDG is market leader among In-Memory Data Grid solutions ● APIs in various programming languages: Java, C#.NET, Python, etc. ● Powerful features ● Huge user base - open source project ● Simple to use key-value data store ● Standard data structures: Map, List, Queue, etc. ● Clients for many programming languages ● Redundancy/fail-over/sc aling built-in
- 33. Why to choose Hazelcast IMDG? Market Leader Rich API Ease of use Distributed data store & computation system ● Hazelcast IMDG is market leader among In-Memory Data Grid solutions ● APIs in various programming languages: Java, C#.NET, Python, etc. ● Powerful features ● Huge user base - open source project ● Simple to use key-value data store ● Standard data structures: Map, List, Queue, etc. ● Clients for many programming languages ● Redundancy/fail-over/ scaling built-in ● Distributed data store ● Distributed computation near stored data
- 34. Business scenario and HLA - overview Business scenario: ● We will use Hazelcast IMDG for developing a Foreign Exchange Quotation Management System. The system is consisting of: ● Two Spring Boot microservices: market-client and trader-cli which are basically Hazelcast clients and are communicating with the grid via APIs. ● One Spring Boot microservice called processing-unit which is basically a Hazelcast server member that will join the cluster when started.
- 35. Hazelcast features used Deployment model: ● Client-Server Cluster discovery mechanism: ● TCP/IP unicast discovery Data structures used: ● Replicated Map ● Partitioned Map
- 36. Client-Server Deployment Model Hazelcast Client Hazelcast Cluster Member For creating a Hazelcast Client Java application we must add the following dependencies: ● Prior to Hazelcast 4.x: ○ com.hazelcast:hazelcast:3.x ○ com.hazelcast:hazelcast-client:3.x ● For projects which are using Hazelcast 4.x: ○ com.hazelcast:hazelcast:4.x For creating a Hazelcast Cluster Member Java application we must add the following dependency: ○ com.hazelcast:hazelcast:${version}
- 37. Hazelcast Cluster Discovery There are multiple ways to establish a discovery mechanism inside our Hazelcast cluster: ● TCP/IP multicast ● TCP/IP unicast ● Discovery plugins or cloud: Eureka, ZooKeeper, K8s, OpenShift, Pivotal Cloud Foundry (PCF), Google Cloud Platform (GCP), AWS, Azure ● Custom discovery mechanism via Discovery SPI
- 38. Hazelcast Cluster Discovery In our cluster members we use TCP/IP unicast discovery. In com.freesoft.fx.trading.processingunit.infrastructure.imdg.HazelcastConfiguration.java:
- 39. Hazelcast Replicated Map In the processing-unit Hazelcast Cluster Member Java application we are using a replicated map data structure for storing the quote prices published by the market-client. The map must have a name, which in our case is “QUOTES_MAP” and is stored in binary format in each cluster member instance.
- 40. Hazelcast Replicated Map The necessary configuration for using a Replicated Map can be found in com.freesoft.fx.trading.processingunit.infrastructure.imdg.HazelcastConfiguration.java file.
- 41. Hazelcast Partitioned Map In the processing-unit Hazelcast Cluster Member Java application we are using a partitioned map data structure for storing all commands (Buy and Sell) published by each trader (trader-cli microservice). Hazelcast uses a hash partitioning for distributing the data across all cluster members.
- 42. Hazelcast Partitioned Map The necessary configuration for using a Partitioned Map can be found in com.freesoft.fx.trading.processingunit.infrastructure.imdg.HazelcastConfiguration.java file.
- 43. User code deployment & Hazelcast-Spring
- 44. User code deployment ● Not enabled by default ● Allows us to load client classes inside cluster members ● There are necessary configurations that must be done in both the client and the cluster member
- 45. User code deployment Client configuration
- 46. User code deployment Client configuration Cluster member configuration
- 48. Hazelcast-Spring ● com.hazelcast:hazelcast-spring:${version} ● Dependency Inversion Principle
- 49. Hazelcast-Spring ● com.hazelcast:hazelcast-spring:${version} ● Dependency Inversion Principle ● @SpringAware
- 50. Hazelcast-Spring ● com.hazelcast:hazelcast-spring:${version} ● Dependency Inversion Principle ● @SpringAware
- 51. User-code deployment & Hazelcast-Spring
- 52. User-code deployment & Hazelcast-Spring
- 53. User-code deployment & Hazelcast-Spring
- 54. User-code deployment & Hazelcast-Spring
- 55. User-code deployment & Hazelcast-Spring
- 56. User-code deployment & Hazelcast-Spring
- 57. User-code deployment & Hazelcast-Spring
- 58. User-code deployment & Hazelcast-Spring
- 59. Demo Time!
- 60. Source code:
- 61. Hazelcast post on Medium!
- 62. Thank you ! ● Don’t forget to follow me on @dinabogdan03 ● Don’t forget to read my articles on @bogdan.dina03 ● Join Bucharest Apache Kafka meetup group !!!