GemFire In Memory Data Grid
Download as PPTX, PDF6 likes7,325 views
This document provides an overview of GemFire, an in-memory data grid that pools memory across processes to manage application data and behavior. Some key points: - GemFire allows distributed applications to achieve low-latency data access through an in-memory shared cache. It supports features like caching, querying, transactions, and event notifications. - Data in GemFire is organized into regions, which allow data to be stored across multiple servers without regard to location. Region types include replicated, partitioned, and local. - The CAP theorem states that only two of three properties - consistency, availability, and partition tolerance - can be achieved in a distributed system. GemFire aims to balance availability and partition tolerance.
GemFire In Memory Data Grid
- 1. GemFire: In-Memory Data GridSeptember 8th, 2011
- 3. Is it easy to scale Data Base?New users means, more application servers and more load to database.Application TierClientsData Base3
- 4. Moore's law: The number of transistors doubles approximately every 24 monthsWhat about data? 90% of today’s data were created in the last 2 yearsWeb logs, financial transactions, medical records, etc4
- 5. “Hardware can give you a generic 20 percent improvement in performance, but there is only so far you can go with hardware.”Rob Wallos,Global Head of marketing data Citi5
- 6. What is latency?Latency – is the amount of time that it takes to get information from one designated point to another.6
- 7. Why worry about it?Amazon - every 100ms of latency cost them 1% in salesGoogle - an extra 0.5seconds in search page generation time dropped traffic by 20%Financial - If a broker's electronic trading platform is 5ms behind the competition it could loose them at least 1% of the flow - that's 4$ million in revenues per ms.7
- 8. How to make data access even fast?Distributed Architecture
- 9. Drop ACID
- 10. Atomicity
- 11. Consistency
- 12. Isolation
- 13. Durability
- 15. Drop Disk8
- 16. Data GridData Grid is the combination of computers what works together to manage information and reach a common goal in a distributed environment.9
- 17. Shared nothing architectureIs a distributed computing architecture in which each node is independent and self-sufficient, and there is no single point of contention across the system.Popularized by BigTable and NoSQL
- 19. Massive scalability of processing10
- 20. In-Memory Data GridData are stored in memory, always available and consistent.Low Latency
- 22. No Single Point of failure
- 23. Associate arrays
- 24. Replicated
- 25. Partitioned11
- 26. GemFireThe GemFire is in-memory distributed data management platform that pools memory across multiple processes to manage application objects and behavior.Caching
- 27. Querying
- 28. Transactions
- 31. CAP TheoremOnly two of these three desirable properties in distributed system can be achieved:Consistent
- 32. Available
- 34. RegionsData region is a logical grouping within a cache for a single data set.A region lets you store data in many VMs in the system without regard to which peer the data is stored on. Work similar to Map interface.14
- 35. Region ExampleCache cache = new CacheFactory().set("cache-xml-file", "cache.xml”).create();CacheServercacheServer = cache.addCacheServer();cacheServer.start();Regionpeople = cache.getRegion(”people");people.put(“John”, john);<cache> <regionname="people"> </region> </cache>Create Cache Server
- 37. Place an John entry into the region15
- 38. Replicated RegionEach replicated region holds the complete data set for the regionHigh Read Performance
- 39. Limited by JVM heap size
- 40. Used for meta data16
- 41. Partitioned RegionGemFire partitions your data so that each peer only stores a part of the region contents.Data spread across nodes
- 42. Members have access to all data
- 43. Used for Large data set
- 45. What happens if one node fails?Recovering redundancy can be configured to take place immediately after one node fail.This gives High Availability for partition regions.18
- 46. Local RegionThe local region has no peer-to-peer distribution activity.Client regions automatically defined as local regions:Direct to distributed system
- 48. Peer DiscoveryTo connect to distributed system the peer should introduce themself:Multicast based discovery
- 49. Locator separate component that maintains a discovery20
- 50. P2P topologyThe cache is embedded within the application process and shares the heap space with the application.21
- 51. Client/Server topologyA central cache is managed in one distributed system tier by a number of server members. Clients maintain their own caches that automatically call upon the server side.22
- 52. Multi-Site CachingDistributed systems at different sites are loosely coupled through gateway system members.23
- 53. Read ThroughWhen an entry is requested that is unavailable in the region, a Cache Loader may be called upon to load it from data source.Operation always managed by the partition node. 24
- 54. Write ThroughTo provide write-through caching with your external data source use CacheWriter.Only one writer is invoked for any event.25
- 55. Write BehindIn the Write-Behind mode, updated cache entries are asynchronously written to the back-end data source. 26
- 56. Event ListenerThe cache event listeners allow you to receive after-event notification of changes to the region and its entries.Handle following entity events:Create
- 57. Update
- 58. Destroy
- 59. InvalidateExecuted in all replicated regionsExecuted only in one partition region27
- 60. Listener Example<regionname=“people” refid=“PARTITION”> <region-attributes> <cache-listener> <class-name>com.mirantis.PeopleCacheListener</class-name> </cache-listener> <cache-loader><class-name>com.mirantis.PeopleCacheLoader</class-name></cache-loader> </region-attributes></region>public class PeopleCacheListener<K,V> extends CacheListenerAdapter<K,V> implements Declarable { public void afterCreate(EntryEvent<K,V> e) {System.out.println(e.getKey() + “ connected”);} public void afterDestroy(EntryEvent<K,V> e) {System.out.println(e.getKey() + “ left”); } …}28
- 61. QueryingObject Query Language (OQL) is SQL like query language standard for object-oriented databases.Support normal query and continuous querying (CQ).SELECT DISTINCT * FROM /portfolios WHERE status = 'active' AND type = ‘XYZ’Queryquery = qryService.newQuery(queryString);SelectResults results = (SelectResults)query.execute();for (Iteratoriter = results.iterator(); iter.hasNext(); ) { Portfolio activeXYZPortfolio = (Portfolio) iter.next(); ...}You can also use indexing to optimize your query performance.29
- 62. Continuous QueryingContinuous Querying (CQ) gives your clients a way to run queries against events.public class TradeEventListener implements CqListener {publicvoidonEvent(CqEventcqEvent) { … }publicvoidonError(CqEventcqEvent) { // handle the error } public void close() { // close the output screen for the trades ... }}CqAttributesFactorycqf = new CqAttributesFactory();cqf.addCqListener(tradeEventListener);CqAttributescqa = cqf.create();CqQuerypriceTracker = queryService.newCq(“tracker“, queryStr, cqa);priceTracker.execute();30
- 63. Function ExecutionApplication functions can be executed on:Members
- 64. Data setSimilar to Map-Reduce31
- 65. You can move the state or behaviorData BaseClientsApplication TierIMDG32
- 66. Example Broker ApplicationHigh Available
- 68. Exchange Server could have only one connection
- 69. Orders are swapped to Data Base
- 70. Scale on Demand 33
- 71. Learn moreVMWareGemFirehttp://www.vmware.com/products/vfabric-gemfire/overview.htmlMonitoring ToolsGemFireCommunity http://community.gemstone.com/display/gemfireHibernate L2 Cache