NoSQL in Practice with TIBCO: Real World Use Cases and Customer Success Stories for In-Memory Data Grids
7 likes3,792 views
The document discusses in-memory computing, highlighting its capabilities beyond mere caching, such as real-time event processing and fault tolerance. It provides a market overview, use cases, and customer success stories demonstrating the impact of in-memory data grids on operational efficiency and business value. The technology is presented as a solution to modern data processing challenges and emphasizes the advantages of combining in-memory storage with NoSQL features.
NoSQL in Practice with TIBCO: Real World Use Cases and Customer Success Stories for In-Memory Data Grids
- 1. NoSQL in Practice Real World Use Cases for In-Memory Data Grids Kai Wähner kwaehner@tibco.com www.kai-waehner.de @KaiWaehner LinkedIn / Xing à Please connect!
- 2. Key Messages In-Memory Computing is used for Acting in Real-Time! In-Memory is NOT just for Caching and Storing – A Data Grid offers much more! Eventing and Fault-Tolerance move In-Memory Computing to another Level!
- 3. © Copyright 2000-2014 TIBCO Software Inc. 3 Agenda • Introduction to In-Memory Computing • Market Overview • Use Cases / Customer Success Stories
- 4. © Copyright 2000-2014 TIBCO Software Inc. 4 Agenda • Introduction to In-Memory Computing • Market Overview • Use Cases / Customer Success Stories
- 5. Time Business Value Business Event Data Ready for Analysis Analysis Completed Decision Made $$$$ $$$ $$ $ Action Taken In-Memory Computing and Event Processing speed action and increase business value by seizing opportunities while they matter Business Value of Events over Time
- 6. © Copyright 2000-2014 TIBCO Software Inc. 6 • Hardware costs declining • Data Processing Requirements exploding • Traditional Approaches not scaling – Relational Databases – Clustered Databases – In-Memory Caches – Messaging Systems Drivers for In-Memory Computing
- 7. © Copyright 2000-2014 TIBCO Software Inc. 7 • Two parallel responses to the 21st century data processing needs • NoSQL Databases – Disk based with some in-memory caching – Horizontal Scalability on Commodity Hardware – Alternatives to Relational Databases and SQL – Basically Available Soft-state Eventually (BASE) – No ACID (transactions / concurrency control) • In-Memory Data Grid Technology – Memory for data storage – Pooling Memory from multiple machines – Use database for persistence – ACID Properties – Eventing – Notifications, Continuous Queries New Categories of Technology
- 8. © Copyright 2000-2014 TIBCO Software Inc. 8 Agenda • Introduction to In-Memory Computing • Market Overview • Use Cases / Customer Success Stories
- 9. © Copyright 2000-2014 TIBCO Software Inc. 9 Database Landscape in 2014 h*p://blogs.the451group.com/ informa@on_management/2014/03/18/ updated-‐data-‐plaForms-‐landscape-‐ map-‐february-‐2014/ In-‐Memory Data Grids
- 10. © Copyright 2000-2014 TIBCO Software Inc. 10 Database Landscape in 2014 h*p://blogs.the451group.com/ informa@on_management/2014/03/18/ updated-‐data-‐plaForms-‐landscape-‐ map-‐february-‐2014/ SAP HANA is not an In-‐Memory Data Grid!
- 11. Product Example: TIBCO ActiveSpaces Distributed In-memory System of Record Stores platform / language independent key-value data structures in memory with the option to persist data in parallel on local disks on a cluster of elastic horizontally scalable commodity hardware High Performance ACID compliant NoSQL Data Grid Offers all benefits of NoSQL databases and immediate consistency with full ACID compliance for transactions and concurrency control Minimal configuration and easy-to-use APIs (Java, C, .NET, “TIBCO Products”) Uses proprietary consistent hashing algorithm that that ensures a single network hop for fetching data. No need for partitioning, no complex XML configuration files Querying Data can be queried using an SQL-like language and queries can be accelerated through full indexing capabilities such as composite indexes and tree or hash index types. Best of both Worlds: NoSQL and In-Memory!
- 12. © Copyright 2000-2014 TIBCO Software Inc. 12 Agenda • Introduction to In-Memory Computing • Market Overview • Use Cases / Customer Success Stories
- 13. LOADER Caching for Fast Data Access • Cache to slower systems • Read-‐only • Not the system of record • No persistence required • Side benefit: Backend load is reduced
- 14. Caching + Dynamic Load • Dynamically loaded into Memory when the data is first accessed by a client applica@on • Service can present a standard interface • Client applica@ons are not required to implement any In-‐Memory specific code (1) Check Cache (2) Load from DB if not in Cache
- 15. Routing Messages to Back-Office Applications • Receive a common data feed that needs to be parsed and routed to several back-‐office applica@ons • In-‐Memory holding reference informa@on for the rou@ng applica@on. The router can quickly determine where to send the data. • Examples: Bank payments, insurance claims processing
- 16. Off-loading expensive systems Expensive in terms of response @me and / or transac@on costs!
- 17. Success Story (CRM): Personalized Customer Experience “With 38 million fans, MGM knows how to put its customers first, it takes more than a smile too. Customers want a personalized, tailored experience, one that knows their name and can an@cipate their needs. With the help of TIBCO technologies that leverage big data and give customers a digital iden@ty, MGM can send personalized offers directly to customers, save them a seat, and have their favorite drink on the way. With mul@ple customer touch points and channels, MGM can reach customers in more ways, and in more places, than ever before.” h*ps://www.youtube.com/watch?v=X-‐7S3kCOx9k Latency Problems: • Several Legacy Systems • Processing via ERP, CRM, Host, etc. In-‐Memory: • Events and Correla@ons • Enable Real Time • Only customers that have checked in
- 18. Fault Tolerance and Disaster Recovery Enabling Active-Active Fault Tolerance in Applications: In-‐Memory Compu@ng is reliable, scalable and fault-‐tolerant!
- 19. Fault Tolerance and Disaster Recovery Multisite Data Replication: In-‐Memory Compu@ng is reliable, scalable and fault-‐tolerant!
- 20. Handling temporary spikes on a slow ‘system of record’ • An In-‐Memory event listener gets no@fied whenever a data value is changed and sends updates through a message queue for upda@ng the master system of record. • The back office system can also be updated through other channels. • Examples: Christmas Shopping in E-‐Commerce, Ticket Sales, Online Bekng
- 21. à In-‐Memory as “system of record” Operational Data Store (Local File System)
- 22. • Low-‐latency, high-‐throughput opera@onal data – Customer data: e.g. account status and balance, purchase history: real-‐@me loyalty (promo@ons, cross-‐selling), fraud detec@on, ... – Market data: e.g. risk assessment, porFolio mgmt, produc@on output op@miza@on, buyer-‐seller matching – Sensor data: e.g. smart metering / grid, public transport safety – Track and trace: e.g. barcode scans, RFID: logis@cs, airlines • Why In-‐Memory? – Much faster than tradi@onal DB, especially many small transac@ons (XTP) – State / data management not addressed by messaging solu@ons – Even@ng is a first class feature, changes can be ‘pushed’ in real-‐@me to interested par@es (subscribe to changes, con@nuous queries) – Provides for distributed process synchroniza@on – Integrated with CEP engines (e.g. TIBCO BusinessEvents, TIBCO StreamBase) Operational Data Store (Local File System)
- 23. Situation • Master data management system stores over 800 million customer records across more than 30 enterprise apps. • Stores real-time inventory data to enable ‘Buy online and pick-up at store’ and ‘Smart fulfillment’ features Problem • Due to lack of correlation between Point of Sale data and inventory, the website contained outdated inventory data. Products were listed as out of stock when there was actually inventory. • Need to leverage store inventory as well as inventory located fulfillment centers Solution • In-Memory stores real-time inventory data for the website, the fulfillment application, and other applications that need access to inventory data Business Impact • Reduction in customer churn • Intelligent fulfillments leading to greater customer satisfaction • Improved overall efficiency of fulfillment centers and store inventory Success Story (Retailer): Inventory Management
- 24. Distribution of Rapidly Changing Data à Examples are monitoring data for a power plant, stock market data, telemetry data for a complex system (example, a satellite), or the status and loca@on of packages for a major logis@cs or shipping company.
- 25. Success Story (Telco): Real-Time Offer Generation and Fulfillment by Different Subcontractors Reload Give 100 free SMS to subscriber who tops-up Total: 12 mio top-up / day Peak: 300 top-up per sec Purchase 3G Package Cross-sell Voice/SMS package to subscriber who purchases 3G Mobile Package Total: 3 mio / day Peak: 50 events per sec Voice Call Give discount VOIP package to subscriber who makes a IDD call Total: 200 mio / day Peak: 12,000 events per sec SMS Usage Give discounted SMS package to subscriber who sends SMS more than 10 times a day Total: 750 mio / day Peak: 27,000 events per sec Event Cloud Purchase BB Package Reload Voice Call IDD Call OnNet Call SMS Usage Event Handling and Processing Touchpoint Integration Billing, Offer Fulfilled Fulfill SMS Package Fulfill 3G Package Fulfill Voice Package Fulfill SMS Package 46.7 million subscribers 2,000 SMS notifications per seconds 500 offer fulfillments per second Offer Message Reminder Message Fulfillment Message
- 26. Success Story (Telco): Real-Time Offer Generation and Fulfillment by Different Subcontractors The numbers • 1 billion events per day • Peaks of 40,000 to 50,000 events per second (for hours, during peak usage period of the day) from Network • 2 TIBCO BusinessWorks servers, 2 TIBCO Ac@veSpaces servers (ac@ve-‐ac@ve)
- 27. • Technical issues in distributed grid compu@ng with large scale data – Work load distribu@on – Process synchroniza@on – Data transfer • Examples – Risk assessment and management – Op@miza@on problems: scheduling, cargo assignment, load distribu@on in power network / grid • Why In-‐Memory? – Many useful synchroniza@on features (e.g. atomic “take”) – Loca@on transparency and fault-‐tolerance – Real-‐@me instead of nightly / weekly / ... Data-‐Warehousing approach Super Fast Compute Grid for Intermediary Calculations for Analytics
- 28. Super Fast Compute Grid for Intermediary Calculations for Analytics
- 29. State-‐full Data Storing State-full Data for Enterprise Tools à In-‐Memory Data Grid as part of Enterprise Tools (ESB, CEP, BPM, etc.)
- 30. Eventing and Fault-Tolerance move In-Memory Computing to another Level! In-Memory is NOT just for Caching and Storing – A Data Grid offers much more! In-Memory Computing is used for Acting in Real-Time! Key Messages
- 31. Questions? Kai Wähner kwaehner@tibco.com @KaiWaehner www.kai-waehner.de LinkedIn / Xing à Please connect!