Neo4j GraphTalk Amsterdam - Next Generation Solutions using Neo4j
Download as PPTX, PDF1 like308 views
The document discusses next generation solutions built on Neo4j. It begins with an overview of solutions using Neo4j and recommendations. It then covers topics including AI/ML, GDPR compliance, and conclusions. Several case studies are presented including how Walmart and eBay use Neo4j for real-time recommendations and routing solutions. The benefits of using a graph database like Neo4j for recommendation engines and GDPR compliance are discussed.
![OSLO, MAY 9, 2018
#3 Consent Management + MDM
Amsterdam
NL
K.Vegter
+31623900
4…
kees@neo
4j.com
kees@gm
ail.com
{ contrib: ‘XYZ’,
permittedFor: [UC1,UC4],
consentUntil : 31-12-19 }
{ contrib: ‘internal’,
permittedFor: [UC3],
consentUntil : 31-12-20 }
{ contrib: ‘internal’,
permittedFor: [UC3],
consentUntil : 31-12-20 }
{ contrib: ‘internal’,
permittedFor: [UC3],
consentUntil : 31-12-20 }
{ contrib: ‘LMN’,
permittedFor: [UC2,UC6],
consentUntil : 31-12-20 }](https://image.slidesharecdn.com/nextgenerationsolutionsusingneo4jams2018-181022072149/85/Neo4j-GraphTalk-Amsterdam-Next-Generation-Solutions-using-Neo4j-58-320.jpg)
![AMSTERDAM, OCT 18, 2018
Conclusion
(graphs)-[:ARE]-> (everywhere)
and
(Solutions)-[:NEED]-> (graphs)](https://image.slidesharecdn.com/nextgenerationsolutionsusingneo4jams2018-181022072149/85/Neo4j-GraphTalk-Amsterdam-Next-Generation-Solutions-using-Neo4j-72-320.jpg)
Neo4j GraphTalk Amsterdam - Next Generation Solutions using Neo4j
- 1. Next Generation Solutions built on Neo4j Kees Vegter, Pre-Sales Engineer Graphtalk Amsterdam, Oct 18 2018
- 2. AMSTERDAM, OCT 18, 2018 Agenda ● Solutions using Neo4j ● Recommendations ● AI ML ● GDPR ● Conclusions
- 3. AMSTERDAM, OCT 18, 2018 Solutions: new mindset required?
- 4. AMSTERDAM, OCT 18, 2018 Solutions: new mindset Yesterday: - Static Applications - Designed to fulfill current requirements - Performance Constraints - Domain experts versus IT experts Tomorrow: - Flexible Applications - Designed to fulfill tomorrows requirements - Performance is not limiting - Domain experts work hand in hand with IT experts
- 5. AMSTERDAM, OCT 18, 2018 New Mindset Store Data in a Different Way
- 6. AMSTERDAM, OCT 18, 2018 Look at this data…
- 7. AMSTERDAM, OCT 18, 2018 Swap glasses…
- 8. AMSTERDAM, OCT 18, 2018 … now look at it again, this time as a graph
- 9. AMSTERDAM, OCT 18, 2018
- 10. AMSTERDAM, OCT 18, 2018 Node Relationship
- 11. Speed: Real time query enabled Graph Based Solutions Enables Up-Sell / Cross-sell Key Features Added Value 360 degree view on data Using data Connections as a value Intuitive: Supports Business Needs Flexible: enabled for additional requirements Finding patterns within the data Detect anomalies Prevent rather than detect Enables conversation across Functions Comply to regulations What-if Analysis Telco OSS GDPR Fraud Telco BSS Recomm endations MDM Resource efficient
- 12. AMSTERDAM, OCT 18, 2018 Evolution using Neo4j Neo4j Platform Graph Transactions Graph Analytics Data Integration Development & Admin Analytics Tooling Drivers & APIs Discovery & Visualization Developers Admins Applications Business Users Data Analysts Data Scientists 3rd Party Tools “The Graph Advantage” Domain know-how Professional Services PS Packages Graph Based Solution
- 13. AMSTERDAM, OCT 18, 2018 Evolution using Neo4j Neo4j Platform Graph Transactions Graph Analytics Data Integration Development & Admin Analytics Tooling Drivers & APIs Discovery & Visualization Developers Admins Applications Business Users Data Analysts Data Scientists 3rd Party Tools “The Graph Advantage” Domain know-how Professional Services PS Packages Graph Based Solution Neo4j enables Graph Based Solutions with a need for: - Agility - Intuitiveness - High Performance to support connected data scenarios - Scalable on traversing through connected data
- 14. OSLO, MAY 9, 2018 Recommendation EnginesBuilding Powerful Recommendation Engines With Neo4j
- 15. AMSTERDAM, OCT 18, 2018 “If you liked this, you might like that…” Powerful, real-time, recommendations and personalization engines have become fundamental for creating superior user experience and commercial success in retail
- 16. AMSTERDAM, OCT 18, 2018 Creating Relevance in an Ocean of Possibilities
- 17. AMSTERDAM, OCT 18, 2018 How Graph Based Recommendations Transformed the Consumer Web People Graph “People you may know” Disruptor: Facebook Industry: Media Ad-business Disruptor: Amazon Industry: Retail People & Products “Other people also bought” People & Content “You might also like” Disruptor: Netflix Industry: Broadcasting Media
- 18. AMSTERDAM, OCT 18, 2018 Today Recommendation Engines are At the Core of Digitization in Retail Product Recommendations Effective product recommendation algorithms has become the new standard in online retail — directly affecting revenue streams and the shopping experience. Logistics/Delivery Routing recommendations allows companies to save money on routing and delivery, and provide better and faster service. Promotion recommendations Building powerful personalized promotion engines is another area within retail that requires input from multiple data sources, and real-time, session based queries, which is an ideal task to solve with Neo4j.
- 19. AMSTERDAM, OCT 18, 2018 ... and Recommendation Engines are at the core of: Content Recommendations Content recommendation algorithms are the basis to use portals providing value added content — directly affecting the behaviour of the users and have them stay on the web page Fraud Taking timely action based on patterns / recommendations you find inside connected data . May require input from multiple data sources, and real-time, session based queries, which is an ideal task to solve with Neo4j. Social Networks Building powerful personalized engines to recommend new contacts, friends, based on patterns, preferences, status „friends-of-friends“ taking advantage of the value of connected data
- 20. AMSTERDAM, OCT 18, 2018 Why Graph Based Recommendation Engines? • Increase revenue • Create Higher Engagement • Mitigate RiskValue • Real-Time capabilities • Ability to use the most recent transaction data • Flexibility to incorporate new data sourcesPerformance
- 21. AMSTERDAM, OCT 18, 2018 The Impact of Bad Recommendations Characteristic Impact for Recommendations Examples • “Batch Oriented Recommendation” • Unable to react on real-time changes • Unable to fulfill real-time needs • Recommending “out-of-stock” products • Content recommendation, eg news: the latest news are the most important ones • Lack of Performance • Recommendation slow down the user interaction • Recommendation alternatives limited • Delayed response time lead to customer dissatisfaction • Recommend just the obvious (“similarities”) and inability to recommend more complex scenarios (Account specific and product specific and buying history and …) • Limited by Data Connections • Recommendations are limited by number of hops • Inability to recommend more complex correlations (eg product hierarchies and dependencies) • No complex recommendation algorithms supported • Missing Feedback Loop • Inability to react on Feedback • Customer never picks Top 3 recommendations • Recommendations are getting meaningless • No Graph Algorithm support • Limitations on Machine Learning approaches • “Centrality” for Products to be recommend can be essential
- 22. AMSTERDAM, OCT 18, 2018 Case Studies
- 23. AMSTERDAM, OCT 18, 2018 Case studySolving real-time recommendations for the World’s largest retailer. Challenge • In its drive to provide the best web experience for its customers, Walmart wanted to optimize its online recommendations. • Walmart recognized the challenge it faced in delivering recommendations with traditional relational database technology. • Walmart uses Neo4j to quickly query customers’ past purchases, as well as instantly capture any new interests shown in the customers’ current online visit – essential for making real-time recommendations. Use of Neo4j “As the current market leader in graph databases, and with enterprise features for scalability and availability, Neo4j is the right choice to meet our demands”. - Marcos Vada, Walmart • With Neo4j, Walmart could substitute a heavy batch process with a simple and real-time graph database. Result/Outcome
- 24. AMSTERDAM, OCT 18, 2018 Case studyeBay Tackles eCommerce Delivery Service Routing with Neo4j Challenge • The queries used to select the best courier for eBays routing system were simply taking too long and they needed a solution to maintain a competitive service. • The MySQL joins being used created a code base too slow and complex to maintain. • eBay is now using Neo4j’s graph database platform to redefine e-commerce, by making delivery of online and mobile orders quick and convenient. Use of Neo4j • With Neo4j eBay managed to eliminate the biggest roadblock between retailers and online shoppers: the option to have your item delivered the same day. • The schema-flexible nature of the database allowed easy extensibility, speeding up development. • Neo4j solution was more than 1000x faster than the prior MySQL Soltution. Our Neo4j solution is literally thousands of times faster than the prior MySQL solution, with queries that require 10-100 times less code. Result/Outcome – Volker Pacher, eBay
- 25. AMSTERDAM, OCT 18, 2018 Example Recommendation Solution Architecture
- 26. Neo4j Database Cluster Neo4j APOC Recommen dation Algorithms (Scheduled) Management Dashboard Neo4j Bolt Driver Data Ingest Mgmt. … Customer Data Sources / Systems / Applications Legend: Neo4j Provided Components Custom built Neo4j/Customer Customer/SI Batch Data Buffering (Queue) Real-Time Admin UI System Specific Adapters / Scripts / Connecters Admin / Superuser Apps Websites Affiliate Programs Points of sale User Interface Retail Web Shop functionality / Shipment / etc.
- 27. AMSTERDAM, OCT 18, 2018 Why Graph is Superior for Recommendation Engines Recommendation Requirement Traditional Approaches Neo4j Approach Usage of connected data over unlimited amount of „hops“ Complex queries with hundreds of join tables Simple single query traverses all enterprise systems Real-time 360 degree view on data within your System Performance limitations with increasing number of connections / hops Traversing over connections in near real-time provided Effort required to add additional data sources to support reco Days to weeks to rewrite schema and queries Minutes to draw new data connections Time to deployment Months to years Weeks to months Response time to Recommendations Minutes to hours per query Milliseconds per query Machine Learning Enablement Static Database scheme leads to static processes ML algorithms can use Graph algorithms and take advantage of connected data Bottom line Long, ineffective and expensive Easy, fast and affordable
- 28. AMSTERDAM, OCT 18, 2018 Why Graph is Superior for Recommendation Engines Recommendation Requirement Traditional Approaches Neo4j Approach Usage of connected data over unlimited amount of „hops“ Complex queries with hundreds of join tables Simple single query traverses all enterprise systems Real-time 360 degree view on data within your System Performance limitations with increasing number of connections / hops Traversing over connections in near real-time provided Effort required to add additional data sources to support reco Days to weeks to rewrite schema and queries Minutes to draw new data connections Time to deployment Months to years Weeks to months Response time to Recommendations Minutes to hours per query Milliseconds per query Machine Learning Enablement Static Database scheme leads to static processes ML algorithms can use Graph algorithms and take advantage of connected data Bottom line Long, ineffective and expensive Easy, fast and affordable A Fortune 500 customer brought in Neo4j to improve content recommendations quality... and will decommission 48 ‘wide column store’ servers (half a million USD in list EC2 hosting costs) in favor of a *3-machine* Neo4j cluster which handles the same load.
- 29. AMSTERDAM, OCT 18, 2018 How Neo4j Differentiates from other Databases Visualization Queries Processing Storage Non-Native Graph DBNative Graph DB RDBMS Optimized for graph workloads
- 30. AMSTERDAM, OCT 18, 2018 Neo4j powered Recommendation Engine Characteristic Benefit for Recommendation Solution • Agility • Constant learning of recommendations given feedback enabled • Enabled for Future Requirements • Solution can be built iteratively • Fast implementation cycles • Schema free DB supports “connect anything” • Intuitiveness • Enable Business Analysts to use technology • All channels and data sources can be easily connected • Speed • Unlimited number of traversals to detect potential recommendations • Response time enables fraud prevention • Leverage Data Connections • 360 degree customer view enabled / provided • Scalability • Hardware efficiency with real-time patterns • TCO/ROI • Adding on top of existing infrastructure protects investments
- 31. Graph-Enhanced AI & ML
- 32. AMSTERDAM, OCT 18, 2018
- 33. AMSTERDAM, OCT 18, 2018 Graphs Provide Connections & Context for ML and AI
- 34. AMSTERDAM, OCT 18, 2018 Knowledge Graphs GraphConnect speakers 2015-2017
- 35. AMSTERDAM, OCT 18, 2018 What Your AI & ML Looks Like Today
- 36. AMSTERDAM, OCT 18, 2018
- 37. AMSTERDAM, OCT 18, 2018
- 38. AMSTERDAM, OCT 18, 2018 “Increasingly we're learning that you can make better predictions about people by getting all the information from their friends and their friends’ friends than you can from the information you have about the person themselves” — Dr. James Fowler
- 39. AMSTERDAM, OCT 18, 2018
- 40. AMSTERDAM, OCT 18, 2018
- 41. AMSTERDAM, OCT 18, 2018 Connected Feature Extraction
- 42. AMSTERDAM, OCT 18, 2018 Four Pillars of Graph-Enhanced AI/ML 1. Knowledge Graphs Context for Decisions 2. Connected Feature Extraction Context for Credibility 4. AI Explainability3. Graph Accelerated AI Context for Efficiency Context for Accuracy
- 43. Relational WorldGraph World Morpheus Generated SQL Graph View HDFS . Hive Metadata SQL Views SQL Database Morpheus: Graph-Relational, Spark-Based Workbench Neo4j This is coming, not available yet!
- 44. GDPR Compliance
- 45. AMSTERDAM, OCT 18, 2018 GDPR Summary • GDPR = General Data Protection Regulation • Adopted by the EU Parliament on 24th May 2016 • Will apply from 25th May 2018 • Applies to both Controllers and Processors • Applies to organisations operating within the EU, as well as organisations outside the EU that offer goods or services to individuals in the EU. • Covers a broad definition of personal data • Defines lawful basis for processing personal data, which include consent and contract • Defines significant fines for non-compliance
- 46. AMSTERDAM, OCT 18, 2018 Individual Rights Under GDPR Right to be informed Right of access Right to rectification Right to erasure Right to restriction of processing Right to data portability Right to object Rights regarding automated decision making
- 47. AMSTERDAM, OCT 18, 2018 Key GDPR Requirements Organizations that embrace the new GDPR regulations and provide the right levels of transparency and traceability for personal information have a big opportunity to win the hearts, minds and business of consumers. What data do you have? Is it accurate? Where is the data stored? How and when did you obtain the data? Why do you have the data? Who has access to the data? Do you have permission to use the data? For what purpose? Is the data secure? How does the data travel through your systems? Does the data ever cross international borders?
- 48. AMSTERDAM, OCT 18, 2018 GDPR: Risk Mitigation vs. Competitive Advantage Be a leader and have a solution ready on time Improve Brand Reduce Risk Leverage connected data to drive analytics for threat detection & business forecasts Competitive Advantage Spend is strategic Increase ROI Reduce Risk Become a trusted enterprise, delight customers and DPA Increase CSAT Become Trusted Improve Brand Strategic solution ensures data governance and solution maintenance Reduce Risk Reduce Cost Stay on the sidelines to see what others are doing Increased Risk Look to get by with bare minimum solution Increased Risk Spend is sunk investment to just mitigate risk Low to No ROI Unknown Risk Mitigation Solution results in less than happy subjects, DPO and DPA Lower CSAT Minimal Risk Reduction Focus on data governance and solution maintenance is low Increased Risk Increased Cost
- 49. OSLO, MAY 9, 2018 Why Graphs?
- 50. AMSTERDAM, OCT 18, 2018 GDPR needs: Connected Data & Visualization Graph database is the perfect solution to this vast amount of connected data; traditional approaches with an RDBMS or other NoSQL databases just cannot cut it
- 51. AMSTERDAM, OCT 18, 2018 Graph Database is the Right GDPR Foundation Neo4j includes powerful visualization tools that enable you to model and track the movement of sensitive data through your systems
- 52. OSLO, MAY 9, 2018 Data Modeling and Definition 1 Data Transformation2 Consent Management 3 Entitlement 4 GRAPHS IN METADATA MANAGEME NT
- 53. OSLO, MAY 9, 2018 #1 Data Modeling and Definition
- 54. AMSTERDAM, OCT 18, 2018 Party CUST_SCHE MA Party First Name CUST_SCH EMA_PART Y.FIRST_NM CUST_SC HEMA.PA RTY Party Last Name CUSTOMER NAMECUSTOMER CUST_SCH EMA_PART Y.LAST_NM Enterprise Ontology Application Logical Model Physical Schema CUST_SC HEMA.RO LE
- 55. OSLO, MAY 9, 2018 #2 Data Transformation
- 56. AMSTERDAM, OCT 18, 2018 ETL_P ROC_1 SALES_S CHEMA Normal ize_Da teSLS_SC HEMA.P RODUC T SLS_SCH EMA.SAL ES.DATE SLS_S CHEMA .SALES #2 Data Transformation Channel _Normal ization SLS_SCH EMA.SAL ES.CHAN NEL Integration MiddlewareOperational Systems Time.time _key Time.day _of_week Enterprise DWH Billing Syste m EDW H CDE: Transa ction_D ate Star_Sch ema Star_S chema .Time
- 57. OSLO, MAY 9, 2018 #3 Consent managemen
- 58. OSLO, MAY 9, 2018 #3 Consent Management + MDM Amsterdam NL K.Vegter +31623900 4… kees@neo 4j.com kees@gm ail.com { contrib: ‘XYZ’, permittedFor: [UC1,UC4], consentUntil : 31-12-19 } { contrib: ‘internal’, permittedFor: [UC3], consentUntil : 31-12-20 } { contrib: ‘internal’, permittedFor: [UC3], consentUntil : 31-12-20 } { contrib: ‘internal’, permittedFor: [UC3], consentUntil : 31-12-20 } { contrib: ‘LMN’, permittedFor: [UC2,UC6], consentUntil : 31-12-20 }
- 59. OSLO, MAY 9, 2018 #4 Entitlement
- 60. OSLO, MAY 9, 2018 #4 Entitlement User 1 User 3User 2 Exclusi on List G1 Resourc e 1 Group 1 Resourc e 2 Group 3
- 61. AMSTERDAM, OCT 18, 2018 #4 Entitlement#3 Consent Management #2 Data Transformation #1 Data Modelling and Definition Graphs in Metadata Management and Data Governance # …
- 62. AMSTERDAM, OCT 18, 2018 Why Graph is Superior for GDPR GDPR Task Traditional Approaches Modern Neo4j Approach Trace data through enterprise systems Complex queries with hundreds of join tables Simple single query traverses all enterprise systems Preserve the integrity of data lineage Broken data paths and lineage, especially with NoSQL databases Continuous, unbroken data paths at all times Effort required to add new data and systems Days to weeks to rewrite schema and queries Minutes to draw new data connections Time to deployment Months to years Weeks to months Response time to GDPR requests Minutes to hours per query Milliseconds per query Form of GDPR responses Text reports that are not visual and prove very little Visuals of personal data and the path it follows through your systems Bottom line Long, ineffective and expensive Easy, fast and affordable
- 63. OSLO, MAY 9, 2018 Dashboards & Visual Reports
- 64. Personal Data Map Role Based Dashboards - Subject View
- 66. Role Based Dashboards - Management View
- 68. Data Lineage Report for ‘John Doe’
- 69. John Doe
- 71. AMSTERDAM, OCT 18, 2018 Graph Database is the Right GDPR Foundation Extract GDPR Events/Data Marketing CRM Customer Service Online Store Logistics Financials
- 72. AMSTERDAM, OCT 18, 2018 Conclusion (graphs)-[:ARE]-> (everywhere) and (Solutions)-[:NEED]-> (graphs)
- 73. AMSTERDAM, OCT 18, 2018 Who can help? Neo4j Platform Graph Transactions Graph Analytics Data Integration Development & Admin Analytics Tooling Drivers & APIs Discovery & Visualization Developers Admins Applications Business Users Data Analysts Data Scientists 3rd Party Tools “The Graph Advantage” Domain know-how Professional Services PS Packages Graph Based Solution Professional Services: - Extend and leverage Domain Expertise - Best Practices - Using Building Blocks - Don’t “re-invent the wheel” - Speed up development and deployment - Access to Neo4j infrastructure (Development, Support, Product management)