[Apache Kafka® Meetup by Confluent] Graph-based stream processing
1 like250 views
The document introduces graph-based stream processing, focusing on the capabilities and applications of graph databases, particularly Memgraph. It explains key concepts such as nodes, relationships, and the Cypher query language, and illustrates practical uses of graph analytics in various applications like social networks and recommendation engines. Additionally, it outlines how to set up a stream processing architecture using graph algorithms to analyze real-time data, specifically using a Twitter dataset.
![Cypher query language
MATCH (u:Customer{customer_id:'customer-one'})-[:BOUGHT]->
(p:Product)<-[:BOUGHT]-(peer:Customer)-[:BOUGHT]->
(reco:Product)
WHERE not (u)-[:BOUGHT]->(reco)
RETURN reco as Recommendation, count(*) as Frequency ORDER
BY Frequency DESC LIMIT 5;
SQL
SELECT product.product_name as Recommendation, count(1) as
Frequency
FROM product, customer_product_mapping, (SELECT
cpm3.product_id, cpm3.customer_id
FROM Customer_product_mapping cpm,
Customer_product_mapping cpm2, Customer_product_mapping
cpm3
WHERE cpm.customer_id = ‘customer-one’
and cpm.product_id = cpm2.product_id
and cpm2.customer_id != ‘customer-one’
and cpm3.customer_id = cpm2.customer_id
and cpm3.product_id not in (select distinct product_id
FROM Customer_product_mapping cpm
WHERE cpm.customer_id = ‘customer-one’)
) recommended_products
WHERE customer_product_mapping.product_id =
product.product_id
and customer_product_mapping.product_id in
recommended_products.product_id
and customer_product_mapping.customer_id =
recommended_products.customer_id
GROUP BY product.product_name
ORDER BY Frequency desc](https://image.slidesharecdn.com/apachekafkameetupbyconfluentgraph-basedstreamprocessing-220406005250/85/Apache-Kafka-Meetup-by-Confluent-Graph-based-stream-processing-8-320.jpg)
![[Apache Kafka® Meetup by Confluent] Graph-based stream processing](https://image.slidesharecdn.com/apachekafkameetupbyconfluentgraph-basedstreamprocessing-220406005250/85/Apache-Kafka-Meetup-by-Confluent-Graph-based-stream-processing-13-320.jpg)
![[Apache Kafka® Meetup by Confluent] Graph-based stream processing](https://image.slidesharecdn.com/apachekafkameetupbyconfluentgraph-basedstreamprocessing-220406005250/85/Apache-Kafka-Meetup-by-Confluent-Graph-based-stream-processing-33-320.jpg)
[Apache Kafka® Meetup by Confluent] Graph-based stream processing
- 2. This is Katarina ● Born at the Croatian coast, in the town of Šibenik ● Master’s in Mathematics and Computer science from the Faculty of Science in Zagreb ● I work as a Developer Relations Engineer at Memgraph ● I love to travel, cook and eat tasty food (check out my Instagram)
- 3. This is Ivan ● Born and living in Croatia ● Master’s in Computer science from the Faculty of Electrical Engineering and Computing in Zagreb ● I work as a Developer Relations Engineer at Memgraph ● I love to travel, cook and watch Netflix too much
- 4. The graph data model When to use a graph database?
- 5. What are graphs? A graph is a network structure that consists of a set of nodes (vertices) and a set of relationships (edges) connecting them. ● Nodes - structures that represent entities ● Relationships - connections between these entities ● Properties - associated values (key-value pairs) belonging to either nodes or relationships Labeled property graph model
- 6. Graph database vs relational database
- 7. Cypher query language Cypher is the most widely adopted, fully-specified, and open query language for property graph databases. It provides an intuitive way to work with property graphs. Cypher contains: - clauses such as MATCH, DELETE, SET, RETURN… - functions such as round(), cos(), toString()... - custom procedures written in Python, C/C++ and Rust
- 8. Cypher query language MATCH (u:Customer{customer_id:'customer-one'})-[:BOUGHT]-> (p:Product)<-[:BOUGHT]-(peer:Customer)-[:BOUGHT]-> (reco:Product) WHERE not (u)-[:BOUGHT]->(reco) RETURN reco as Recommendation, count(*) as Frequency ORDER BY Frequency DESC LIMIT 5; SQL SELECT product.product_name as Recommendation, count(1) as Frequency FROM product, customer_product_mapping, (SELECT cpm3.product_id, cpm3.customer_id FROM Customer_product_mapping cpm, Customer_product_mapping cpm2, Customer_product_mapping cpm3 WHERE cpm.customer_id = ‘customer-one’ and cpm.product_id = cpm2.product_id and cpm2.customer_id != ‘customer-one’ and cpm3.customer_id = cpm2.customer_id and cpm3.product_id not in (select distinct product_id FROM Customer_product_mapping cpm WHERE cpm.customer_id = ‘customer-one’) ) recommended_products WHERE customer_product_mapping.product_id = product.product_id and customer_product_mapping.product_id in recommended_products.product_id and customer_product_mapping.customer_id = recommended_products.customer_id GROUP BY product.product_name ORDER BY Frequency desc
- 9. Graph analytics in action When to use graph analytics?
- 10. Graph analytics Graph analytics, also called network analysis, generates insights hidden in the relationships of the network structure. Some common algorithms include: ● Clustering & community detection ● Connected components ● PageRank ● Shortest path ● BFS & DFS
- 11. Graph analytics in action Google was built on the PageRank algorithm measuring the importance of web pages. Facebook’s social graph uses Community Detection to infer unknown data about their users based on similar network behavior of other users to power their ad-targeting engine. Amazon uses Collaborative Filtering to deliver high quality real-time product recommendations. Pinterest uses Random Walks and Graph-Machine Learning to deliver high-quality personalized recommendations responsible for more than 80% of all user engagements. “Graph-Machine Learning features proved the most valuable of all other features when determining the quality and relevancy of our dish and restaurant recommendations.”
- 12. Social networks
- 16. Fraud detection
- 17. Graph stream processing Analyze your data in real-time instead of in batches
- 19. Graph stream processing ● Stream processing is a type of big data architecture in which the data is analyzed in real-time ● It is used whenever you need to extract additional information from you data as it’s being consumed ● The processing happens asynchronously - the data source and the stream processing work independently without waiting for a response from one another
- 20. Memgraph Memgraph is a platform for graph computation on streaming data powered by an in-memory graph database. Memgraph is used to: ● Store graph data in memory ● Run graph analytics ● Analyze streaming data
- 21. GQLAlchemy GQLAlchemy is a fully open-source Python library. It is an Object Graph Mapper (OGM) - a link between Graph Database objects and Python objects. GQLAlchemy includes: ● OGM capabilities ● Query builder ● On-disk storage ● Graph schema validation Jupyter Notebook Demo: ● https://github.com/memgraph/jupyter-memgrap h-tutorials/tree/main/twitter_network_analysis
- 22. The stream processing architecture What do you need for stream processing?
- 24. Demo time!
- 26. Analyzing a Twitter network We are going to find the most christmassy person on Twitter using the hashtag #christmas by analyzing retweets of the original tweet: ● Dynamic PageRank algorithm is used to find the most influential account ● Dynamic community detection algorithm is used to figure out which accounts belong to the same community
- 27. Graph stream processing with Memgraph
- 28. 1. Transformation module Transforming Kafka messages into graph objects:
- 29. 2. PageRank Set up everything and run the dynamic PageRank algorithm:
- 30. 3. Create and start the stream
- 32. Stream processing in Memgraph 1. Create a transformation module in Python 2. Run the PageRank and community detection algorithms with GQLAlchemy 3. Create and start the stream with GQLAlchemy 4. Create a trigger and module to send PageRank results back to a Kafka topic
- 34. Want to play with the Twitter dataset? Check out the Memgraph Playground: https://playground.memgraph.com/sandbox/twitter-christmas-retweets
- 35. Thank you for your attention!
- 36. www.memgraph.io If you like what we do, throw us a star! ⭐ https://github.com/memgraph/memgraph Check out the Twitter network analysis repository! https://github.com/memgraph/twitter-network-analysis
- 37. www.memgraph.io Join our Discord server! memgr.ph/discord supe_katarina katarina.supe@memgraph.io ivan_g_despot ivan.despot@memgraph.io