Distributed graph mining
Download as PPTX, PDF2 likes1,133 views
This document discusses distributed graph mining using MapReduce. It describes how partitioning graph data across multiple machines can make processing very large graphs feasible. The document outlines two partitioning techniques - MRGP which assigns partitions sequentially, and DGP which balances partitions based on density. It also discusses how local support counts are adjusted compared to global support when graphs are partitioned across many machines. An experiment environment using Hadoop and both synthetic and real-world graph datasets is also mentioned.
![Local Support Count
• Each individual node has only partial graph
data set.
• Support Count need to be adjusted relative to
the original dataset.
• This adjusted support count is Local Support
Count.
• Local Support Count = Tolerance Rate * Global
Support [Tolerance rate is between 1 and 0]](https://image.slidesharecdn.com/distributedgraphmining-141107112119-conversion-gate02/85/Distributed-graph-mining-15-320.jpg)
Distributed graph mining
- 1. Distributed Graph Mining Presented By Sayeed Mahmud
- 2. Motivation
- 3. Motivation • The reason BigData is here – To make processing data easier which is impossible or overwhelming to process with our existing problem. • Some Graph Database might be too big for a single machine – Easier for a distributed system by sharing load • Graph Database may itself be scattered around the globe – Google search records.
- 4. Distributed Graph Mining • Partition based • Divide the problem into independent sub-problems – Each node of the system can process it independently – Parallel processing – Speedup computation – Enhance scalability of solutions
- 5. Techniques • MRPF • MapReduce – We are mainly interested in this
- 6. Map Reduce • A programming model for distributed platforms. • Proposed by Google • Abundant open source implementations – Hadoop • Divides the problem in to sub-problems to be processed in nodes – Mapping • Combining the processing results – Reduce
- 7. Map Reduce Example • Problem: Find frequency of a word in documents available on a system. …wor d…. word … … …wor d…. … … …wor d…. word … … <word, count> Map Distributed System <word, 2> <word, 1> <word, 2> <word, 2 + 1 + 2 = 5> Reduce
- 8. Graph Mining using Map Reduce • Problem: Find frequent sub-graphs of a graph database in a MapReduce programming model (Local Support 2) Graph Dataset Map Distributed System Run gSpan Run gSpan 3 2 5 Reduce
- 9. Data Partitioning • Performance and load balancing will be depending on Mapping portion – Termed “Partitioning” – Which portion of the graph dataset will go to which – Loss of Data and Load Balancing directly dependent on partitioning. • Two approach – MRGP (Map Reduced Partitioning) – DGP (Density Based Partitioning)
- 10. MRGP • Followed in common Map Reduce problems. • Assigned sequentially • Simple Graph Size (KB) Density G1 1 0.25 G2 2 0.5 G3 2 0.6 G4 1 0.25 G5 2 0.5 G6 2 0.5 G7 2 0.5 G8 2 0.6 G9 2 0.6 G10 2 0.7 G11 3 0.7 G12 3 0.8 4 Partition 6KB Each G1, G2, G3, G4 G5, G6, G7 G8, G9, G10 G11, G12
- 11. DGP • Goes for a balanced distribution • Uses intermediary Bucket • First graphs are sorted according to densities. Graph Size (KB) Density G1 1 0.25 G2 2 0.5 G3 2 0.6 G4 1 0.25 G5 2 0.5 G6 2 0.5 G7 2 0.5 G8 2 0.6 G9 2 0.6 G10 2 0.7 G11 3 0.7 G12 3 0.8 G1 (0.25) G4 (0.25) G2 (0.5) G5 (0.5) G6 (0.5) G7 (0.5) G3 (0.6) G8 (0.6) G9 (0.6) G10 (0.7) G11 (0.7) G12 (0.8)
- 12. DGP cont.. • Lets say bucket count for this demo is 2 • Next we equally distribute the sorted list to two buckets. Bucket 1 Bucket 2 G1 G G2 5 G6 G7 G4 G3 G G8 10 G11 G12 G9 Make 4 PaDrivtiidtei oeancsh iBnu ctkoett ainl 4 Non Empty Sub-Bucket
- 13. DGP Cont.. • Now take one partition from each and form final partitions G1 G G2 5 G6 G7 G4 G3 G G8 10 G11 G12 G9 G1, G2, G3, G8 G4, G5, G9, G10 G6, G11 G7, G12
- 14. Support Count • There are two types of support counts to be considered in distributed graph mining – Global Support Count – Local Support Count • Global Support is the same as in normal graph mining • When each mapper is running individual job it considers local support count.
- 15. Local Support Count • Each individual node has only partial graph data set. • Support Count need to be adjusted relative to the original dataset. • This adjusted support count is Local Support Count. • Local Support Count = Tolerance Rate * Global Support [Tolerance rate is between 1 and 0]
- 16. Loss of Data • Some frequent sub-graph are lost • The loss can be mitigated by choosing an optimal tolerance rate. – Theoretically tolerance rate = 1 means there will be no loss of data. – But usually higher run time.
- 17. Experiment Environment • Language : Perl • MapReduce Framework : Hadoop (0.20.1) • Cluster Size : 5 • Node Specification: – Processor AMD Opteron Quad Core 2.4 GHz – 4GB Main memory
- 18. Data Sets • Synthetic (Size Ranging from 18MB to 69GB) • Real – Chemical Compound Dataset from National Cancer Institute.
- 19. Loss Rate for gSpan Support 30%
- 20. Loss Rate for Gaston and FSG Support 30%
- 21. Runtime
- 22. Thank You