STING: Spatio-Temporal Interaction Networks and Graphs for Intel Platforms
1 like2,289 views
This document summarizes the STING project, which aims to develop techniques for analyzing massive graphs in a streaming fashion on Intel platforms. It outlines the overall streaming approach taken by STING to process batches of graph updates and maintain metrics and properties. It describes the STINGER graph data structure and discusses initial results for maintaining clustering coefficients and connected components under a stream of insertions and deletions. It also outlines work in progress on community detection algorithms. The document provides details on experimental setups and performance results for the different graph analysis techniques.
![Experimental setup
Unless otherwise noted
Line Model Speed (GHz) Sockets Cores
Nehalem X5570 2.93 2 4
Westmere E7-8870 2.40 4 10
• Westmere loaned by Intel (thank you!)
• All memory: 1067MHz DDR3, installed appropriately
• Implementations: OpenMP, gcc 4.6.1, Linux ≈ 3.0 kernel
• Artificial graph and edge stream generated by
R-MAT[Chakrabarti, et al.].
• Scale x, edge factor f ⇒ 2x vertices, ≈ f · 2x edges.
• Edge actions: 7/8th insertions, 1/8th deletions
• Results over five batches of edge actions.
• Caveat: No vector instructions, low-level optimizations yet.
14 / 33](https://image.slidesharecdn.com/gt-sting-for-intel-slides-110929162201-phpapp02/85/STING-Spatio-Temporal-Interaction-Networks-and-Graphs-for-Intel-Platforms-16-320.jpg)
![Clustering coefficients
• Used to measure
“small-world-ness”[Watts and Strogatz] i m
and potential community structure
• Larger clustering coefficient ⇒ more v
inter-connected
j n
• Roughly the ratio of the number of actual
to potential triangles
• Defined in terms of triplets.
• i – v – j is a closed triplet (triangle).
• m – v – n is an open triplet.
• Clustering coefficient:
# of closed triplets / total # of triplets
• Locally around v or globally for entire graph.
15 / 33](https://image.slidesharecdn.com/gt-sting-for-intel-slides-110929162201-phpapp02/85/STING-Spatio-Temporal-Interaction-Networks-and-Graphs-for-Intel-Platforms-17-320.jpg)
![Updating triangle counts
Given Edge {u, v } to be inserted (+) or deleted (-)
Approach Search for vertices adjacent to both u and v , update
counts on those and u and v
Three methods
Brute force Intersect neighbors of u and v by iterating over each,
O(du dv ) time.
Sorted list Sort u’s neighbors. For each neighbor of v , check if in
the sorted list.
Compressed bits Summarize u’s neighbors in a bit array. Reduces
check for v ’s neighbors to O(1) time each.
Approximate with Bloom filters. [MTAAP10]
All rely on atomic addition.
16 / 33](https://image.slidesharecdn.com/gt-sting-for-intel-slides-110929162201-phpapp02/85/STING-Spatio-Temporal-Interaction-Networks-and-Graphs-for-Intel-Platforms-18-320.jpg)
![Connected components: Deleted edges
The difficult case
• Very few deletions
matter.
• Determining which
matter may require a
large graph search.
• Re-running static
component
detection.
• (Long history, see
related work in
[MTAAP11].)
• Coping mechanisms:
• Heuristics.
• Second level of
batching.
20 / 33](https://image.slidesharecdn.com/gt-sting-for-intel-slides-110929162201-phpapp02/85/STING-Spatio-Temporal-Interaction-Networks-and-Graphs-for-Intel-Platforms-23-320.jpg)
![Agglomerative community detection
Parallel greedy, agglomerative partitoning [PPAM11]
• Score edges by optimization criteria.
• Chose a maximal, heavy-weight matching.
• Negate edge scores if minimizing conductance.
• Contract those edges.
• Mimics sequential optimizers, but produces different results.
24 / 33](https://image.slidesharecdn.com/gt-sting-for-intel-slides-110929162201-phpapp02/85/STING-Spatio-Temporal-Interaction-Networks-and-Graphs-for-Intel-Platforms-27-320.jpg)
STING: Spatio-Temporal Interaction Networks and Graphs for Intel Platforms
- 1. STING: Spatio-Temporal Interaction Networks and Graphs for Intel Platforms David Bader, Jason Riedy, Henning Meyerhenke, David Ediger, Timothy Mattson 29 August 2011
- 2. Outline Motivation Technical Overall streaming approach Clustering coefficients Connected components Community detection (in progress) Related Pasqual, a scalable de novo sequence assembler Plans 2 / 33
- 3. Exascale Data Analysis Health care Finding outbreaks, population epidemiology Social networks Advertising, searching, grouping Intelligence Decisions at scale, regulating algorithms Systems biology Understanding interactions, drug design Power grid Disruptions, conservation Simulation Discrete events, cracking meshes 3 / 33
- 4. Graphs are pervasive • Sources of massive data: petascale simulations, experimental devices, the Internet, scientific applications. • New challenges for analysis: data sizes, heterogeneity, uncertainty, data quality. Astrophysics Bioinformatics Social Informatics Problem Identifying target Problem Emergent behavior, Problem Outlier detection proteins information spread Challenges Massive data Challenges Data Challenges New analysis, sets, temporal variation heterogeneity, quality data uncertainty Graph problems Matching, Graph problems Centrality, Graph problems Clustering, clustering clustering flows, shortest paths 4 / 33
- 5. These are not easy graphs. Yifan Hu’s (AT&T) visualization of the Livejournal data set 5 / 33
- 6. Intel’s non-numeric computing program Supporting massive, dynamic graph analysis across the spectrum of Intel platforms. Interactions • Workshop on Scalable Graph Libraries • Co-sponsored by Georgia Tech & PNNL • Hosted at Georgia Tech, 29-30 June 2011 • Attended by 33 from sponsors, industry, and academia. (Timothy Mattson, Roger Golliver, Aydin Bulu¸, and John c Gilbert) • Hosting Intel-loaned Westmere server • Quad E7-8870 with 0.25TiB of memory • Active access by Guy Blelloch’s benchmark group, John Gilbert’s KDT group • Program-related access for PAPI counter support • Graph analysis minisymposium at SIAM PP, Feb. 2012 6 / 33
- 7. 10th DIMACS Implementation Challenge Graph Partitioning and Graph Clustering • Many application areas identify vertex subsets with many internal and few external edges. Problems addressed include: • What are the communities within an (online) social network? • How do I speed up a numerical simulation by mapping it efficiently onto a parallel computer? • How must components be organized on a computer chip such that they can communicate efficiently with each other? • What are the segments of a digital image? • Which functions are certain genes (most likely) responsible for? • 12-13 February 2012, Atlanta, Georgia • Paper deadline: 21 October 2011 • Co-sponsored by DIMACS, by the Command, Control, and Interoperability Center for Advanced Data Analysis (CCICADA); Pacific Northwest National Laboratory; Sandia National Laboratories; and Deutsche Forschungsgemeinschaft (DFG). http://www.cc.gatech.edu/dimacs10/ 7 / 33
- 8. Overall streaming approach Protein interactions, Giot et al., “A Protein Interaction Map of Drosophila melanogaster”, Jason’s network via LinkedIn Labs Science 302, 1722-1736, 2003. Assumptions • A graph represents some real-world phenomenon. • But not necessarily exactly! • Noise comes from lost updates, partial information, ... 8 / 33
- 9. Overall streaming approach Protein interactions, Giot et al., “A Protein Interaction Map of Drosophila melanogaster”, Jason’s network via LinkedIn Labs Science 302, 1722-1736, 2003. Assumptions • We target massive, “social network” graphs. • Small diameter, power-law degrees • Small changes in massive graphs often are unrelated. 8 / 33
- 10. Overall streaming approach Protein interactions, Giot et al., “A Protein Interaction Map of Drosophila melanogaster”, Jason’s network via LinkedIn Labs Science 302, 1722-1736, 2003. Assumptions • The graph changes but we don’t need a continuous view. • We can accumulate changes into batches... • But not so many that it impedes responsiveness. 8 / 33
- 11. Difficulties for performance • What partitioning methods apply? • Geometric? Nope. • Balanced? Nope. • Is there a single, useful decomposition? Not likely. • Some partitions exist, but they don’t often help with balanced bisection or memory locality. • Performance needs new approaches, not just standard scientific Jason’s network via LinkedIn Labs computing methods. 9 / 33
- 12. STING’s focus Control action prediction summary Source data Simulation / query Viz • STING manages queries against changing graph data. • Visualization and control often are application specific. • Ideal: Maintain many persistent graph analysis kernels. • Keep one current snapshot of the graph resident. • Let kernels maintain smaller histories. • Also (a harder goal), coordinate the kernels’ cooperation. 10 / 33
- 13. STING and STINGER Batch of insertions / deletions Pre-process batch: Sort by source vertex, reconcile ins/del. Pre-change hook STINGER Alter graph (may “age off”old edges) graph Affected vertices Post-change hook Change in metric • Batches provide two levels of parallelism. • Busy loci of change: Know to share the busy points. • Scattered changes: Parallel across (likely) independent changes. • The massive graph is maintained in a data structure named STINGER. 11 / 33
- 14. STINGER STING Extensible Representation: • Rule #1: No explicit locking. • Rely on atomic operations. • Massive graph: Scattered updates, scattered reads rarely conflict. • Use time stamps for some view of time. 12 / 33
- 15. Initial results Prototype STING and STINGER Monitoring the following properties: 1 clustering coefficients, 2 connected components, and 3 community structure (in progress). High-level • Support high rates of change, over 10k updates per second. • Performance scales somewhat with available processing. • Gut feeling: Scales as much with sockets as cores. http://www.cc.gatech.edu/~bader/code.html 13 / 33
- 16. Experimental setup Unless otherwise noted Line Model Speed (GHz) Sockets Cores Nehalem X5570 2.93 2 4 Westmere E7-8870 2.40 4 10 • Westmere loaned by Intel (thank you!) • All memory: 1067MHz DDR3, installed appropriately • Implementations: OpenMP, gcc 4.6.1, Linux ≈ 3.0 kernel • Artificial graph and edge stream generated by R-MAT[Chakrabarti, et al.]. • Scale x, edge factor f ⇒ 2x vertices, ≈ f · 2x edges. • Edge actions: 7/8th insertions, 1/8th deletions • Results over five batches of edge actions. • Caveat: No vector instructions, low-level optimizations yet. 14 / 33
- 17. Clustering coefficients • Used to measure “small-world-ness”[Watts and Strogatz] i m and potential community structure • Larger clustering coefficient ⇒ more v inter-connected j n • Roughly the ratio of the number of actual to potential triangles • Defined in terms of triplets. • i – v – j is a closed triplet (triangle). • m – v – n is an open triplet. • Clustering coefficient: # of closed triplets / total # of triplets • Locally around v or globally for entire graph. 15 / 33
- 18. Updating triangle counts Given Edge {u, v } to be inserted (+) or deleted (-) Approach Search for vertices adjacent to both u and v , update counts on those and u and v Three methods Brute force Intersect neighbors of u and v by iterating over each, O(du dv ) time. Sorted list Sort u’s neighbors. For each neighbor of v , check if in the sorted list. Compressed bits Summarize u’s neighbors in a bit array. Reduces check for v ’s neighbors to O(1) time each. Approximate with Bloom filters. [MTAAP10] All rely on atomic addition. 16 / 33
- 19. Batches of 10k actions Brute force Bloom filter Sorted list 1.7e+04 3.4e+04 8.8e+04 1.2e+05 8.8e+04 1.4e+05 105.5 1.5e+04 1.3e+05 1.2e+05 5.1e+03 2.4e+04 2.2e+04 Updates per seconds, both metric and STINGER 3.9e+03 2.0e+04 1.7e+04 q q q q q q q q q 5 q q q q 10 q q qq q q q q q q q q Machine 104.5 q a 4 x E7−8870 q q a 2 x X5570 q q q q q 104 103.5 0 20 40 60 80 0 20 40 60 80 0 20 40 60 80 Threads Graph size: scale 22, edge factor 16 17 / 33
- 20. Different batch sizes Brute force Bloom filter Sorted list 105.5 105 q q q q q q q q q q Updates per seconds, both metric and STINGER q q q 100 104.5 q q q q q q q q q q 104 q qq q q 103.5 q 105.5 105 q q q q q q q q Machine 1000 104.5 q q 4 x E7−8870 q q q 2 x X5570 104 q q q q q q 103.5 105.5 q q q q q q q 105 qq q q q qqq q q q q q q 10000 q 4.5 q 10 q q q q q q 104 103.5 0 20 40 60 80 0 20 40 60 80 0 20 40 60 80 Threads Graph size: scale 22, edge factor 16 18 / 33
- 21. Connected components • Maintain a mapping from vertex to component. • Global property, unlike triangle counts • In “scale free” social networks: • Often one big component, and • many tiny ones. • Edge changes often sit within components. • Remaining insertions merge components. • Deletions are more difficult... 19 / 33
- 22. Connected components • Maintain a mapping from vertex to component. • Global property, unlike triangle counts • In “scale free” social networks: • Often one big component, and • many tiny ones. • Edge changes often sit within components. • Remaining insertions merge components. • Deletions are more difficult... 19 / 33
- 23. Connected components: Deleted edges The difficult case • Very few deletions matter. • Determining which matter may require a large graph search. • Re-running static component detection. • (Long history, see related work in [MTAAP11].) • Coping mechanisms: • Heuristics. • Second level of batching. 20 / 33
- 24. Deletion heuristics Rule out effect-less deletions • Use the spanning tree by-product of static connected component algorithms. • Ignore deletions when one of the following occur: 1 The deleted edge is not in the spanning tree. 2 If the endpoints share a common neighbor∗ . 3 If the loose endpoint can reach the root∗ . • In the last two (∗), also fix the spanning tree. Rules out 99.7% of deletions. 21 / 33
- 25. Connected components: Performance 106 2.4e+03 1.6e+04 6.4e+03 3.2e+03 2.0e+03 5 10 100 Updates per seconds, both metric and STINGER 104 103 106 1.7e+04 7.7e+04 1.4e+04 1.9e+04 2.0e+04 105 Machine 1000 104 a 4 x E7−8870 a 2 x X5570 3 10 106 5.8e+04 1.3e+05 1.5e+05 5.5e+04 1.1e+05 5 10 10000 104 103 12 4 6 8 12 16 24 32 40 48 56 64 72 80 Threads Graph size: scale 22, edge factor 16 22 / 33
- 26. Community detection (work in progress) Greedy, agglomerative partitioning • Partition to maximize modularity, minimize conductance, ... Seed set expansion • Grow an optimal / ”relevant” community around selection. • (Work with Jonny Dimond of KIT.) 23 / 33
- 27. Agglomerative community detection Parallel greedy, agglomerative partitoning [PPAM11] • Score edges by optimization criteria. • Chose a maximal, heavy-weight matching. • Negate edge scores if minimizing conductance. • Contract those edges. • Mimics sequential optimizers, but produces different results. 24 / 33
- 28. Performance q • R-MAT on right. 2.5 • Livejournal Modularity q 0.284 • 15M vertex, q 0.286 2.0 q 0.288 184M edge q 0.290 Time (hours) q • 6-12 hours on q 0.292 0.294 E7-8870 q q 0.296 0.298 1.5 • Highly variable CPU q Dual X5570 (2.93GHz) performance. Quad E7−8870 (2.40GHz) q • Algorithm under 1.0 q development. q q 22 23 24 25 26 Processors 25 / 33
- 29. Related: Pasqual A scalable de novo assembler for next-gen gene sequencing Work by David Bader, Henning Meyerhenke, Xing Liu, Pushkar Pande. • Next-generation sequencers produce mountains of small gene sequences. • Assembling into a genome: Yet another large graph problem. • Pasqual forms a compressed overlap graph and traces paths. • Only scalable and correct shared-memory assembler. • Faster and uses less memory than other existing systems. • Evaluation against the few distributed assemblers is ongoing. • Algorithm extends to metagenomics. http://www.cc.gatech.edu/pasqual/ 26 / 33
- 30. Human genome, 33.5Mbp (cov 30) Similar speed, better results Length Code Time (min) N50 (bp) Errors 35 Velvet >12h 1355 683 Edena 451.15 1375 3 ABySS 56.52 1412 521 SOAPdenovo 15.62 1470 485 Pasqual 15.50 1451 5 100 Velvet 132.68 6635 175 Edena 466.12 6545 7 ABySS 92.92 6229 136 SOAPdenovo 18.05 6879 142 Pasqual 19.15 7712 6 27 / 33
- 31. Zebrafish, 61Mbp (cov 30) Far better speed and results Length Code Time (min) N50 (bp) Errors 100 Velvet 256.02 4045 799 Edena >12h 2661 0 ABySS — — — SOAPdenovo 31.83 3998 725 Pasqual 57.27 4535 0 200 Velvet — — — Edena — — — ABySS — — — SOAPdenovo — — — Pasqual 38.07 7911 0 28 / 33
- 32. Performance v. SOAPdenovo 166.67m 104 97.38m 103.5 Coverage 40.32m q 30 q q 50 Time q Code q 21.70m q SoapDenovo q q Pasqual 103 q q q q q q 9.62m q q q 7.22m q q 5.97m q q 102.5 4.65m 1 4 8 2 16 32 40 64 80 Threads 29 / 33
- 33. Plans Community detection Improving the algorithm, pushing into streaming by de-agglomerating and restarting. Seed set expansion Maintaining not only one expanded set, but multiple for high-throughput monitoring. Microbenchmarks Expand on initial promising work on characterizing performance by peak number of memory operations achieved, find bottlenecks by comparing with microbenchmarks. Distributed/PGAS STINGER fits a PGAS model well (think SCC). Interested in exploring distributed algorithms. Packaging Wrap STING into an easily downloaded and installed tool. 30 / 33
- 34. Bibliography I D. Ediger, K. Jiang, E. J. Riedy, and D. A. Bader. Massive streaming data analytics: A case study with clustering coefficients. In Proceedings of the Workshop on Multithreaded Architectures and Applications (MTAAP’10), Apr. 2010. D. Ediger, E. J. Riedy, and D. A. Bader. Tracking structure of streaming social networks. In Proceedings of the Workshop on Multithreaded Architectures and Applications (MTAAP’11), May 2011. K. Madduri and D. A. Bader. Compact graph representations and parallel connectivity algorithms for massive dynamic network analysis. In 23rd IEEE International Parallel and Distributed Processing Symposium (IPDPS), Rome, Italy, May 2009. 31 / 33
- 35. Bibliography II E. J. Riedy, D. A. Bader, K. Jiang, P. Pande, and R. Sharma. Detecting communities from given seeds in social networks. Technical Report GT-CSE-11-01, Georgia Institute of Technology, Feb. 2011. E. J. Riedy, H. Meyerhenke, D. Ediger, and D. A. Bader. Parallel community detection for massive graphs. In Proceedings of the 9th International Conference on Parallel Processing and Applied Mathematics, Torun, Poland, Sept. 2011. 32 / 33
- 36. References I D. Chakrabarti, Y. Zhan, and C. Faloutsos. R-MAT: A recursive model for graph mining. In Proc. 4th SIAM Intl. Conf. on Data Mining (SDM), Orlando, FL, Apr. 2004. SIAM. D. J. Watts and S. H. Strogatz. Collective dynamics of ‘small-world’ networks. Nature, 393(6684):440–442, Jun 1998. 33 / 33