Dato vs GraphX
Download as PPTX, PDF0 likes754 views
This document compares the graph processing frameworks Dato and Spark GraphX. It details experiments run on four datasets from the Stanford Network Collection to test the frameworks' performance on triangle counting, PageRank, and connected components algorithms. The results show that Dato has faster execution times than GraphX for processing large graphs, though GraphX is free while Dato charges fees. Further experiments could compare the frameworks' abilities to combine graph and non-graph computations.
Dato vs GraphX
- 1. DATO VS. SPARK GRAPHX KEIRA ZHOU OCT, 2015 Details: https://github.com/keiraqz/dato-vs-graphx
- 2. SETTINGS • 1 master node and 3 work nodes on AWS • m4.large instances with 8GB of RAM with 2 cores
- 3. DATO • A graph-based, asynchronous, high performance, distributed computation framework written in C++ • 30-days free trial, then a service fee • Install GraphLab Create on the local machine and Dato Distributed on a cluster
- 4. SPARK GRAPHX • Come with Spark import org.apache.spark._ import org.apache.spark.graphx._
- 5. EXPERIMENTS • Graph Algorithms • Triangle-counting • PageRank • Connected Components • Datasets: Stanford Large Network Dataset Collection (SNAP) • Facebook: • Nodes: 4039 | Edges: 88234 | Number of triangles: 1612010 • YouTube: • Nodes: 1134890 | Edges: 2987624 | Number of triangles: 3056386 • Pokec: • Nodes: 1632803 | Edges: 30622564 | Number of triangles: 32557458 • LiveJournal: • Nodes: 3997962 | Edges: 34681189 | Number of triangles: 177820130
- 6. EXPERIMENTS (CONT’D) • Default settings • Dato: • GRAPHLAB_FILEIO_MAXIMUM_CACHE_CAPACITY = 4G • GraphX • Start with executor memory = 1G • Change into 2G later
- 7. RESULTS • Triangle Counting: both Dato and GraphX (if it finishes the job) returns the correct answer as listed on the SNAP website. • For Pokec and LiveJournal data, GraphX has trouble finishing the computation
- 8. TAKE-AWAY FOR GRAPHX • What I observed was that certain stages within the job kept failing • A stage in Spark will operate on one partition of the RDD at a time (and load the data in that partition into memory) • Potential Solution • Increasing the executor memory • Increase the number of partitions of the RDD so that each stage is processing smaller amount of data
- 9. RESULTS (CONT’D) • PageRank: The threshold for PageRank is set to 0.001
- 10. RESULTS (CONT’D) • Connected Components
- 11. CONCLUSIONS • Quick setups for both of the tools without fine-tune runtime parameters, but • Dato has clear advantages over GraphX in terms of execution time for processing large scale graph data • However, GraphX is free while Dato charges a service fee after the free trial. • The goal of the GraphX project is to unify graph-parallel and data- parallel computation in one system with a single composable API. • Further experiments can be done to compare the overall performance of a specific task that contains both graph algorithms and other data-parallel computation
- 13. REFERENCES • Dato: • https://dato.com/ • Spark GraphX: • https://spark.apache.org/docs/1.1.0/graphx-programming- guide.html • Stanford Large Network Dataset Collection (SNAP): • https://snap.stanford.edu/data/