Speeding up Distributed Big Data Recommendation in Spark
2 likes892 views
The document discusses advancements in distributed computing frameworks for big data analytics, particularly focusing on the Spark platform. It introduces a new algorithm that accelerates the Alternating Least Squares (ALS) method for collaborative filtering in recommendation systems, demonstrating that it can significantly outperform traditional ALS implementations. The performance improvements are achieved through an efficient parallel implementation that harnesses Spark's capabilities for scalable and fault-tolerant processing.
Speeding up Distributed Big Data Recommendation in Spark
- 1. AUSTRALIA CHINA INDIA ITALY MALAYSIA SOUTH AFRICA monash.edu Algorithmic Acceleration of Parallel ALS for Collaborative Filtering: “Speeding up Distributed Big Data Recommendation in Spark” Hans De Sterck1,2, Manda Winlaw2, Mike Hynes2, Anthony Caterini2 1 Monash University, School of Mathematical Sciences 2 University of Waterloo, Canada, Applied Mathematics ICPADS 2015, Melbourne, December 2015
- 2. hans.desterck@monash.edu ICPADS 2015 a talk on algorithms for parallel big data analytics ... 1. distributed computing frameworks for Big Data analytics – Spark (vs HPC, MPI, Hadoop, ...) 2. recommendation – the Netflix prize problem 3. our contribution: an algorithm to speed up ALS for recommendation 4. our contribution: efficient parallel speedup of ALS recommendation in Spark
- 3. hans.desterck@monash.edu ICPADS 2015 1. distributed computing frameworks for Big Data analytics – Spark (vs HPC, MPI, Hadoop, ...) ■ my research background: – scalable scien>fic compu>ng algorithms (HPC) – e.g., parallel algebraic mul>grid (AMG) for solving linear systems Ax=b – e.g., on Blue Gene (100,000s of cores), MPI
- 4. hans.desterck@monash.edu ICPADS 2015 distributed computing frameworks for Big Data analytics – Spark (vs HPC, MPI, Hadoop, ...) ■ more recently: there is a new game of large-scale distributed computing in town! – Google PageRank (1998) (already 17 years...) • commodity hardware (fault-‐tolerant ...) • compute where the data is (data-‐locality) • scalability is essen>al! (just like in HPC) • beginning of “Big Data”, “Cloud”, “Data Analy>cs”, ... – new Big Data analy>cs applica>ons are now appearing everywhere! web crawl
- 5. hans.desterck@monash.edu ICPADS 2015 distributed computing frameworks for Big Data analytics – Spark (vs HPC, MPI, Hadoop, ...) ■ “Data Analytics” has grown its own “eco-system”, “culture”, “software stack” (very different from HPC!) • MapReduce • Hadoop • Spark, ... • data locality • “implicit” communica>on (restricted (vs MPI), “shuffle”) • not fast (vs HPC), but scalable • fault-‐tolerant (replicate data, restart tasks) (from “Spark: In-‐Memory Cluster Compu>ng for Itera>ve and Interac>ve Applica>ons”)
- 6. hans.desterck@monash.edu ICPADS 2015 distributed computing frameworks for Big Data analytics – Spark (vs HPC, MPI, Hadoop, ...) ■ MapReduce/Hadoop: – major disadvantage for itera>ve algorithms: writes everything to disk between itera>ons!, extremely slow (and: not programmer-‐friendly) è only very simple algorithms are feasible in MapReduce ■ the Spark “revolution”: – store state between itera>ons in memory – more general opera>ons than Hadoop/MapReduce
- 7. hans.desterck@monash.edu ICPADS 2015 distributed computing frameworks for Big Data analytics – Spark (vs HPC, MPI, Hadoop, ...)
- 8. hans.desterck@monash.edu ICPADS 2015 distributed computing frameworks for Big Data analytics – Spark (vs HPC, MPI, Hadoop, ...) ■ the Spark “revolution”: – store state between itera>ons in memory – more general opera>ons than Hadoop/MapReduce èmuch faster than Hadoop! (but s>ll much slower than MPI) • data locality • scalable • fault-‐tolerant • “implicit” communica>on (restricted (vs MPI), “shuffle”) sea change (vs Hadoop): more advanced iterative algorithms for Data Analytics/Machine Learning are feasible in Spark
- 9. hans.desterck@monash.edu ICPADS 2015 k 2. recommendation – the Netflix prize problem ■ sparse ratings matrix R ■ k latent features: user factors U, movie factors M ■ similar to SVD, but only match known ratings ■ minimize f=||R – UTM||2’ , and UTM gives predicted ratings (collaborative filtering) R n users m movies 1 2 5 i j ≈ UT i j x x x x x x M k
- 10. hans.desterck@monash.edu ICPADS 2015 k recommendation – the Netflix prize problem minimize f=||R – UTM||2’ : alternating least squares (ALS) ■ minimize ||R – U(0)T M(0)||2’ : freeze U(0), compute M(0) (LS) ■ minimize ||R – U(1)TM(0)||2’ : freeze M(0), compute U(1) (LS) ■ ... : local least squares problems (parallelizable) R n users m movies 1 2 5 i j ≈ UT i j x x x x x x M k
- 11. hans.desterck@monash.edu ICPADS 2015 recommendation – the Netflix prize problem minimize f=||R – UTM||2’ : alternating least squares (ALS) ■ ALS can converge very slowly (block nonlinear Gauss-Seidel) (g = grad f = 0)
- 12. hans.desterck@monash.edu ICPADS 2015 3. our contribution: an algorithm to speed up ALS for recommendation min f(U,M)=||R – UTM||2’ , or g(U,M) = grad f(U,M) = 0 ■ nonlinear conjugate gradient (NCG) optimization algorithm for min f(x):
- 13. hans.desterck@monash.edu ICPADS 2015 our contribution: an algorithm to speed up ALS for recommendation min f(x)=||R – UTM||2’ , or g(x) = grad f(x) = 0 ■ our idea: use ALS as a nonlinear preconditioner for NCG define a preconditioned gradient direction: (De Sterck and Winlaw, 2015)
- 14. hans.desterck@monash.edu ICPADS 2015 our contribution: an algorithm to speed up ALS for recommendation min f(x)=||R – UTM||2’ , or g(x) = grad f(x) = 0 ■ our idea: use ALS as a nonlinear preconditioner for NCG (NCG accelerates ALS)
- 15. hans.desterck@monash.edu ICPADS 2015 our contribution: an algorithm to speed up ALS for recommendation min f(x)=||R – UTM||2’ , or g(x) = grad f(x) = 0 ■ our idea: use ALS as a nonlinear preconditioner for NCG
- 16. hans.desterck@monash.edu ICPADS 2015 our contribution: an algorithm to speed up ALS for recommendation min f(x)=||R – UTM||2’ , or g(x) = grad f(x) = 0 ■ our idea: use ALS as a nonlinear preconditioner for NCG ALS-‐NCG is much faster than the widely used ALS!
- 17. hans.desterck@monash.edu ICPADS 2015 4. our contribution: efficient parallel speedup of ALS recommendation in Spark ■ Spark “Resilient Distributed Datasets” (RDDs) – par>>oned collec>on of (key, value) pairs – can be cached in memory – built using data flow operators on other RDDs (map, join, group-‐by-‐key, reduce-‐by-‐key, ...) – fault-‐tolerance: rebuild from lineage – “implicit” communica>on (shuffling) (≠ MPI) key (value1, value2, ...) 0 1 2 3
- 18. hans.desterck@monash.edu ICPADS 2015 our contribution: efficient parallel speedup of ALS recommendation in Spark ■ efficient Spark programming: similar challenges as efficient GPU programming with CUDA! – of course, they have different design objec>ves (GPU: close to metal, as fast as possible; Spark: scalable, fault-‐tolerant, data locality...) – but ... similari>es in how one gets good performance: • Spark, CUDA: it is easy to write code that produces the correct result (but may be very far from achievable speed) • Spark, CUDA: it is very hard to write efficient code! – implementa>on choices that are crucial for performance are most ofen not explicit in the language – programmer needs very extensive “under the hood” knowledge to write efficient code – this is a research topic (also for Spark), moving target
- 19. hans.desterck@monash.edu ICPADS 2015 our contribution: efficient parallel speedup of ALS recommendation in Spark ■ existing implementation of ALS in Spark (Chris Johnson, Spotify) minimize f=||R – UTM||2’ – store both R and RT – local LS problems: to update user factor i, need all movie factors j that i has rated (shuffle!) (efficient) R 0 1 2 3 0 1 2 3 RT 0 1 2 3 M U 1 0 2 3 i j1 j2
- 20. hans.desterck@monash.edu ICPADS 2015 our contribution: efficient parallel speedup of ALS recommendation in Spark ■ our work: efficient parallel implementation of ALS-NCG in Spark minimize f(x)=||R – UTM||2’ – store our vectors x and g consistent with ALS RDDs, and employ similar efficient shuffling scheme for gradient – BLAS vector opera>ons – line search: f(x) is a polynomial of degree 4: compute coefficients once in parallel U
- 21. hans.desterck@monash.edu ICPADS 2015 our contribution: efficient parallel speedup of ALS recommendation in Spark ■ performance: linear granularity scaling for ALS-NCG as for ALS (no new parallel boFlenecks for the more advanced algorithm)
- 22. hans.desterck@monash.edu ICPADS 2015 our contribution: efficient parallel speedup of ALS recommendation in Spark ■ performance: ALS-NCG much faster than ALS (20M MovieLens data, 8 nodes/128 cores)
- 23. hans.desterck@monash.edu ICPADS 2015 our contribution: efficient parallel speedup of ALS recommendation in Spark ■ performance: ALS-NCG speeds up ALS on 16 nodes/256 cores in Spark for 800M ratings by a factor of about 5 (great speedup, in parallel, in Spark, for large problem on 256 cores)
- 24. hans.desterck@monash.edu ICPADS 2015 some general conclusions ... ■ Spark enables advanced algorithms for Big Data analytics (linear algebra, optimization, machine learning, ...) (lots of work: investigate algorithms, implementations, scalability, ... in Spark) ■ Spark offers a suitable environment for compute-intensive work! ■ slower than MPI/HPC, but data locality, fault-tolerance, situated within Big Data “eco-system” (HDFS data, familiar software stack, ...) ■ will HPC and Big Data hardware/software converge? (also for “exascale” ...), and if so, which aspects of the Spark (and others ...) or MPI/HPC approaches will prevail?