Apache SystemML - Declarative Large-Scale Machine Learning
1 like308 views
Apache SystemML is a declarative machine learning library that: 1) Allows users to write custom machine learning algorithms declaratively without needing to handle performance or distribution details. 2) Generates and optimizes low-level execution plans to transparently distribute and run algorithms at scale on data-parallel frameworks like Spark. 3) Originally started as projects at IBM Research to handle machine learning on Hadoop and address how data scientists create ML solutions, and has since become an Apache open-source project.
Apache SystemML - Declarative Large-Scale Machine Learning
- 1. Apache SystemML - Declarative Large-Scale Machine Learning Romeo Kienzler (IBM Waston IoT) Berthold Reinwald (IBM Almaden Research Center) Frederick R. Reiss (IBM Almaden Research Center) Matthias Rieke (IBM Analytics) Swiss Data Science Conference 16 - ZHAW - Winterthur
- 2. –Assembler vs. Python? “High-level programming”
- 3. Why another lib? • Custom machine learning algorithms • Declarative ML • Transparent distribution on data-parallel framework • Scale-up • Scale-out • Cost-based optimiser generates low level execution plans
- 4. Why on Spark? • Unification of SQL, Graph, Stream, ML • Common RDD structure • General DAG execution engine • lazy evaluation • distributed in-memory caching
- 5. 200920082007 2007-2008: Multiple projects at IBM Research – Almaden involving machine learning on Hadoop. 2010 2009-2010: Through engagements with customers, we observe how data scientists create ML solutions. 2009: We form a dedicated team for scalable ML
- 7. 20162015 June 2015: IBM Announces open- source SystemML September 2015: Code available on Github November 2015: SystemML enters Apache incubation June 2016: Second Apache release (0.10) February 2016: First release (0.9) of Apache SystemML
- 8. SystemML at Moved from Hadoop MapReduce to Spark SystemML supports both frameworks Exact same code 300X faster on 1/40th as many nodes
- 10. Products Customers i j Customer i bought product j. Alternating Least Squares
- 11. Products Customers i j Customer i bought product j. Alternating Least Squares
- 12. Products Customers i j Customer i bought product j. Alternating Least Squares
- 13. Products Customers i j Customer i bought product j. Products Factor CustomersFactor
- 14. Products Customers i j Customer i bought product j. Products Factor CustomersFactor
- 15. Products Customers i j Customer i bought product j. Products Factor CustomersFactor
- 16. Products Customers i j Customer i bought product j. Products Factor CustomersFactor Multiply these two factors to produce a less- sparse matrix. ×
- 17. Products Customers i j Customer i bought product j. Products Factor CustomersFactor Multiply these two factors to produce a less- sparse matrix. × New nonzero values become product suggestions.
- 24. Every line has a clear purpose!
- 26. 25 lines’ worth of algorithm… …mixed with 800 lines of performance code https://github.com/apache/spark/blob/master/ mllib/src/main/scala/org/apache/spark/mllib/ recommendation/ALS.scala
- 31. Architecture High-Level Operations (HOPs) General representation of statements in the data analysis language Low-Level Operations (LOPs) General representation of operations in the runtime framework High-level language front-ends Multiple execution environments Cost Based Optimizer
- 33. W S U U × S *( ( t(U) t(U)×(W*(U×S)))( × Large dense intermediate Can compute directly from U, S, and W! t(U)(%*%((W(*((U(%*%(S))( wdivmm WU S 1.2GB sparse 800MB dense 800MB dense 800MB dense (weighted divide matrix multiplication)
- 34. %*% WU S *t() %*% 1.2GB sparse 80GB dense 80GB dense 800MB dense 800MB dense 800MB dense 800MB dense All operands fit into heap ! use one node WDivMM (MapWDivMM)
- 38. Browse the source! Try out some tutorials! Contribute to the project!
- 39. Browse the source! Try out some tutorials! Contribute to the project!Download the binary release!
- 40. Demo