Introduction to Sparkling Water - Spark Summit East 2016
1 like955 views
The document introduces Sparkling Water, a tool that integrates the H2O.ai platform with Apache Spark to leverage advanced machine learning algorithms in Spark workflows. It outlines the background of Michal Malohlava and the capabilities of H2O's open-source data science platform, including various machine learning algorithms and data processing functionalities. Key features of Sparkling Water include seamless data parsing, model building, and support for real-time predictions and data munging.
Introduction to Sparkling Water - Spark Summit East 2016
- 1. An introduction to Sparkling Water Michal Malohlava h2o.ai
- 2. Who Am I? Background • PhD in CS from Charles University in Prague, Czech Republic • Postdoc at Purdue University experimenting with algos for large-scale computation • Now software engineer at H2O.ai Experience with domain-specific languages, distributed system, software engineering, and big data.
- 3. H2O.ai H2Oteam Sri Ambati Cliff Click Co-Founders Stephen Boyd Rob Tibshirani Trevor Hastie Scientific Advisory Council
- 4. H2O Open-Source In-Memory Data Science Platform • Highly optimized Java code (in-house) • Distributed in-memory K-V store and map/ reduce computation framework • Data parser (HDFS, S3, NFS, HTTP, local drives, etc.) • Read/write access to distributed data frames (R/Pandas-style) • ML algos - Deep Learning, GBM, DRF, GLM, GLRM, K-Means, PCA, CoxPH, Ensembles • REST API: clients Interactive UI/R/Python Sparkling Water
- 5. Sparkling Water Provides • Transparent integration of H2O into Spark ecosystem • Use H2O Frames and algorithms with Spark API Excels in existing Spark workflows requiring advanced Machine Learning algorithms
- 7. Where to use Sparkling Water? Data Source Model building Modelling Deep Learning, GBM DRF, GLM, GLRM K-Means, PCA CoxPH, Ensembles Prediction processingData munging
- 8. Where to use Sparkling Water? Data Source Dataparsing munging Modelling Data load/munging/ exploration Load and parse data directly into H2OFrame Ad hoc data transformation
- 9. Where to use Sparkling Water? Data Source Off-line model training Stream processing Data Stream Data munging Model prediction Deploy the model Export model in a binary format or as code Modelling
- 10. WHAT IS INSIDE?
- 11. Cluster manager Worker node Spark executor Scala/Py main program Driver node H2OContext SparkContext Worker node Spark executor Worker node Spark executor
- 13. TIME FOR DEMO!
- 14. Key Points to Remember Sparkling Water integrates H2O to Spark • Enables using advanced machine learning algorithms inside Spark workflows • Offers eager computation model, mutable data structure H2OFrame
- 15. THANK YOU. @h2oai @mmalohlava h2o.ai/download github.com/h2oai/sparkling-water Visit our booth K27 for live demos and more!