Podling Hivemall in the Apache Incubator
4 likes955 views
Hivemall is a scalable machine learning library built as a collection of Hive UDFs. It was accepted into the Apache Incubator in September 2016. Hivemall can be used across multiple platforms like Hive, Spark, Pig, and is designed to be easy to use, versatile, and scalable for large datasets. It allows SQL developers to perform machine learning tasks in a parallel and scalable way on Hadoop clusters.
Podling Hivemall in the Apache Incubator
- 3. • Makoto Yui <Treasure Data> • Takeshi Yamamuro <NTT> Ø Hivemall on Apache Spark • Daniel Dai <Hortonworks> Ø Hivemall on Apache Pig Ø Apache Pig PMC member • Tsuyoshi Ozawa <NTT> ØApache Hadoop PMC member • Kai Sasaki <Treasure Data> 3 Initial committers 2016/11/08 Apache Hadoop Meetup at CWT 2016
- 4. Champion Nominated Mentors 4 Project mentors • Reynold Xin <Databricks, ASF member> Apache Spark PMC member • Markus Weimer <Microsoft, ASF member> Apache REEF PMC member • Xiangrui Meng <Databricks, ASF member> Apache Spark PMC member • Roman Shaposhnik <Pivotal, ASF member> Apache Bigtop/Incubator PMC member 2016/11/08 Apache Hadoop Meetup at CWT 2016
- 6. Hivemall is easy and scalable … Classification with Mahout CREATE TABLE lr_model AS SELECT feature, -- reducers perform model averaging in parallel avg(weight) as weight FROM ( SELECT logress(features,label,..) as (feature,weight) FROM train ) t -- map-only task GROUP BY feature; -- shuffled to reducers ML made easy for SQL developers Born to be parallel and scalable This SQL query automatically runs in parallel on Hadoop cluster 62016/11/08 Apache Hadoop Meetup at CWT 2016 Ease-of-use Scalable
- 7. 2016/11/08 Apache Hadoop Meetup at CWT 2016 7 Hivemall is a multi/cross-platform ML library HiveQL SparkSQL/Dataframe API Pig Latin Hivemall is Multi/Cross platform .. Multi/Cross platform prediction models built by Hive can be used from Spark, and conversely, prediction models build by Spark can be used from Hive
- 12. 2016/11/08 Apache Hadoop Meetup at CWT 2016 12 Versatile Hivemall is a Versatile library .. ü Hivemall is not only for Machine Learning ü Hivemall provides bunch of generic utility functions (e.g., top-k, NLP) Each organization has own sets of UDFs for data preprocessing! Don’t Repeat Yourself! Don’t Repeat Yourself!