Machine learning using spark
Download as PPTX, PDF3 likes509 views
The document details a machine learning use case for predicting house prices in Tel Aviv using Apache Spark and its MLlib library. It outlines the technology stack, data preprocessing steps, various algorithms utilized (linear regression, decision trees, and random forests), and methods for evaluating model performance, such as root mean squared error (RMSE). The overall process includes exploring data, assessing algorithms, and optimizing parameters to improve predictions.
![One-Hot Encoding
RoofStyle: [“Gable”,”Hip”,”Flat”]
[{“Gable”,0},{“Hip”,1},{“Flat”,2}]
Line # 1: RoofStyle: “Gable” Vector: [1.0, 0, 0]
Line # 2: RoofStyle: “Hip” Vector: [0, 1.0, 0]
Line # 3: RoofStyle: “Flat” Vector: [0, 0, 1.0]](https://image.slidesharecdn.com/machinelearningusingspark-170407062040/85/Machine-learning-using-spark-15-320.jpg)
![Find Correlations
val seriesX: RDD[Double] = sc.parallelize(grdLivArea) // X-axis
val seriesY: RDD[Double] = sc.parallelize(SalePrice) // Y-axis
val correlation: Double = Statistics.corr(seriesX, seriesY)
Compute correlation
using scala Statistics lib](https://image.slidesharecdn.com/machinelearningusingspark-170407062040/85/Machine-learning-using-spark-21-320.jpg)
Machine learning using spark
- 1. Machine Learning & Spark Predict Prices of Houses
- 2. About me Ran Silberman Architect at Tikal Knowledge Big Data Consultant mailto:ran@tikalk.com
- 3. Predict house prices Use Case: Predict house prices in Tel Aviv based on parameters of houses Use data from Kaggle
- 4. Hypotheses: 1. ML: not only for mathematicians! 2. ML + Big-Data = Spark!
- 5. Technology Stack Spark Machine Learning library Scala
- 6. Apache Spark - RDD Block-1 Block-2 Block-3 Block-4 HDFS Input File Partition-1 Partition-2 Partition-3 Partition-4 RDD Partition-1 Partition-2 Partition-3 Partition-4 RDD Load Map Block-1 Block-2 HDFS Output Block-3 Block-4 Write
- 7. Spark MlLib Block-1 Block-2 Block-3 Training Data Partition-1 Partition-2 Partition-3 RDD Algorithm Function ML Model Load Build Model Block-1 Block-2 Block-3 Test Data Predict Partition-1 Partition-2 Partition-3 RDD Load
- 8. Spark MLlib API RDD-based API DataFrame-based API Spark 2.0 Package: spark.mllib Package: spark.ml
- 9. Approach 1. Explore the Data 2. Assess Algorithms 3. Put it all together
- 10. Explore the Data
- 11. Cond Bsmt Cond Year built Bsmt area Roof Style Grnd Liv Area Grg cars Grg area Year sold Sale Price 7 Good 2003 856 Gable 1710 2 548 2008 208500 6 Good 1976 1262 Gable 1262 2 460 2007 181500 7 Exc 2001 920 Hip 1786 2 608 2008 223500 7 Fair 1915 756 Hip 1717 3 642 2006 140000 8 Typical 2000 1145 Gable 2198 3 836 2008 250000 8 No Bsmt 2004 1686 Flat 1694 (null) 636 2007 307000
- 12. Data Format 208500 1:7.00 2:5.00 3:2003.00 4:856.00 5:1710.00 6:2.00 7:548.00 8:856.00 9:2.00 10:8.00 181500 1:6.00 2:8.00 3:1976.00 4:1262.00 5:1262.00 6:2.00 7:460.00 8:1262.00 9:2.00 10:6.00 223500 1:7.00 2:5.00 3:2001.00 4:920.00 5:1786.00 6:2.00 7:608.00 8:920.00 9:2.00 10:6.00 “Label” value 1st feature 2nd feature 3rd feature 10th feature
- 14. Roof Types
- 15. One-Hot Encoding RoofStyle: [“Gable”,”Hip”,”Flat”] [{“Gable”,0},{“Hip”,1},{“Flat”,2}] Line # 1: RoofStyle: “Gable” Vector: [1.0, 0, 0] Line # 2: RoofStyle: “Hip” Vector: [0, 1.0, 0] Line # 3: RoofStyle: “Flat” Vector: [0, 0, 1.0]
- 16. Pipelines in Spark MLlib val indexer = new StringIndexer() .setInputCol("roofType").setOutputCol("roofIndex") val encoder = new OneHotEncoder() .setInputCol("roofIndex").setOutputCol("features") val lr = new LinearRegression() val pipeline = new Pipeline().setStages(Array(indexer, encoder, lr)) val model = pipeline.fit(dataFrame) Transform Strings to numeric index encode index values to one-hot vector Linear Regression algorithm Pipeline put it all togetherTrain Model
- 18. Linear Regression using Spark-MLlib val spark = SparkSession.builder().getOrCreate() val training = spark.read.format("libsvm").load(trainingFile) val lr = new LinearRegression() val model = lr.fit(training) model.transform(test).show println(s" ${model.summary.rootMeanSquaredError}") Linear Regression Class Train model Predict using model Load data file Print RMSE Set Spark Session
- 19. Cost Function - Root Mean squared Error Way to define how well our model can predict
- 20. Root Mean Square Error Linear Regression - 10 Features 49972
- 21. Find Correlations val seriesX: RDD[Double] = sc.parallelize(grdLivArea) // X-axis val seriesY: RDD[Double] = sc.parallelize(SalePrice) // Y-axis val correlation: Double = Statistics.corr(seriesX, seriesY) Compute correlation using scala Statistics lib
- 22. 2. Find Best Correlated Features
- 23. Correlation Matrix - Best features to use
- 24. Bivariate Analysis of several Features
- 25. Linear Regression - Improve Training Data val spark = SparkSession.builder().getOrCreate() val training = spark.read.format("libsvm").load(betterFeaturesFile) val lr = new LinearRegression() val model = lr.fit(training) model.transform(test).show println(s" ${model.summary.rootMeanSquaredError}") Features set with best correlation
- 26. Root Mean Square Error Linear Regression - 10 Arbitrary Features 49972 Linear Regression - 10 Best Features 35654
- 27. Linear Regression (Bivariate Analysis)
- 28. Gradient Descent of Cost Function on one feature = RMSE
- 29. Linear Regression - Set more iterations val lr = new LinearRegression() val model = lr.fit(training).setMaxIter(50) Set Higher number of iterations (was 10) numIterations: 33 RMSE: 35583 Output Window
- 30. Root Mean Square Error Linear Regression - 10 Arbitrary Features 49972 Linear Regression - 10 Best Features 35654 Linear Regression - Higher num of iterations 35583
- 31. Decision Tree Regressor val data = spark.read.format("libsvm").load(logFile) val Array(train, test) = data.randomSplit(Array(0.7, 0.3)) val dt = new DecisionTreeRegressor() .setMaxDepth(3) val model = dt.fit(train) val predictions = model.transform(test) val rmse = new RegressionEvaluator().evaluate(predictions) println(s"RMSE = $rmse") Decision Tree Regressor Class Train model Predict using model Load data file Print RMSE Split data to training and test
- 32. Decision Tree a > 7 a > 6 b>1389 $126,901 a > 8 b>1964 b>1964 c>1996 $162,206 $194,954 $257,919 $252,771 $320,193 $551,666$368,137 a: Overall Quality (1-10) b: Grd. Living Area (ft.) c: Year built yes yes yesyes yes yesyes no no no no no no no level-1 level-2 level-3
- 33. Root Mean Square Error Linear Regression - 10 Arbitrary Features 49972 Linear Regression - 10 Best Features 35654 Linear Regression - Higher num of iterations 35583 Decision Tree 46643
- 34. Decision Tree - set Tree Depth val dt = new DecisionTreeRegressor() .setMaxDepth(3) RMSE = 46643 val dt = new DecisionTreeRegressor() .setMaxDepth(20) RMSE = 40571 val dt = new DecisionTreeRegressor() .setMaxDepth(8) RMSE = 37198
- 36. Root Mean Square Error Linear Regression - 10 Arbitrary Features 49972 Linear Regression - 10 Best Features 35654 Linear Regression - Higher num of iterations 35583 Decision Tree (depth = 3) 46643 Decision Tree (depth = 20) 40571 Decision Tree (depth = 8) 37198
- 37. Random Forest Regressor val data = spark.read.format("libsvm").load(logFile) val Array(train, test) = data.randomSplit(Array(0.7, 0.3)) val dt = new RandomForestRegressor() .setMaxDepth(8).setNumTrees(20) val model = dt.fit(train) val predictions = model.transform(test) val rmse = new RegressionEvaluator().evaluate(predictions) println(s"RMSE = $rmse") Random Forest Regressor Class Set num of trees
- 38. Random Forest
- 39. Root Mean Square Error Linear Regression - 10 Arbitrary Features 49972 Linear Regression - 10 Best Features 35654 Linear Regression - Higher num of iterations 35583 Decision Tree (depth = 3) 46643 Decision Tree (depth = 20) 40571 Decision Tree (depth = 8) 37198 Random Forest (depth = 8, #Trees = 10) 35023
- 40. Random Forest - set Number of Trees val dt = new RandomForestRegressor() .setMaxDepth(8).setNumTrees(10) RMSE = 35023 val dt = new RandomForestRegressor() .setMaxDepth(20).setNumTrees(100) RMSE = 34607
- 41. Root Mean Square Error Linear Regression - 10 Arbitrary Features 49972 Linear Regression - 10 Best Features 35654 Linear Regression - Higher num of iterations 35583 Decision Tree (depth = 3) 46643 Decision Tree (depth = 20) 40571 Decision Tree (depth = 8) 37198 Random Forest (depth = 8, #Trees = 10) 35023 Random Forest (depth = 20, #Trees = 100) 34607
- 42. Summary 1. Explored the Data - data transformations, correlations 2. Assessed Algorithms: Regression, Tree, Forest 3. Play with parameters: # iterations, tree depth, # of trees
- 43. Thank You!
Editor's Notes
- #17: Look how this code is simple…Can deal with very big data But need to make all features have float values...
- #19: Look how this code is simple…Can deal with very big data But need to make all features have float values...
- #22: Look how this code is simple…Can deal with very big data But things are not that simple...
- #26: Look how this code is simple…Can deal with very big data But things are not that simple...
- #30: Look how this code is simple…Can deal with very big data But things are not that simple...
- #33: Some problems with trees: If the tree has few levels of depths then all predicted values are among a predefined list If the tree has too many levels of depths, we get an overfitting problem
- #35: Look how this code is simple…Can deal with very big data But things are not that simple...
- #41: Look how this code is simple…Can deal with very big data But things are not that simple...