We're so skewed_presentation
4 likes1,241 views
This document summarizes a Kaggle competition to predict housing prices in Ames, Iowa using machine learning techniques. It describes the data provided, which includes over 1400 observations and 79 variables for houses. It also details the various steps taken to process and analyze the data, including handling missing values, outliers, and categorical variables. Several machine learning algorithms were tested including random forest, gradient boosting, XGBoost, and linear regression. The best performing model was an ensemble approach with a RMSE of $9000 on average house prices. Key factors found to influence prices included size, age, quality, neighborhood, commercial zoning, and year of sale.
We're so skewed_presentation
- 2. Introduction Challenge: Predicting Housing Prices in Ames, Iowa using various machine learning techniques Data: ◦ Train Data Set: 1460 Observations x 80 Variables (Including Response Variable: Sale Price) ◦ Test Data Set: 1459 Observations x 79 Variables Useful Links: ◦ Kaggle Homepage: https://www.kaggle.com/c/house-prices-advanced-regression- techniques ◦ Data Description: https://storage.googleapis.com/kaggle-competitions- data/kaggle/5407/data_description.txt
- 3. Understanding the Data Total Predictor Variables Provided: 79 ◦ Continuous Variables: 28 ◦ Categorical Variables: 51 Combined test and train data sets to get a holistic view of each variable ◦ (i.e., total missing values, total categories in categorical variable)
- 7. Processing the Data: Handling Missing Data 1) Are data really missing? Ex. Pool Quality ◦ 2909 out of 2919 observations have ”NA” values ◦ Most NAs are due to houses not having pools ◦ Solution: ◦ Replace (most) NAs with new category: “None”
- 10. 3) Use domain knowledge Ex. Lot Frontage (486 NAs) ◦ Houses in close proximity likely have similar lot areas ◦ Solution: use categorical variable to impute numerical ◦ Use median Lot Frontage by neighborhood to impute missing value Processing the Data: Handling Missing Data
- 13. Processing the Data: Categorical Variables (Nominal) Some machine learning algorithms cannot handle non-numerical values Ex. Land Contour ◦ Solution: One-hot encoding technique: binarizing classes of each variable
- 16. Processing the Data: Near Zero Variance Predictors Low variance predictors add little value to models ◦ Calculate ratio of most frequent vs. second most frequent value ◦ Ratios >> 1 suggest very low variance ◦ Solution: Remove near zero predictors with cutoffs of 95:5
- 18. Feature Engineering Ideas for new features: ◦ Remodeled – Year Built not equal to Year Additional Remodeling ◦ Seasonality – Combine Month Sold and Year Sold ◦ New House – Year Built same as Year Sold ◦ Total Area – sum all variables denoting square footage ◦ Inside Area – sum all variables denoting square footage referring to space inside the house ◦ Overall Basement – Basement Quality and Basement Condition ◦ Overall Condition – Condition 1 and Condition 2 ◦ Overall Quality – External Quality and External Condition ◦ Overall Sale – Sale Type and Sale Condition ◦ Sale and Condtion – Sale Type and Overall Condition
- 19. Pros Cons Hyperparameters Cross-Validated RMSE Score Kaggle Score Random Forest Lower variance, Decorrelates data, Scale invariant High bias, Difficult to interpret Num features = 48, Num trees = 1000 0.14997 0.14758 Models
- 21. Pros Cons Hyperparameters Cross-Validated RMSE Score Kaggle Score Random Forest Lower variance, Decorrelates data, Scale invariant High bias, Difficult to interpret Num features = 48, Num trees = 1000 0.14997 0.14758 Gradient Boost Feature scaling not needed, High accuracy Computationally expensive, Overfitting Num trees = 1000, Depth = 2, Num Features = sqrt, Samples/leaf = 15, Learning rate = 0.05 0.1128 0.12421 Models
- 23. Pros Cons Hyperparameters Cross-Validated RMSE Score Kaggle Score Random Forest Lower variance, Decorrelates data, Scale invariant High bias, Difficult to interpret Num features = 48, Num trees = 1000 0.14997 0.14758 Gradient Boost Feature scaling not needed, High accuracy Computationally expensive, Overfitting Num trees = 1000, Depth = 2, Num Features = sqrt, Samples/leaf = 15, Learning rate = 0.05 0.1128 0.12421 XGBoost Extremely fast, Allows parallel computing Difficult to interpret, Overfits vs gradient boosting Num trees = 2724, Max depth = 30, Gamma = 0.0, Minimum child weight = 4 0.13642 0.13082 Models
- 25. Pros Cons Hyperparameters Cross-Validated RMSE Score Kaggle Score Random Forest Lower variance, Decorrelates data, Scale invariant High bias, Difficult to interpret Num features = 48, Num trees = 1000 0.14997 0.14758 Gradient Boost Feature scaling not needed, High accuracy Computationally expensive, Overfitting Num trees = 1000, Depth = 2, Num Features = sqrt, Samples/leaf = 15, Learning rate = 0.05 0.1128 0.12421 XGBoost Extremely fast, Allows parallel computing Difficult to interpret, Overfits vs gradient boosting Num trees = 2724, Max depth = 30, Gamma = 0.0, Minimum child weight = 4 0.13642 0.13082 Regularize Linear Regression Easily interpretable, Computationally inexpensive, Less prone to overfitting Requires scaled variables, Requires numerical variables Lambda = 0.0005, Alpha = 0.9 0.1111 0.11922 Models
- 27. Pros Cons Hyperparameters Cross-Validated RMSE Score Kaggle Score Random Forest Lower variance, Decorrelates data, Scale invariant High bias, Difficult to interpret Num features = 48, Num trees = 1000 0.14997 0.14758 Gradient Boost Feature scaling not needed, High accuracy Computationally expensive, Overfitting Num trees = 1000, Depth = 2, Num Features = sqrt, Samples/leaf = 15, Learning rate = 0.05 0.1128 0.12421 XGBoost Extremely fast, Allows parallel computing Difficult to interpret, Overfits vs gradient boosting Num trees = 2724, Max depth = 30, Gamma = 0.0, Minimum child weight = 4 0.13642 0.13082 Regularize Linear Regression Easily interpretable, Computationally inexpensive, Less prone to overfitting Requires scaled variables, Requires numerical variables Lambda = 0.0005, Alpha = 0.9 0.1111 0.11922 Ensembling Can improve accuracy Lose interpretability Lasso, Enet, Gradient Boost, Gradient Boost Lite 0.1071 0.11751 Models
- 29. Questions?