Advanced regression and model selection
1 like1,547 views
The document discusses various machine learning algorithms and model selection techniques, highlighting the advantages and disadvantages of each method, including Naive Bayes, logistic regression, decision trees, and SVM. It also covers advanced regression issues and regularization techniques, detailing ridge and lasso regression's pros and cons along with their applications in gene selection problems. Lastly, it introduces additional methods such as Poisson regression and piecewise linear regression for specific types of data.
Advanced regression and model selection
- 1. Advanced Regression and Model Selection UpGrad Live Session - Ankit Jain
- 2. Model Selection Techniques ā If you are looking for a good place to start to choose a machine learning algorithm for your dataset here are some general guidelines. ā How large is your training set? ā Small -- Prefer high bias/low variance classifiers (e.g. Naive Bayes) over low bias/high variance classifiers (e.g. KNN) to avoid overfitting. ā Large - Low Bias/High Variance classifiers tend to produce more accurate models
- 3. Adv/Disadv of Various Algorithms ā Naive Bayes: ā Very simple to implement as itās just a bunch of counts. ā If conditional independence exists, it converges faster than say Logistic Regression and thus requires less training data. ā If you want something fast,easy and performs well NB is a good choice ā Biggest disadvantage is that it canāt learn interactions in the dataset
- 4. Adv/Disadv of Various Algorithms ā Logistic Regression: ā Lots of ways to regularize the model and no need to worry about features being correlated like in Naive Bayes. ā Nice probabilistic interpretation. Helpful in problems like churn prediction etc . ā Online algorithm: Easy to update the model with the new data (using an online gradient descent method)
- 5. Adv/Disadv of Various Algorithms ā Decision Trees: ā Easy to explain and interpret (at least for some people) ā Easily handles feature interactions. ā No need to worry about outliers or whether data is linearly separable or not. ā Doesnāt support online learning. Rebuilding the model with new data every time can be painful. ā Tend to easily overfit. Solution: ensemble methods (RF)
- 6. Adv/Disadv of Various Algorithms ā SVM: ā High accuracy for many datasets ā With appropriate kernel, can work well even if your data isnāt linearly separable in the base feature space. ā Popular in text processing applications where high dimensionality is a norm ā Memory intensive, hard to interpret and kind of annoying to run and tune
- 8. Linear Regression Issues ā Sensitivity to outliers ā Multicollinearity leads to high variance of the estimator. ā Prone to overfit if there are lot of variables ā Hard to interpret when the number of predictors is large.Need a smaller subset that exhibits strongest effects.
- 9. Regularization Techniques ā Regularization techniques typically work by penalizing the magnitude of coefficients of features along with minimizing the error between predicted and actual observations ā Different types of penalization ā Ridge Regression: Penalize on squared coefficients ā Lasso Regression: Penalize on absolute value of coefficients
- 10. Why penalize on model coefficients? Model1 = beta0 + beta1*x Model2 = beta0 + beta1*x + ā¦ beta10*x^10 beta1 = -0.58 beta1 = -1.4e05
- 11. Ridge Regression ā L2 penalty ā Pros ā Variables >> Rows ā Multicollinearity ā Increased bias and lower variance from Linear Regression ā Cons ā Doesnāt produce parsimonious model Letās see a collinearity example in R
- 12. Example: Luekemia Prediction ā Leukemia Data, Golub et al. Science 1999 ā There are 38 training samples and 34 test samples with total genes ~ 7000 (p >> n) ā Xij is the gene expression value for sample i and gene j ā Sample i either has tumor type AML or ALL ā We want to select genes relevant to tumor type ā eliminate the trivial genes ā grouped selection as many genes are highly correlated ā Ridge Regression can help to pursue this modeling
- 13. Grouped Selection ā If two predictors are highly correlated among themselves, the estimated coefficients will be similar for them. ā if some variables are exactly identical, they will have same coefficients Ridge is good for grouped selection but not good for eliminating trivial genes
- 14. LASSO ā Pros ā Allow p >> n ā Enforce sparsity in parameters ā Cons ā If a group of predictors are highly correlated among themselves, LASSO tends to pick only one of them and shrink the other to zero ā can not do grouped selection, tend to select one variable LASSO is good for eliminating trivial genes but not good for grouped selection
- 15. Elastic Net ā Weighted combination of L1 and L2 penalty ā Helps in enforcing sparsity ā Encourage grouping effect in highly correlated predictors In gene selection problem, it can achieve both purposes of removing trivial genes and doing group selection
- 16. Other Advanced Regression Methods Poisson Regression ā Typically used when the Y variable follows poisson distribution (typically counts of events within a time t) ā # times a customer will visit an ecommerce website next month
- 17. Piecewise Linear Regression ā Polynomial regression wonāt work perfectly as it will have high tendency to overfit/underfit ā Instead, splitting the curve into separate linear pieces and building linear model for each piece leads to better results
- 18. QUESTIONS