Scalable machine learning
0 likes762 views
This document discusses scaling machine learning algorithms to large datasets. It introduces machine learning and linear regression, explaining how to represent data with features and labels. Linear regression finds the line of best fit by minimizing a squared loss function. Computing the closed-form solution is inefficient for large datasets due to expensive matrix operations. Gradient descent is introduced as an iterative alternative that is easily parallelized. It works by taking steps in the direction of steepest descent until converging to a solution.
Scalable machine learning
- 1. Scalable Machine Learning BerkeleyX: CS190.1x Scalable Machine Learning
- 2. outline 1. what is machine learning ? 2. linear regression 3. how to scale linear regression 4. gradient descent 5. how to scale gradient descent
- 5. Machine learning explores the study and construction of algorithms that can learn from and make predictions on data A Definition Face recognition Link prediction document classification
- 6. Raw data Raw data comes from many sources apache log email image
- 7. Raw data Feature Extraction Initial observations can be in arbitrary format We extract features to represent observations We can incorporate domain knowledge We typically want numeric features Success of entire pipeline often depends on choosing good descriptions of observations!!
- 8. Raw data Feature Extraction Supervised Learning Train a supervised model using labeled data, e.g., Classification or Regression model weather sunny rainy cloudy vehicle count 1000 99 5
- 9. Raw data Feature Extraction Supervised Learning Evaluation TP FP FN TN Has cancer No cancer Predict cancer Predict No cancer confusion matrix
- 10. Raw data Feature Extraction Learning Model Evaluation Predict
- 11. full dataset training set validation set testing set classifier turning model model builder final classifier predict
- 13. Example: Predicting shoe size from height, gender, and weight For each observation we have a feature vector, x, and label, y We assume a linear mapping between features and label:
- 14. Goal: find the line of best fit x coordinate: features y coordinate: labels 1 D Example
- 15. Linear Least Squares Regression Assume we have n training points, where x(i) denotes the ith point Recall two earlier points: • Linear assumption : y = wTx • squared loss : ( y - y )2
- 16. Linear Least Squares Regression
- 17. Computing Closed Form Solution Computational bottlenecks: • Matrix multiply of XT X : O(nd2) operations • Matrix inverse: O(d3)operations d nX: n dXT: O(nd2)O(dn2) ?
- 18. Matrix Multiplication via Inner Products Matrix Multiplication via Outer Products result
- 21. if Big n and Big d ?
- 24. Gradient Descent Start at a random point
- 25. Gradient Descent Start at a random point 1. Determine a descent direction 2. Choose a step size 3. Update
- 26. Gradient Descent Start at a random point 1. Determine a descent direction 2. Choose a step size 3. Update Repeat Until stopping criterion is satisfied
- 27. Choosing Descent Direction (1D) 1. Determine a descent direction
- 28. Choosing Step Size 2. Choose a step size
- 29. =
- 30. Parallel Gradient Descent for Least Squares
- 31. Gradient Descent Summary • Easily parallelized • Cheap at each iteration positive • Slow convergence • Requires communication across nodes! negative