ML_basics_lecture1_linear_regression.pdf
0 likes9 views
This document provides an overview of machine learning basics and linear regression. It defines machine learning as a program that improves its performance on tasks through experience. Linear regression aims to fit a linear model to training data by minimizing the empirical loss between predicted and true target values. It works by finding the weights that minimize the mean squared error loss on the training data according to the normal equation. The bias term can be incorporated by augmenting features with 1s.
![Math formulation
• Given training data 𝑥𝑖, 𝑦𝑖 : 1 ≤ 𝑖 ≤ 𝑛 i.i.d. from distribution 𝐷
• Find 𝑦 = 𝑓(𝑥) ∈ 𝓗 using training data
• s.t. the expected loss is small
𝐿 𝑓 = 𝔼 𝑥,𝑦 ~𝐷[𝑙(𝑓, 𝑥, 𝑦)] Various loss functions](https://image.slidesharecdn.com/mlbasicslecture1linearregression-240414082825-e094c87c/85/ML_basics_lecture1_linear_regression-pdf-17-320.jpg)
![Math formulation
• Given training data 𝑥𝑖, 𝑦𝑖 : 1 ≤ 𝑖 ≤ 𝑛 i.i.d. from distribution 𝐷
• Find 𝑦 = 𝑓(𝑥) ∈ 𝓗 using training data
• s.t. the expected loss is small
𝐿 𝑓 = 𝔼 𝑥,𝑦 ~𝐷[𝑙(𝑓, 𝑥, 𝑦)]
• Examples of loss functions:
• 0-1 loss: 𝑙 𝑓, 𝑥, 𝑦 = 𝕀[𝑓 𝑥 ≠ 𝑦] and 𝐿 𝑓 = Pr[𝑓 𝑥 ≠ 𝑦]
• 𝑙2 loss: 𝑙 𝑓, 𝑥, 𝑦 = [𝑓 𝑥 − 𝑦]2 and 𝐿 𝑓 = 𝔼[𝑓 𝑥 − 𝑦]2](https://image.slidesharecdn.com/mlbasicslecture1linearregression-240414082825-e094c87c/85/ML_basics_lecture1_linear_regression-pdf-18-320.jpg)
![Math formulation
• Given training data 𝑥𝑖, 𝑦𝑖 : 1 ≤ 𝑖 ≤ 𝑛 i.i.d. from distribution 𝐷
• Find 𝑦 = 𝑓(𝑥) ∈ 𝓗 using training data
• s.t. the expected loss is small
𝐿 𝑓 = 𝔼 𝑥,𝑦 ~𝐷[𝑙(𝑓, 𝑥, 𝑦)] How to use?](https://image.slidesharecdn.com/mlbasicslecture1linearregression-240414082825-e094c87c/85/ML_basics_lecture1_linear_regression-pdf-19-320.jpg)
![Math formulation
• Given training data 𝑥𝑖, 𝑦𝑖 : 1 ≤ 𝑖 ≤ 𝑛 i.i.d. from distribution 𝐷
• Find 𝑦 = 𝑓(𝑥) ∈ 𝓗 that minimizes
𝐿 𝑓 =
1
𝑛
σ𝑖=1
𝑛
𝑙(𝑓, 𝑥𝑖, 𝑦𝑖)
• s.t. the expected loss is small
𝐿 𝑓 = 𝔼 𝑥,𝑦 ~𝐷[𝑙(𝑓, 𝑥, 𝑦)]
Empirical loss](https://image.slidesharecdn.com/mlbasicslecture1linearregression-240414082825-e094c87c/85/ML_basics_lecture1_linear_regression-pdf-20-320.jpg)
![Linear regression: optimization
• Set the gradient to 0 to get the minimizer
𝛻𝑤
𝐿 𝑓𝑤 = 𝛻𝑤
1
𝑛
⃦𝑋𝑤 − 𝑦 ⃦2
2
= 0
𝛻𝑤[ 𝑋𝑤 − 𝑦 𝑇(𝑋𝑤 − 𝑦)] = 0
𝛻𝑤[ 𝑤𝑇𝑋𝑇𝑋𝑤 − 2𝑤𝑇𝑋𝑇𝑦 + 𝑦𝑇𝑦] = 0
2𝑋𝑇𝑋𝑤 − 2𝑋𝑇𝑦 = 0
w = 𝑋𝑇𝑋 −1𝑋𝑇𝑦](https://image.slidesharecdn.com/mlbasicslecture1linearregression-240414082825-e094c87c/85/ML_basics_lecture1_linear_regression-pdf-29-320.jpg)
ML_basics_lecture1_linear_regression.pdf
- 1. Machine Learning Basics Lecture 1: Linear Regression Princeton University COS 495 Instructor: Yingyu Liang
- 3. What is machine learning? • “A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T as measured by P, improves with experience E.” ------- Machine Learning, Tom Mitchell, 1997
- 4. Example 1: image classification Task: determine if the image is indoor or outdoor Performance measure: probability of misclassification
- 5. Example 1: image classification indoor outdoor Experience/Data: images with labels Indoor
- 6. Example 1: image classification • A few terminologies • Training data: the images given for learning • Test data: the images to be classified • Binary classification: classify into two classes
- 7. Example 1: image classification (multi-class) ImageNet figure borrowed from vision.standford.edu
- 8. Example 2: clustering images Task: partition the images into 2 groups Performance: similarities within groups Data: a set of images
- 9. Example 2: clustering images • A few terminologies • Unlabeled data vs labeled data • Supervised learning vs unsupervised learning
- 10. Math formulation Color Histogram Red Green Blue Indoor 0 Feature vector: 𝑥𝑖 Label: 𝑦𝑖 Extract features
- 11. Math formulation Color Histogram Red Green Blue outdoor 1 Feature vector: 𝑥𝑗 Label: 𝑦𝑗 Extract features
- 12. Math formulation • Given training data 𝑥𝑖, 𝑦𝑖 : 1 ≤ 𝑖 ≤ 𝑛 • Find 𝑦 = 𝑓(𝑥) using training data • s.t. 𝑓 correct on test data What kind of functions?
- 13. Math formulation • Given training data 𝑥𝑖, 𝑦𝑖 : 1 ≤ 𝑖 ≤ 𝑛 • Find 𝑦 = 𝑓(𝑥) ∈ 𝓗 using training data • s.t. 𝑓 correct on test data Hypothesis class
- 14. Math formulation • Given training data 𝑥𝑖, 𝑦𝑖 : 1 ≤ 𝑖 ≤ 𝑛 • Find 𝑦 = 𝑓(𝑥) ∈ 𝓗 using training data • s.t. 𝑓 correct on test data Connection between training data and test data?
- 15. Math formulation • Given training data 𝑥𝑖, 𝑦𝑖 : 1 ≤ 𝑖 ≤ 𝑛 i.i.d. from distribution 𝐷 • Find 𝑦 = 𝑓(𝑥) ∈ 𝓗 using training data • s.t. 𝑓 correct on test data i.i.d. from distribution 𝐷 They have the same distribution i.i.d.: independently identically distributed
- 16. Math formulation • Given training data 𝑥𝑖, 𝑦𝑖 : 1 ≤ 𝑖 ≤ 𝑛 i.i.d. from distribution 𝐷 • Find 𝑦 = 𝑓(𝑥) ∈ 𝓗 using training data • s.t. 𝑓 correct on test data i.i.d. from distribution 𝐷 What kind of performance measure?
- 17. Math formulation • Given training data 𝑥𝑖, 𝑦𝑖 : 1 ≤ 𝑖 ≤ 𝑛 i.i.d. from distribution 𝐷 • Find 𝑦 = 𝑓(𝑥) ∈ 𝓗 using training data • s.t. the expected loss is small 𝐿 𝑓 = 𝔼 𝑥,𝑦 ~𝐷[𝑙(𝑓, 𝑥, 𝑦)] Various loss functions
- 18. Math formulation • Given training data 𝑥𝑖, 𝑦𝑖 : 1 ≤ 𝑖 ≤ 𝑛 i.i.d. from distribution 𝐷 • Find 𝑦 = 𝑓(𝑥) ∈ 𝓗 using training data • s.t. the expected loss is small 𝐿 𝑓 = 𝔼 𝑥,𝑦 ~𝐷[𝑙(𝑓, 𝑥, 𝑦)] • Examples of loss functions: • 0-1 loss: 𝑙 𝑓, 𝑥, 𝑦 = 𝕀[𝑓 𝑥 ≠ 𝑦] and 𝐿 𝑓 = Pr[𝑓 𝑥 ≠ 𝑦] • 𝑙2 loss: 𝑙 𝑓, 𝑥, 𝑦 = [𝑓 𝑥 − 𝑦]2 and 𝐿 𝑓 = 𝔼[𝑓 𝑥 − 𝑦]2
- 19. Math formulation • Given training data 𝑥𝑖, 𝑦𝑖 : 1 ≤ 𝑖 ≤ 𝑛 i.i.d. from distribution 𝐷 • Find 𝑦 = 𝑓(𝑥) ∈ 𝓗 using training data • s.t. the expected loss is small 𝐿 𝑓 = 𝔼 𝑥,𝑦 ~𝐷[𝑙(𝑓, 𝑥, 𝑦)] How to use?
- 20. Math formulation • Given training data 𝑥𝑖, 𝑦𝑖 : 1 ≤ 𝑖 ≤ 𝑛 i.i.d. from distribution 𝐷 • Find 𝑦 = 𝑓(𝑥) ∈ 𝓗 that minimizes 𝐿 𝑓 = 1 𝑛 σ𝑖=1 𝑛 𝑙(𝑓, 𝑥𝑖, 𝑦𝑖) • s.t. the expected loss is small 𝐿 𝑓 = 𝔼 𝑥,𝑦 ~𝐷[𝑙(𝑓, 𝑥, 𝑦)] Empirical loss
- 21. Machine learning 1-2-3 • Collect data and extract features • Build model: choose hypothesis class 𝓗 and loss function 𝑙 • Optimization: minimize the empirical loss
- 22. Wait… • Why handcraft the feature vectors 𝑥, 𝑦? • Can use prior knowledge to design suitable features • Can computer learn the features on the raw images? • Learn features directly on the raw images: Representation Learning • Deep Learning ⊆ Representation Learning ⊆ Machine Learning ⊆ Artificial Intelligence
- 23. Wait… • Does MachineLearning-1-2-3 include all approaches? • Include many but not all • Our current focus will be MachineLearning-1-2-3
- 24. Example: Stock Market Prediction 2013 2014 2015 2016 Stock Market (Disclaimer: synthetic data/in another parallel universe) Orange MacroHard Ackermann Sliding window over time: serve as input 𝑥; non-i.i.d.
- 26. Real data: Prostate Cancer by Stamey et al. (1989) Figure borrowed from The Elements of Statistical Learning 𝑦: prostate specific antigen (𝑥1, … , 𝑥8): clinical measures
- 27. Linear regression • Given training data 𝑥𝑖, 𝑦𝑖 : 1 ≤ 𝑖 ≤ 𝑛 i.i.d. from distribution 𝐷 • Find 𝑓𝑤 𝑥 = 𝑤𝑇𝑥 that minimizes 𝐿 𝑓𝑤 = 1 𝑛 σ𝑖=1 𝑛 𝑤𝑇𝑥𝑖 − 𝑦𝑖 2 𝑙2 loss; also called mean square error Hypothesis class 𝓗
- 28. Linear regression: optimization • Given training data 𝑥𝑖, 𝑦𝑖 : 1 ≤ 𝑖 ≤ 𝑛 i.i.d. from distribution 𝐷 • Find 𝑓𝑤 𝑥 = 𝑤𝑇𝑥 that minimizes 𝐿 𝑓𝑤 = 1 𝑛 σ𝑖=1 𝑛 𝑤𝑇𝑥𝑖 − 𝑦𝑖 2 • Let 𝑋 be a matrix whose 𝑖-th row is 𝑥𝑖 𝑇 , 𝑦 be the vector 𝑦1, … , 𝑦𝑛 𝑇 𝐿 𝑓𝑤 = 1 𝑛 𝑖=1 𝑛 𝑤𝑇𝑥𝑖 − 𝑦𝑖 2 = 1 𝑛 ⃦𝑋𝑤 − 𝑦 ⃦2 2
- 29. Linear regression: optimization • Set the gradient to 0 to get the minimizer 𝛻𝑤 𝐿 𝑓𝑤 = 𝛻𝑤 1 𝑛 ⃦𝑋𝑤 − 𝑦 ⃦2 2 = 0 𝛻𝑤[ 𝑋𝑤 − 𝑦 𝑇(𝑋𝑤 − 𝑦)] = 0 𝛻𝑤[ 𝑤𝑇𝑋𝑇𝑋𝑤 − 2𝑤𝑇𝑋𝑇𝑦 + 𝑦𝑇𝑦] = 0 2𝑋𝑇𝑋𝑤 − 2𝑋𝑇𝑦 = 0 w = 𝑋𝑇𝑋 −1𝑋𝑇𝑦
- 30. Linear regression: optimization • Algebraic view of the minimizer • If 𝑋 is invertible, just solve 𝑋𝑤 = 𝑦 and get 𝑤 = 𝑋−1𝑦 • But typically 𝑋 is a tall matrix 𝑋 𝑤 = 𝑦 𝑋𝑇 𝑋 𝑤 = 𝑋𝑇 𝑦 Normal equation: w = 𝑋𝑇 𝑋 −1 𝑋𝑇 𝑦
- 31. Linear regression with bias • Given training data 𝑥𝑖, 𝑦𝑖 : 1 ≤ 𝑖 ≤ 𝑛 i.i.d. from distribution 𝐷 • Find 𝑓𝑤,𝑏 𝑥 = 𝑤𝑇𝑥 + 𝑏 to minimize the loss • Reduce to the case without bias: • Let 𝑤′ = 𝑤; 𝑏 , 𝑥′ = 𝑥; 1 • Then 𝑓𝑤,𝑏 𝑥 = 𝑤𝑇 𝑥 + 𝑏 = 𝑤′ 𝑇 (𝑥′ ) Bias term