CS571: Gradient Descent
1 like1,902 views
Gradient descent is an optimization algorithm used in supervised learning problems to minimize a loss function. It works by taking steps in the negative direction of the gradient of the loss function with respect to the weights at each iteration. Stochastic gradient descent is a variant that updates the weights for each training example, rather than for the full batch. The perceptron is an algorithm that updates the weights only when the prediction is incorrect. It converges when the weights correctly classify all training examples.
CS571: Gradient Descent
- 1. Gradient Descent Natural Language Processing Emory University Jinho D. Choi
- 2. ˆE(f) = 1 n nX i=1 `(ˆyi; yi) E(f) = Z `(ˆy; y) · P(x, y) Supervised Learning 2 (X, Y ) = {(x1, y1), . . . , (xn, yn)} ˆy = f(x) predicts the output of x input prediction loss function joint distribution Expected risk unknown! Empirical risk minimize! output y = ±1 binomial distribution
- 3. `(w, x; y) = 1 2 (wT x y)2 ˆE(f) = 1 n nX i=1 `(ˆyi; yi) Linear Prediction 3 least squares linear function Find a weight vector that minimizes the loss. `(ˆy; y) = 1 2 (ˆy y)2 ˆy = f(x) = wT (x) = wT x feature vector
- 4. wt+1 wt ⌘t 1 n nX i=1 @ @w `(wt, xi; yi) Gradient Descent 4 learning rate derivative of the loss Minimize loss Derivative → 0 Global optimum? Convex optimization
- 5. Gradient Descent 5 How often is the weight vector updated? wt+1 wt ⌘t 1 n nX i=1 @ @w `(wt, xi; yi) `(w, x; y) = 1 2 (wT x y)2 @ @w `(w, x; y) = @ @w 1 2 (wT x y)2 = (wT x y)x wt+1 wt ⌘t 1 n nX i=1 (wT xi yi)xi
- 6. Stochastic Gradient Descent 6 wt+1 wt ⌘t 1 n nX i=1 (wT xi yi)xi wt+1 wt ⌘t(wT t xi yi)xi 0 + - w0 0 wT 0 x1 > 0 wT 1 x2 < 0 wT 2 x3 < 0 w3 w2 ⌘( 1)x3 w2 w1 ⌘( + 1)x2 w1 w0 ⌘( + 1)x1 wT 3 x4 > 0 w4 w3 ⌘( 1)x4 updated for every instance
- 7. Perceptron 7 wt+1 wt ⌘t ` Stochastic gradient descent wt+1 wt + ⌘t ⇢ x · y wT t x · y < 0 0 otherwise `(w, x; y) = 1 2 (wT x y)2 ` = (wT x y)x Least squares ` = ⇢ x · y wT x · y < 0 0 otherwise `(w, x; y) = max{0, wT x · y} Perceptron
- 8. Averaged Perceptron 8 The final hyperplane may be overfitted to later instances. Take the average of all hyperplanes including ones that are not updated.
- 9. Averaged Perceptron 9 c c + 1 Initialization: Update rule: for every instance c 1 sparse vector? wt+1 wt + ⌘t(x · y) vt+1 vt + ⌘t · c(x · y) w w 1 c · v wt+1 wt + ⌘t(x · y) if wT t x · y < 0 w 1 c c 1X t=0 wt
- 10. Emory University Logo Guidelines - Multinomial Perceptron 10 Binomial distribution requires 1 hyperplane to separate 2 classes. Multinomial distribution requires m hyperplanes to separate m classes. How many for m classes?
- 11. Multinomial Perceptron 11 a b c d ew = 1 0 0 1 0x = wT x = a + d ˆy = ⇢ 1 wT x 0 1 otherwise a0 a1 a2 a3 b0 b1 b2 b3 c0 c1 c2 c3 d0 d1 d2 d3 e0 e1 e2 e3w = 5 features (including bias) Binomial Multinomial y = {0, 1, 2, 3} ˆy = arg max y wT y xwT y x = ay + dy y = { 1, 1}
- 12. Binomial vs. Multinomial Perceptron 12 wt+1 wt + ⌘t(x · y) Binomial wy,t+1 wy,t + ⌘t · x Multinomial wˆy,t+1 wˆy,t ⌘t · x if wT t x · y < 0 , y 6= ˆy
- 13. Hinge Loss 13 ` = ⇢ x · y wT x · y < 0 0 otherwise `(w, x; y) = max{0, wT x · y} Perceptron Hinge loss `(w, x; y) = max{0, 1 wT x · y} ` = ⇢ x · y wT x · y < 1 0 otherwise
- 14. Adaptive Gradient Descent 14 if wT t x · y < 0 Perceptron if wT t · y < 1 Hinge loss wt+1 wt + ⌘t(x · y) gt+1 gt + x x wt+1 wt + ⌘ ⇢ + p gt+1 · (x · y)