Introduction to batch normalization
1 like372 views
Batch normalization is a technique introduced in 2015 by Google researchers to address issues like internal covariate shift and vanishing gradients. It works by normalizing the inputs to each unit to have zero mean and unit variance based on the statistics of the mini-batch. This helps the network train deeper models with higher learning rates and be less sensitive to initialization. Batch normalization is applied before the activation function of each layer during both training and inference.
Introduction to batch normalization
- 2. Batch Normalization Distribution changes Higher learning rate Less care about initialization 2015, ICML, Google Researchers Mini-Batch
- 3. Recap When training neural networks we'll feed in observations and compare the expected output to the true output of the network. We'll then use gradient descent to update the parameters of the model in the direction which will minimize the difference between our expected (or ideal) outcome and the true outcome. In other words, we're attempting to minimize the error we observe in our model's predictions. Ultimately, gradient descent is a search among a loss function surface in an attempt to find the values for each parameter such that the loss function is minimized. In other words, we're looking for the lowest value on the loss function surface. 3 Source: jeremyjordan
- 4. Recap height age NBA? age height Contour Plot (Loss function) 4
- 7. Learning Rate Matters 7 Source: jeremyjordan
- 8. Why Feature Scaling Matter Player Age Height NBA #1 45 190 No #2 19 192 Yes #3 32 195 No #4 48 200 No #5 25 193 Yes #6 31 182 Yes #7 49 177 No #8 22 201 Yes 8
- 9. Feature Scaling (Normalization) 9 age Contour Plot (Loss function) age height Contour Plot (Loss function) height
- 10. 10 Z-Score Normalization Min-Max Normalization Source: Codecademy
- 12. 12 平均數 (Mean)
- 14. Batch, Stochastic & Mini Batch Gradient Descent 14 (Batch) Gradient Descent Stochastic Gradient Descent Mini Batch Gradient Descent (Batch = 4)
- 15. Vanishing gradient Problem 15 Source: Andre Ye
- 16. Batch Normalization Distribution changes Higher learning rate Less care about initialization 2015, ICML, Google Researchers Mini-Batch
- 17. 17 直覺 ★ 既然 feature scaling 對於 input layer 有幫助，那 hidden layer 應該也有幫助 ★ 如果你把麥克風放大器的旋鈕轉到接近0，別人就聽不到你的聲音，但如果你 把它轉到接近最大聲，你的聲音就會飽和。 ★ 現在想像一下你將這種放大器串接起來，你必須正確的設定它們，才能讓聲音 在串接的末端既響亮又清晰。你的聲音從各個放大器出來時的振幅必須和它進 入該放大器時一樣。
- 18. BN Layer c b a output Y w1 w2 w3 x = aw1 +bw2 +cw3 f(x) 18 Linear Combination Activation function
- 19. 19
- 20. 20 Before/After Activation Function Input Output Source: 莫煩
- 21. 21 Back to Paper μ and σ2 are calculated on a per-batch basis while γ and β are learned parameters used across all batches.
- 23. 23 Stable Training 2018, NIPS, MIT Internal Covariate Shift Smooth Landscape
- 24. 24 Source: How Does Batch Normalization Help Optimization?
- 25. Pros and Cons Pros ➔ Allow sub-optimal starts (weight initialization) ➔ Speed up training (Larger learning rate) ➔ Solve vanishing gradient problem ➔ Acts a regularizer (Introduce randomness) Cons ➔ Small batch size leads to unstable mean and variance (Batch >= 32) ➔ Not for RNN (Recurrent neural networks) 25
- 26. Further Reading - 什麼是 Batch Normalization 批標準化 (Youtube 5:08) - Batch Normalization - Explained (Youtube 8:48) - Batch Normalization 介紹 - (Medium in Mandarin) - Normalizing your data (specifically, input and batch normalization) 26