![Example: character-level language processing
X
y
RNN
Training sequence:
”hello”
Vocabulary:
[e, h, l, o]
0
1
0
0
1
0
0
0
0
0
1
0
0
0
0
1
“h”“e” “l” “0”
𝑊ℎℎ
𝑊ℎ𝑦
𝑊𝑥ℎ](https://image.slidesharecdn.com/rnnpresentation-170215220631/85/Recurrent-Networks-and-LSTM-deep-dive-5-320.jpg)
![hX Y
𝑊ℎℎ = 4.1
𝑊𝑥ℎ = [3.6 −4.8 0.35 −0.26]
𝑊ℎ𝑦 =
−12.
−0.67
−0.85
14.
P
𝑏ℎ = 0.41
𝑏 𝑦 =
−0.2
−2.9
6.1
−3.4
“hello” RNN](https://image.slidesharecdn.com/rnnpresentation-170215220631/85/Recurrent-Networks-and-LSTM-deep-dive-6-320.jpg)
![ℎ1ℎ0
1 1 2
3
ℎ2
𝑥0 𝑥1 𝑥2
𝐿 = 𝑓(𝑊𝑥ℎ, 𝑊ℎℎ, 𝑊ℎ𝑦)𝑊𝑥ℎ = 0.078
𝑊ℎ𝑦 = 0.051
𝑊ℎℎ = 0.024
𝑤 𝑥ℎ ≔ 𝑤 𝑥ℎ − 0.01 ∙
𝜕𝐿
𝜕𝑤 𝑥ℎ
𝑤ℎℎ ≔ 𝑤ℎℎ − 0.01 ∙
𝜕𝐿
𝜕𝑤ℎℎ
𝑤ℎ𝑦 ≔ 𝑤ℎ𝑦 − 0.01 ∙
𝜕𝐿
𝜕𝑤ℎ𝑦
Training is hard with vanilla RNNs
𝛻𝐿 = [
𝜕𝐿
𝜕𝑤 𝑥ℎ
,
𝜕𝐿
𝜕𝑤ℎℎ
,
𝜕𝐿
𝜕𝑤ℎ𝑦
]
𝑊𝑥ℎ
𝑊ℎℎ
𝑊ℎ𝑦
<— Forward pass
<— Backward pass](https://image.slidesharecdn.com/rnnpresentation-170215220631/85/Recurrent-Networks-and-LSTM-deep-dive-42-320.jpg)
Recurrent Networks and LSTM deep dive
- 1. Recurrent Neural Networks Alex Kalinin alex@alexkalinin.com
- 2. Content 1. Example of Vanilla RNN 2. RNN Forward pass 3. RNN Backward pass 4. LSTM design RNN Training problem
- 4. X y RNN h 𝑊ℎℎ 𝑊ℎ𝑦 𝑊𝑥ℎ Vanilla recurrent network 1) ℎ 𝑡 = tanh 𝑊ℎℎℎ 𝑡−1 + 𝑊𝑥ℎ 𝑥 + 𝑏ℎ 2) 𝑦 = 𝑊ℎ𝑦ℎ 𝑡 + 𝑏 𝑦
- 5. Example: character-level language processing X y RNN Training sequence: ”hello” Vocabulary: [e, h, l, o] 0 1 0 0 1 0 0 0 0 0 1 0 0 0 0 1 “h”“e” “l” “0” 𝑊ℎℎ 𝑊ℎ𝑦 𝑊𝑥ℎ
- 6. hX Y 𝑊ℎℎ = 4.1 𝑊𝑥ℎ = [3.6 −4.8 0.35 −0.26] 𝑊ℎ𝑦 = −12. −0.67 −0.85 14. P 𝑏ℎ = 0.41 𝑏 𝑦 = −0.2 −2.9 6.1 −3.4 “hello” RNN
- 7. hX Y P 0 1 0 0 “h” ℎ0 = 0 “h”
- 8. hX Y P 0 1 0 0 “h” ℎ 𝑡 = tanh 𝑊ℎℎℎ 𝑡−1 + 𝑊𝑥ℎ 𝑥 + 𝑏ℎ ℎ0 = 0 “h”
- 9. hX Y P 0 1 0 0 “h” ℎ = −0.99 “h”
- 10. hX Y P 0 1 0 0 “h” ℎ = −0.99 𝑦 = 𝑊ℎ𝑦ℎ 𝑡 + 𝑏 𝑦 “h”
- 11. hX Y P 0 1 0 0 “h” ℎ = −0.99 𝑦 = 11. −2.2 6.9 −17 “h”
- 12. hX Y P 0 1 0 0 “h” ℎ = −0.99 𝑦 = 11. −2.2 6.9 −17 𝑝 = 0.99 0 0.01 0 “h”
- 13. hX Y P 0 1 0 0 “h” ℎ = −0.99 𝑦 = 11. −2.2 6.9 −17 𝑝 = 0.99 0 0.01 0 1 0 0 0 “e” “h”
- 14. hX Y P 1 0 0 0 “e” ℎ = −0.99 “h” “e”
- 15. hX Y P 1 0 0 0 “e” ℎ = −0.99 ℎ 𝑡 = tanh 𝑊ℎℎℎ 𝑡−1 + 𝑊𝑥ℎ 𝑥 + 𝑏ℎ “h” “e”
- 16. hX Y P 1 0 0 0 “e” ℎ = −0.09 “h” “e”
- 17. hX Y P 1 0 0 0 “e” ℎ = −0.09 𝑦 = 𝑊ℎ𝑦ℎ 𝑡 + 𝑏 𝑦 “h” “e”
- 18. hX Y P 1 0 0 0 “e” ℎ = −0.09 𝑦 = 0.86 −2.8 6.2 −4.6 “h” “e”
- 19. hX Y P 1 0 0 0 “e” ℎ = −0.09 𝑦 = 0.86 −2.8 6.2 −4.6 𝑝 = 0 0 0.99 0 “h” “e”
- 20. hX Y P 1 0 0 0 “e” ℎ = −0.09 𝑦 = 0.86 −2.8 6.2 −4.6 𝑝 = 0 0 0.99 0 0 0 1 0 “l” “h” “e”
- 21. hX Y P 0 0 1 0 “l” ℎ = −0.09 “h” “e” “l”
- 22. hX Y P 0 0 1 0 “l” ℎ = 0.38 “h” “e” “l”
- 23. hX Y P 0 0 1 0 “l” ℎ = 0.38 𝑦 = −4.7 −3.2 5.8 1.9 “h” “e” “l”
- 24. hX Y P 0 0 1 0 “l” ℎ = 0.38 𝑦 = −4.7 −3.2 5.8 1.9 𝑝 = 0 0 0.98 0.02 “h” “e” “l”
- 25. hX Y P 0 0 1 0 “l” ℎ = 0.38 𝑦 = −4.7 −3.2 5.8 1.9 𝑝 = 0 0 0.98 0.02 0 0 1 0 “l” “h” “e” “l”
- 26. hX Y P 0 0 1 0 “l” ℎ = 0.38 “h” “e” “l” “l”
- 27. hX Y P 0 0 1 0 “l” ℎ = 0.98 “h” “e” “l” “l”
- 28. hX Y P 0 0 1 0 “l” ℎ = 0.98 “h” “e” “l” “l” 𝑦 = −12. −3.6 5.3 10.
- 29. hX Y P 0 0 1 0 “l” ℎ = 0.98 “h” “e” “l” “l” 𝑦 = −12. −3.6 5.3 10. 𝑝 = 0 0 0.01 0.99
- 30. hX Y P 0 0 1 0 “l” ℎ = 0.98 “h” “e” “l” “l” 𝑦 = −12. −3.6 5.3 10. 𝑝 = 0 0 0.01 0.99 0 0 0 1 “o”
- 31. hX Y P ℎ = 0.98 “h” “e” “l” “l” “o”
- 32. hX Y P “h” ℎ0 = 0 “e”⨁ “e” ℎ1 =-0.99 “l”⨁ “l” ℎ2 =-0.09 “l”⨁ “l” ℎ3 =0.38 “o”⨁
- 33. hX Y P “hello” “hello” “hello ben” “hello ben” “hello world” “hello world”
- 34. hX Y P “it was” “it was” “it was the” “it was the” “it was the best” “it was the best” “It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness… “, A Tale of Two Cities, Charles Dickens 50,000 300,000 (loss = 1.6066) 1,000,000 (loss = 1.8197) “it was the best of” “it wes the best of” 2,000,000 (loss = 4.0844)
- 35. hX Y P … epoch 500000, loss: 6.447782290456328 … epoch 1000000, loss: 5.290576956983398 … epoch 1800000, loss: 4.267105168323299 epoch 1900000, loss: 4.175163586546514 epoch 2000000, loss: 4.0844739848413285
- 36. X y RNN h 𝑊ℎℎ 𝑊ℎ𝑦 𝑊𝑥ℎ Vanilla recurrent network 1) ℎ 𝑡 = tanh 𝑊ℎℎℎ 𝑡−1 + 𝑊𝑥ℎ 𝑥 + 𝑏ℎ 2) 𝑦 = 𝑊ℎ𝑦ℎ 𝑡 + 𝑏 𝑦
- 37. Input: Target: i t “ “ w a s “ “ t “ “ w a s “ “ t h t
- 38. RNNs for Different Problems Vanilla Neural Network
- 39. RNNs for Different Problems Image Captioning image -> sequence of words
- 40. RNNs for Different Problems Sentiment Analysis sequence of words -> class
- 41. RNNs for Different Problems Translation sequence of words -> sequence of words
- 42. ℎ1ℎ0 1 1 2 3 ℎ2 𝑥0 𝑥1 𝑥2 𝐿 = 𝑓(𝑊𝑥ℎ, 𝑊ℎℎ, 𝑊ℎ𝑦)𝑊𝑥ℎ = 0.078 𝑊ℎ𝑦 = 0.051 𝑊ℎℎ = 0.024 𝑤 𝑥ℎ ≔ 𝑤 𝑥ℎ − 0.01 ∙ 𝜕𝐿 𝜕𝑤 𝑥ℎ 𝑤ℎℎ ≔ 𝑤ℎℎ − 0.01 ∙ 𝜕𝐿 𝜕𝑤ℎℎ 𝑤ℎ𝑦 ≔ 𝑤ℎ𝑦 − 0.01 ∙ 𝜕𝐿 𝜕𝑤ℎ𝑦 Training is hard with vanilla RNNs 𝛻𝐿 = [ 𝜕𝐿 𝜕𝑤 𝑥ℎ , 𝜕𝐿 𝜕𝑤ℎℎ , 𝜕𝐿 𝜕𝑤ℎ𝑦 ] 𝑊𝑥ℎ 𝑊ℎℎ 𝑊ℎ𝑦 <— Forward pass <— Backward pass
- 43. ℎ1ℎ0 1 1 2 3 ℎ2 𝑥0 𝑥1 𝑥2 ℎ0 = tanh(𝑊𝑥ℎ 𝑥0) ℎ1 = tanh(𝑊ℎℎℎ0 + 𝑊𝑥ℎ 𝑥1) ℎ2 = tanh(𝑊ℎℎℎ1 + 𝑊𝑥ℎ 𝑥2) 𝑦 = 𝑊ℎ𝑦ℎ2 𝜕𝐿 𝜕𝑤ℎℎ =? 𝐿 = (𝑦 − 3)2 𝐿 =? y
- 44. 𝜕𝐿 𝜕𝑤 = 𝜕𝑓 𝜕𝑔 ∙ 𝜕𝑔 𝜕ℎ ∙ 𝜕ℎ 𝜕𝑘 ∙ 𝜕𝑘 𝜕𝑙 ∙ 𝜕𝑙 𝜕𝑚 ∙ 𝜕𝑚 𝜕𝑛 ∙ 𝜕𝑛 𝜕𝑤 𝐿 = 𝑓(𝑔 ℎ(𝑘(𝑙(𝑚 𝑛(𝑤) ))) ) 𝜕𝐿 𝜕𝑤ℎℎ =? 𝐿 = ( 𝑊ℎℎtanh(𝑊ℎℎtanh(𝑊ℎℎtanh(𝑊𝑥ℎ 𝑥0) + 𝑊𝑥ℎ 𝑥1) + 𝑊𝑥ℎ 𝑥2) − 3)2 Compute gradient Recursive application of chain rule: 𝜕𝐿 𝜕𝑤 =? 𝑓 = 𝑓(𝑔)𝑔 = 𝑔(ℎ)ℎ = ℎ(𝑘)
- 45. Gradient by hand
- 46. 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 𝑊𝑥ℎ = 0.078 𝑊ℎ𝑦 = 0.051 𝑊ℎℎ = 0.024 1 Forward Pass ℎ0 = tanh(𝑊𝑥ℎ 𝑥0) ℎ1 = tanh(𝑊ℎℎℎ0 + 𝑊𝑥ℎ 𝑥1) ℎ2 = tanh(𝑊ℎℎℎ1 + 𝑊𝑥ℎ 𝑥2) 𝑦 = 𝑊ℎ𝑦ℎ2 𝐿 = (𝑦 − 3)2 0.078 1. 𝑊𝑥ℎ 𝑥0
- 47. 𝑊𝑥ℎ = 0.078 𝑊ℎ𝑦 = 0.051 𝑊ℎℎ = 0.024 Forward Pass ℎ0 = tanh(𝑊𝑥ℎ 𝑥0) ℎ1 = tanh(𝑊ℎℎℎ0 + 𝑊𝑥ℎ 𝑥1) ℎ2 = tanh(𝑊ℎℎℎ1 + 𝑊𝑥ℎ 𝑥2) 𝑦 = 𝑊ℎ𝑦ℎ2 𝐿 = (𝑦 − 3)2 * 0.078 1. 𝑊𝑥ℎ 𝑥0 0.078 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 1
- 48. 𝑊𝑥ℎ = 0.078 𝑊ℎ𝑦 = 0.051 𝑊ℎℎ = 0.024 Forward Pass ℎ0 = tanh(𝑊𝑥ℎ 𝑥0) ℎ1 = tanh(𝑊ℎℎℎ0 + 𝑊𝑥ℎ 𝑥1) ℎ2 = tanh(𝑊ℎℎℎ1 + 𝑊𝑥ℎ 𝑥2) 𝑦 = 𝑊ℎ𝑦ℎ2 𝐿 = (𝑦 − 3)2 * 0.078 1. 𝑊𝑥ℎ 𝑥0 0.078 tanh 0.0778 ℎ0 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 1
- 49. 𝑊𝑥ℎ = 0.078 𝑊ℎ𝑦 = 0.051 𝑊ℎℎ = 0.024 Forward Pass ℎ0 = tanh(𝑊𝑥ℎ 𝑥0) ℎ1 = tanh(𝑊ℎℎℎ0 + 𝑊𝑥ℎ 𝑥1) ℎ2 = tanh(𝑊ℎℎℎ1 + 𝑊𝑥ℎ 𝑥2) 𝑦 = 𝑊ℎ𝑦ℎ2 𝐿 = (𝑦 − 3)2 * 0.078 1. 𝑊𝑥ℎ 𝑥0 𝑊ℎℎ 0.024 0.078 tanh 0.0778 ℎ0 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 1
- 50. 𝑊𝑥ℎ = 0.078 𝑊ℎ𝑦 = 0.051 𝑊ℎℎ = 0.024 Forward Pass ℎ0 = tanh(𝑊𝑥ℎ 𝑥0) ℎ1 = tanh(𝑊ℎℎℎ0 + 𝑊𝑥ℎ 𝑥1) ℎ2 = tanh(𝑊ℎℎℎ1 + 𝑊𝑥ℎ 𝑥2) 𝑦 = 𝑊ℎ𝑦ℎ2 𝐿 = (𝑦 − 3)2 * 0.078 1. 𝑊𝑥ℎ 𝑥0 𝑊ℎℎ 0.024 0.078 tanh 0.0778 * 0.00187 ℎ0 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 1
- 51. 𝑊𝑥ℎ = 0.078 𝑊ℎ𝑦 = 0.051 𝑊ℎℎ = 0.024 Forward Pass ℎ0 = tanh(𝑊𝑥ℎ 𝑥0) ℎ1 = tanh(𝑊ℎℎℎ0 + 𝑊𝑥ℎ 𝑥1) ℎ2 = tanh(𝑊ℎℎℎ1 + 𝑊𝑥ℎ 𝑥2) 𝑦 = 𝑊ℎ𝑦ℎ2 𝐿 = (𝑦 − 3)2 * 0.078 1. 𝑊𝑥ℎ 𝑥0 𝑊ℎℎ 0.024 0.078 tanh 0.0778 * 0.00187 0.078 1. 𝑊𝑥ℎ 𝑥1 ℎ0 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 1
- 52. 𝑊𝑥ℎ = 0.078 𝑊ℎ𝑦 = 0.051 𝑊ℎℎ = 0.024 Forward Pass ℎ0 = tanh(𝑊𝑥ℎ 𝑥0) ℎ1 = tanh(𝑊ℎℎℎ0 + 𝑊𝑥ℎ 𝑥1) ℎ2 = tanh(𝑊ℎℎℎ1 + 𝑊𝑥ℎ 𝑥2) 𝑦 = 𝑊ℎ𝑦ℎ2 𝐿 = (𝑦 − 3)2 * 0.078 1. 𝑊𝑥ℎ 𝑥0 𝑊ℎℎ 0.024 0.078 tanh 0.0778 * 0.00187 * 0.078 1. 𝑊𝑥ℎ 𝑥1 0.078 ℎ0 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 1
- 53. 𝑊𝑥ℎ = 0.078 𝑊ℎ𝑦 = 0.051 𝑊ℎℎ = 0.024 Forward Pass ℎ0 = tanh(𝑊𝑥ℎ 𝑥0) ℎ1 = tanh(𝑊ℎℎℎ0 + 𝑊𝑥ℎ 𝑥1) ℎ2 = tanh(𝑊ℎℎℎ1 + 𝑊𝑥ℎ 𝑥2) 𝑦 = 𝑊ℎ𝑦ℎ2 𝐿 = (𝑦 − 3)2 * 0.078 1. 𝑊𝑥ℎ 𝑥0 𝑊ℎℎ 0.024 0.078 tanh 0.0778 * 0.00187 * 0.078 1. 𝑊𝑥ℎ 𝑥1 0.078 ℎ0 + 0.07987 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 1
- 54. 𝑊𝑥ℎ = 0.078 𝑊ℎ𝑦 = 0.051 𝑊ℎℎ = 0.024 Forward Pass ℎ0 = tanh(𝑊𝑥ℎ 𝑥0) ℎ1 = tanh(𝑊ℎℎℎ0 + 𝑊𝑥ℎ 𝑥1) ℎ2 = tanh(𝑊ℎℎℎ1 + 𝑊𝑥ℎ 𝑥2) 𝑦 = 𝑊ℎ𝑦ℎ2 𝐿 = (𝑦 − 3)2 * 0.078 1. 𝑊𝑥ℎ 𝑥0 𝑊ℎℎ 0.024 0.078 tanh 0.0778 * 0.00187 * 0.078 1. 𝑊𝑥ℎ 𝑥1 0.078 ℎ0 + 0.07987 ℎ1 0.07970 tanh 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 1
- 55. 𝑊𝑥ℎ = 0.078 𝑊ℎ𝑦 = 0.051 𝑊ℎℎ = 0.024 Forward Pass ℎ0 = tanh(𝑊𝑥ℎ 𝑥0) ℎ1 = tanh(𝑊ℎℎℎ0 + 𝑊𝑥ℎ 𝑥1) ℎ2 = tanh(𝑊ℎℎℎ1 + 𝑊𝑥ℎ 𝑥2) 𝑦 = 𝑊ℎ𝑦ℎ2 𝐿 = (𝑦 − 3)2 * 0.078 1. 𝑊𝑥ℎ 𝑥0 𝑊ℎℎ 0.024 0.078 tanh 0.0778 * 0.00187 * 0.078 1. 𝑊𝑥ℎ 𝑥1 0.078 ℎ0 + 0.07987 ℎ1 0.07970 𝑊ℎℎ 0.024 tanh 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 1
- 56. 𝑊𝑥ℎ = 0.078 𝑊ℎ𝑦 = 0.051 𝑊ℎℎ = 0.024 Forward Pass ℎ0 = tanh(𝑊𝑥ℎ 𝑥0) ℎ1 = tanh(𝑊ℎℎℎ0 + 𝑊𝑥ℎ 𝑥1) ℎ2 = tanh(𝑊ℎℎℎ1 + 𝑊𝑥ℎ 𝑥2) 𝑦 = 𝑊ℎ𝑦ℎ2 𝐿 = (𝑦 − 3)2 * 0.078 1. 𝑊𝑥ℎ 𝑥0 𝑊ℎℎ 0.024 0.078 tanh 0.0778 * 0.00187 * 0.078 1. 𝑊𝑥ℎ 𝑥1 0.078 ℎ0 + 0.07987 ℎ1 0.07970 𝑊ℎℎ 0.024 * 0.0019 tanh 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 1
- 57. 𝑊𝑥ℎ = 0.078 𝑊ℎ𝑦 = 0.051 𝑊ℎℎ = 0.024 Forward Pass ℎ0 = tanh(𝑊𝑥ℎ 𝑥0) ℎ1 = tanh(𝑊ℎℎℎ0 + 𝑊𝑥ℎ 𝑥1) ℎ2 = tanh(𝑊ℎℎℎ1 + 𝑊𝑥ℎ 𝑥2) 𝑦 = 𝑊ℎ𝑦ℎ2 𝐿 = (𝑦 − 3)2 * 0.078 1. 𝑊𝑥ℎ 𝑥0 𝑊ℎℎ 0.024 0.078 tanh 0.0778 * 0.00187 * 0.078 1. 𝑊𝑥ℎ 𝑥1 0.078 ℎ0 + 0.07987 ℎ1 0.07970 0.078 2. 𝑊𝑥ℎ 𝑥2 𝑊ℎℎ 0.024 * 0.0019 tanh 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 1
- 58. 𝑊𝑥ℎ = 0.078 𝑊ℎ𝑦 = 0.051 𝑊ℎℎ = 0.024 Forward Pass ℎ0 = tanh(𝑊𝑥ℎ 𝑥0) ℎ1 = tanh(𝑊ℎℎℎ0 + 𝑊𝑥ℎ 𝑥1) ℎ2 = tanh(𝑊ℎℎℎ1 + 𝑊𝑥ℎ 𝑥2) 𝑦 = 𝑊ℎ𝑦ℎ2 𝐿 = (𝑦 − 3)2 * 0.078 1. 𝑊𝑥ℎ 𝑥0 𝑊ℎℎ 0.024 0.078 tanh 0.0778 * 0.00187 * 0.078 1. 𝑊𝑥ℎ 𝑥1 0.078 ℎ0 + 0.07987 ℎ1 0.07970 * 0.078 2. 𝑊𝑥ℎ 𝑥2 0.156 𝑊ℎℎ 0.024 * 0.0019 tanh 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 1
- 59. 𝑊𝑥ℎ = 0.078 𝑊ℎ𝑦 = 0.051 𝑊ℎℎ = 0.024 Forward Pass ℎ0 = tanh(𝑊𝑥ℎ 𝑥0) ℎ1 = tanh(𝑊ℎℎℎ0 + 𝑊𝑥ℎ 𝑥1) ℎ2 = tanh(𝑊ℎℎℎ1 + 𝑊𝑥ℎ 𝑥2) 𝑦 = 𝑊ℎ𝑦ℎ2 𝐿 = (𝑦 − 3)2 * 0.078 1. 𝑊𝑥ℎ 𝑥0 𝑊ℎℎ 0.024 0.078 tanh 0.0778 * 0.00187 * 0.078 1. 𝑊𝑥ℎ 𝑥1 0.078 ℎ0 + 0.07987 ℎ1 0.07970 * 0.078 2. 𝑊𝑥ℎ 𝑥2 0.156 𝑊ℎℎ 0.024 * 0.0019 + 0.1579 tanh 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 1
- 60. 𝑊𝑥ℎ = 0.078 𝑊ℎ𝑦 = 0.051 𝑊ℎℎ = 0.024 Forward Pass ℎ0 = tanh(𝑊𝑥ℎ 𝑥0) ℎ1 = tanh(𝑊ℎℎℎ0 + 𝑊𝑥ℎ 𝑥1) ℎ2 = tanh(𝑊ℎℎℎ1 + 𝑊𝑥ℎ 𝑥2) 𝑦 = 𝑊ℎ𝑦ℎ2 𝐿 = (𝑦 − 3)2 * 0.078 1. 𝑊𝑥ℎ 𝑥0 𝑊ℎℎ 0.024 0.078 tanh 0.0778 * 0.00187 * 0.078 1. 𝑊𝑥ℎ 𝑥1 0.078 ℎ0 + 0.07987 ℎ1 0.07970 * 0.078 2. 𝑊𝑥ℎ 𝑥2 0.156 𝑊ℎℎ 0.024 * 0.0019 + 0.1579 0.1566 ℎ2 tanh tanh 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 1
- 61. 𝑊𝑥ℎ = 0.078 𝑊ℎ𝑦 = 0.051 𝑊ℎℎ = 0.024 Forward Pass ℎ0 = tanh(𝑊𝑥ℎ 𝑥0) ℎ1 = tanh(𝑊ℎℎℎ0 + 𝑊𝑥ℎ 𝑥1) ℎ2 = tanh(𝑊ℎℎℎ1 + 𝑊𝑥ℎ 𝑥2) 𝑦 = 𝑊ℎ𝑦ℎ2 𝐿 = (𝑦 − 3)2 * 0.078 1. 𝑊𝑥ℎ 𝑥0 𝑊ℎℎ 0.024 0.078 tanh 0.0778 * 0.00187 * 0.078 1. 𝑊𝑥ℎ 𝑥1 0.078 ℎ0 + 0.07987 ℎ1 0.07970 * 0.078 2. 𝑊𝑥ℎ 𝑥2 0.156 𝑊ℎℎ 0.024 * 0.0019 + 0.1579 0.1566 ℎ2 0.051𝑊ℎ𝑦 tanh tanh 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 1
- 62. 𝑊𝑥ℎ = 0.078 𝑊ℎ𝑦 = 0.051 𝑊ℎℎ = 0.024 Forward Pass ℎ0 = tanh(𝑊𝑥ℎ 𝑥0) ℎ1 = tanh(𝑊ℎℎℎ0 + 𝑊𝑥ℎ 𝑥1) ℎ2 = tanh(𝑊ℎℎℎ1 + 𝑊𝑥ℎ 𝑥2) 𝑦 = 𝑊ℎ𝑦ℎ2 𝐿 = (𝑦 − 3)2 * 0.078 1. 𝑊𝑥ℎ 𝑥0 𝑊ℎℎ 0.024 0.078 tanh 0.0778 * 0.00187 * 0.078 1. 𝑊𝑥ℎ 𝑥1 0.078 ℎ0 + 0.07987 ℎ1 0.07970 * 0.078 2. 𝑊𝑥ℎ 𝑥2 0.156 𝑊ℎℎ 0.024 * 0.0019 + 0.1579 0.1566 ℎ2 0.051𝑊ℎ𝑦 * 0.0080 𝑦 tanh tanh 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 1
- 63. 𝑊𝑥ℎ = 0.078 𝑊ℎ𝑦 = 0.051 𝑊ℎℎ = 0.024 Forward Pass ℎ0 = tanh(𝑊𝑥ℎ 𝑥0) ℎ1 = tanh(𝑊ℎℎℎ0 + 𝑊𝑥ℎ 𝑥1) ℎ2 = tanh(𝑊ℎℎℎ1 + 𝑊𝑥ℎ 𝑥2) 𝑦 = 𝑊ℎ𝑦ℎ2 𝐿 = (𝑦 − 3)2 * 0.078 1. 𝑊𝑥ℎ 𝑥0 𝑊ℎℎ 0.024 0.078 tanh 0.0778 * 0.00187 * 0.078 1. 𝑊𝑥ℎ 𝑥1 0.078 ℎ0 + 0.07987 ℎ1 0.07970 * 0.078 2. 𝑊𝑥ℎ 𝑥2 0.156 𝑊ℎℎ 0.024 * 0.0019 + 0.1579 0.1566 ℎ2 0.051𝑊ℎ𝑦 * 0.0080 𝑦 -3 + -2.99 tanh tanh 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 1
- 64. 𝑊𝑥ℎ = 0.078 𝑊ℎ𝑦 = 0.051 𝑊ℎℎ = 0.024 Forward Pass ℎ0 = tanh(𝑊𝑥ℎ 𝑥0) ℎ1 = tanh(𝑊ℎℎℎ0 + 𝑊𝑥ℎ 𝑥1) ℎ2 = tanh(𝑊ℎℎℎ1 + 𝑊𝑥ℎ 𝑥2) 𝑦 = 𝑊ℎ𝑦ℎ2 𝐿 = (𝑦 − 3)2 * 0.078 1. 𝑊𝑥ℎ 𝑥0 𝑊ℎℎ 0.024 0.078 tanh 0.0778 * 0.00187 * 0.078 1. 𝑊𝑥ℎ 𝑥1 0.078 ℎ0 + 0.07987 ℎ1 0.07970 * 0.078 2. 𝑊𝑥ℎ 𝑥2 0.156 𝑊ℎℎ 0.024 * 0.0019 + 0.1579 0.1566 ℎ2 0.051𝑊ℎ𝑦 * 0.0080 𝑦 -3 + ** -2.99 8.95 𝐿 tanh tanh 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 1
- 65. 𝜕𝐿 𝜕𝑤 = 𝜕𝑓 𝜕𝑔 ∙ 𝜕𝑔 𝜕ℎ ∙ 𝜕ℎ 𝜕𝑘 ∙ 𝜕𝑘 𝜕𝑙 ∙ 𝜕𝑙 𝜕𝑚 ∙ 𝜕𝑚 𝜕𝑛 ∙ 𝜕𝑛 𝜕𝑤 𝐿 = 𝑓(𝑔 ℎ(𝑘(𝑙(𝑚 𝑛(𝑤) ))) ) 𝜕𝐿 𝜕𝑤ℎℎ =? Compute gradient Recursive application of chain rule:
- 66. Backward Pass * 0.078 1. 𝑊𝑥ℎ 𝑥0 𝑊ℎℎ 0.024 0.078 tanh 0.0778 * 0.00187 * 0.078 1. 𝑊𝑥ℎ 𝑥1 0.078 ℎ0 + 0.07987 ℎ1 0.07970 * 0.078 2. 𝑊𝑥ℎ 𝑥2 0.156 𝑊ℎℎ 0.024 * 0.0019 + 0.1579 0.1566 ℎ2 0.051𝑊ℎ𝑦 * 0.0080 𝑦 -3 + ** -2.99 8.95 𝐿 𝜕𝐿 𝜕𝑤 𝑥ℎ = 𝜕𝑓 𝜕𝑔 ∙ 𝜕𝑔 𝜕ℎ ∙ 𝜕ℎ 𝜕𝑘 ∙ 𝜕𝑘 𝜕𝑙 ∙ 𝜕𝑙 𝜕𝑚 ∙ 𝜕𝑚 𝜕𝑛 ∙ 𝜕𝑛 𝜕𝑤 𝑥ℎ tanh tanh 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 1
- 67. Backward Pass * 0.078 1. 𝑊𝑥ℎ 𝑥0 𝑊ℎℎ 0.024 0.078 tanh 0.0778 * 0.00187 * 0.078 1. 𝑊𝑥ℎ 𝑥1 0.078 ℎ0 + 0.07987 ℎ1 0.07970 * 0.078 2. 𝑊𝑥ℎ 𝑥2 0.156 𝑊ℎℎ 0.024 * 0.0019 + 0.1579 0.1566 ℎ2 0.051𝑊ℎ𝑦 * 0.0080 𝑦 -3 + ** -2.99 8.95 𝐿 𝜕𝐿 𝜕𝑤 𝑥ℎ = 𝝏𝒇 𝝏𝒇 ∙ 𝜕𝑓 𝜕𝑔 ∙ 𝜕𝑔 𝜕ℎ ∙ 𝜕ℎ 𝜕𝑘 ∙ 𝜕𝑘 𝜕𝑙 ∙ 𝜕𝑙 𝜕𝑚 ∙ 𝜕𝑚 𝜕𝑛 ∙ 𝜕𝑛 𝜕𝑤 𝑥ℎ 1 tanh tanh 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 1
- 68. Backward Pass * 0.078 1. 𝑊𝑥ℎ 𝑥0 𝑊ℎℎ 0.024 0.078 tanh 0.0778 * 0.00187 * 0.078 1. 𝑊𝑥ℎ 𝑥1 0.078 ℎ0 + 0.07987 ℎ1 0.07970 * 0.078 2. 𝑊𝑥ℎ 𝑥2 0.156 𝑊ℎℎ 0.024 * 0.0019 + 0.1579 0.1566 ℎ2 0.051𝑊ℎ𝑦 * 0.0080 𝑦 -3 + ** -2.99 8.95 𝐿 𝜕𝐿 𝜕𝑤 𝑥ℎ = 𝝏𝒇 𝝏𝒇 ∙ 𝝏𝒇 𝝏𝒈 ∙ 𝜕𝑔 𝜕ℎ ∙ 𝜕ℎ 𝜕𝑘 ∙ 𝜕𝑘 𝜕𝑙 ∙ 𝜕𝑙 𝜕𝑚 ∙ 𝜕𝑚 𝜕𝑛 ∙ 𝜕𝑛 𝜕𝑤 𝑥ℎ 1 𝜕𝑓 𝜕𝑔 =? 𝑓 = (𝑔)2 tanh tanh 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 1
- 69. Backward Pass * 0.078 1. 𝑊𝑥ℎ 𝑥0 𝑊ℎℎ 0.024 0.078 tanh 0.0778 * 0.00187 * 0.078 1. 𝑊𝑥ℎ 𝑥1 0.078 ℎ0 + 0.07987 ℎ1 0.07970 * 0.078 2. 𝑊𝑥ℎ 𝑥2 0.156 𝑊ℎℎ 0.024 * 0.0019 + 0.1579 0.1566 ℎ2 0.051𝑊ℎ𝑦 * 0.0080 𝑦 -3 + ** -2.99 8.95 𝐿 𝜕𝐿 𝜕𝑤 𝑥ℎ = 𝝏𝒇 𝝏𝒇 ∙ 𝝏𝒇 𝝏𝒈 ∙ 𝜕𝑔 𝜕ℎ ∙ 𝜕ℎ 𝜕𝑘 ∙ 𝜕𝑘 𝜕𝑙 ∙ 𝜕𝑙 𝜕𝑚 ∙ 𝜕𝑚 𝜕𝑛 ∙ 𝜕𝑛 𝜕𝑤 𝑥ℎ 1 𝜕𝑓 𝜕𝑔 = 𝜕𝑔2 𝜕𝑔 = 2𝑔 = 2 −2.99 = −5.98 𝑓 = (𝑔)2 -5.98 tanh tanh 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 1
- 70. Backward Pass * 0.078 1. 𝑊𝑥ℎ 𝑥0 𝑊ℎℎ 0.024 0.078 tanh 0.0778 * 0.00187 * 0.078 1. 𝑊𝑥ℎ 𝑥1 0.078 ℎ0 + 0.07987 ℎ1 0.07970 * 0.078 2. 𝑊𝑥ℎ 𝑥2 0.156 𝑊ℎℎ 0.024 * 0.0019 + 0.1579 0.1566 ℎ2 0.051𝑊ℎ𝑦 * 0.0080 𝑦 -3 + ** -2.99 8.95 𝐿 𝜕𝐿 𝜕𝑤 𝑥ℎ = 𝝏𝒇 𝝏𝒇 ∙ 𝝏𝒇 𝝏𝒈 ∙ 𝝏𝒈 𝝏𝒉 ∙ 𝜕ℎ 𝜕𝑘 ∙ 𝜕𝑘 𝜕𝑙 ∙ 𝜕𝑙 𝜕𝑚 ∙ 𝜕𝑚 𝜕𝑛 ∙ 𝜕𝑛 𝜕𝑤 𝑥ℎ 1-5.98 𝑔 = ℎ − 3 𝜕𝑔 𝜕ℎ = 1 -5.98 tanh tanh 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 1
- 71. Backward Pass * 0.078 1. 𝑊𝑥ℎ 𝑥0 𝑊ℎℎ 0.024 0.078 tanh 0.0778 * 0.00187 * 0.078 1. 𝑊𝑥ℎ 𝑥1 0.078 ℎ0 + 0.07987 ℎ1 0.07970 * 0.078 2. 𝑊𝑥ℎ 𝑥2 0.156 𝑊ℎℎ 0.024 * 0.0019 + 0.1579 0.1566 ℎ2 0.051𝑊ℎ𝑦 * 0.0080 𝑦 -3 + ** -2.99 8.95 𝐿 𝜕𝐿 𝜕𝑤 𝑥ℎ = 𝝏𝒇 𝝏𝒇 ∙ 𝝏𝒇 𝝏𝒈 ∙ 𝝏𝒈 𝝏𝒉 ∙ 𝝏𝒉 𝝏𝒌 ∙ 𝜕𝑘 𝜕𝑙 ∙ 𝜕𝑙 𝜕𝑚 ∙ 𝜕𝑚 𝜕𝑛 ∙ 𝜕𝑛 𝜕𝑤 𝑥ℎ 1-5.98 -5.98 ℎ = 𝑊ℎ𝑦 𝑘 𝜕ℎ 𝜕𝑘 = 𝑊ℎ𝑦 0.051tanh tanh 𝜕ℎ 𝜕𝑊ℎ𝑦 = 𝑘 0.1566 -0.304 0.936 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 1
- 72. Backward Pass * 0.078 1. 𝑊𝑥ℎ 𝑥0 𝑊ℎℎ 0.024 0.078 tanh 0.0778 * 0.00187 * 0.078 1. 𝑊𝑥ℎ 𝑥1 0.078 ℎ0 + 0.07987 ℎ1 0.07970 * 0.078 2. 𝑊𝑥ℎ 𝑥2 0.156 𝑊ℎℎ 0.024 * 0.0019 + 0.1579 0.1566 ℎ2 0.051𝑊ℎ𝑦 * 0.0080 𝑦 -3 + ** -2.99 8.95 𝐿 𝜕𝐿 𝜕𝑤 𝑥ℎ = 𝝏𝒇 𝝏𝒇 ∙ 𝝏𝒇 𝝏𝒈 ∙ 𝝏𝒈 𝝏𝒉 ∙ 𝝏𝒉 𝝏𝒌 ∙ 𝜕𝑘 𝜕𝑙 ∙ 𝜕𝑙 𝜕𝑚 ∙ 𝜕𝑚 𝜕𝑛 ∙ 𝜕𝑛 𝜕𝑤 𝑥ℎ 1-5.98 -5.98 ℎ = 𝑊ℎ𝑦 𝑘 𝜕ℎ 𝜕𝑘 = 𝑊ℎ𝑦 tanh tanh 𝜕ℎ 𝜕𝑊ℎ𝑦 = 𝑘 -0.304 0.936 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 1
- 73. Backward Pass * 0.078 1. 𝑊𝑥ℎ 𝑥0 𝑊ℎℎ 0.024 0.078 tanh 0.0778 * 0.00187 * 0.078 1. 𝑊𝑥ℎ 𝑥1 0.078 ℎ0 + 0.07987 ℎ1 0.07970 * 0.078 2. 𝑊𝑥ℎ 𝑥2 0.156 𝑊ℎℎ 0.024 * 0.0019 + 0.1579 0.1566 ℎ2 0.051𝑊ℎ𝑦 * 0.0080 𝑦 -3 + ** -2.99 8.95 𝐿 𝜕𝐿 𝜕𝑤 𝑥ℎ = 𝝏𝒇 𝝏𝒇 ∙ 𝝏𝒇 𝝏𝒈 ∙ 𝝏𝒈 𝝏𝒉 ∙ 𝝏𝒉 𝝏𝒌 ∙ 𝝏𝒌 𝝏𝒍 ∙ 𝜕𝑙 𝜕𝑚 ∙ 𝜕𝑚 𝜕𝑛 ∙ 𝜕𝑛 𝜕𝑤 𝑥ℎ 1-5.98 -5.98 𝑘 = tanh(𝑙) 𝜕𝑘 𝜕𝑙 = 1 − 𝑘2 = 1−.15662 = .975 -0.304-0.297 tanh tanh 0.936 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 1
- 74. Backward Pass * 0.078 1. 𝑊𝑥ℎ 𝑥0 𝑊ℎℎ 0.024 0.078 tanh 0.0778 * 0.00187 * 0.078 1. 𝑊𝑥ℎ 𝑥1 0.078 ℎ0 + 0.07987 ℎ1 0.07970 * 0.078 2. 𝑊𝑥ℎ 𝑥2 0.156 𝑊ℎℎ 0.024 * 0.0019 + 0.1579 0.1566 ℎ2 0.051𝑊ℎ𝑦 * 0.0080 𝑦 -3 + ** -2.99 8.95 𝐿 𝜕𝐿 𝜕𝑤 𝑥ℎ = 𝝏𝒇 𝝏𝒇 ∙ 𝝏𝒇 𝝏𝒈 ∙ 𝝏𝒈 𝝏𝒉 ∙ 𝝏𝒉 𝝏𝒌 ∙ 𝝏𝒌 𝝏𝒍 ∙ 𝝏𝒍 𝝏𝒎 ∙ 𝜕𝑚 𝜕𝑛 ∙ 𝜕𝑛 𝜕𝑤 𝑥ℎ 1-5.98 -5.98 -0.297 tanh tanh -0.297-0.0071 0.936 -0.304 -0.297 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 1
- 75. Backward Pass * 0.078 1. 𝑊𝑥ℎ 𝑥0 𝑊ℎℎ 0.024 0.078 tanh 0.0778 * 0.00187 * 0.078 1. 𝑊𝑥ℎ 𝑥1 0.078 ℎ0 + 0.07987 ℎ1 0.0797 * 0.078 2. 𝑊𝑥ℎ 𝑥2 0.156 𝑊ℎℎ 0.024 * 0.0019 + 0.1579 0.1566 ℎ2 0.051𝑊ℎ𝑦 * 0.0080 𝑦 -3 + ** -2.99 8.95 𝐿 𝜕𝐿 𝜕𝑤 𝑥ℎ = 𝝏𝒇 𝝏𝒇 ∙ 𝝏𝒇 𝝏𝒈 ∙ 𝝏𝒈 𝝏𝒉 ∙ 𝝏𝒉 𝝏𝒌 ∙ 𝝏𝒌 𝝏𝒍 ∙ 𝝏𝒍 𝝏𝒎 ∙ 𝝏𝒎 𝝏𝒏 ∙ 𝜕𝑛 𝜕𝑤 𝑥ℎ 1-5.98 -5.98 -0.297 tanh tanh -0.297-0.0071 1 − 𝑘2 = 1−.07972 = .993 -0.0071 0.936 -0.304 -0.297 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 1
- 76. Backward Pass * 0.078 1. 𝑊𝑥ℎ 𝑥0 𝑊ℎℎ 0.024 0.078 tanh 0.0778 * 0.00187 * 0.078 1. 𝑊𝑥ℎ 𝑥1 0.078 ℎ0 + 0.07987 ℎ1 0.0797 * 0.078 2. 𝑊𝑥ℎ 𝑥2 0.156 𝑊ℎℎ 0.024 * 0.0019 + 0.1579 0.1566 ℎ2 0.051𝑊ℎ𝑦 * 0.0080 𝑦 -3 + ** -2.99 8.95 𝐿 1-5.98 -5.98 -0.297 tanh tanh -0.297-0.0071-0.0071 -0.0071 -0.00017 0.936 -0.304 𝜕𝐿 𝜕𝑤 𝑥ℎ = 𝝏𝒇 𝝏𝒇 ∙ 𝝏𝒇 𝝏𝒈 ∙ 𝝏𝒈 𝝏𝒉 ∙ 𝝏𝒉 𝝏𝒌 ∙ 𝝏𝒌 𝝏𝒍 ∙ 𝝏𝒍 𝝏𝒎 ∙ 𝝏𝒎 𝝏𝒏 ∙ 𝜕𝑛 𝜕𝑤 𝑥ℎ -0.0005 -0.297 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 1
- 77. Backward Pass * 0.078 1. 𝑊𝑥ℎ 𝑥0 𝑊ℎℎ 0.024 0.078 tanh 0.0778 * 0.00187 * 0.078 1. 𝑊𝑥ℎ 𝑥1 0.078 ℎ0 + 0.07987 ℎ1 0.0797 * 0.078 2. 𝑊𝑥ℎ 𝑥2 0.156 𝑊ℎℎ 0.024 * 0.0019 + 0.1579 0.1566 ℎ2 0.051𝑊ℎ𝑦 * 0.0080 𝑦 -3 + ** -2.99 8.95 𝐿 1-5.98 -5.98 -0.297 tanh tanh -0.297-0.0071-0.0071 -0.0071 -0.00017 1 − 𝑘2 = 1−.07782 = .993 0.936 -0.304 𝜕𝐿 𝜕𝑤 𝑥ℎ = 𝝏𝒇 𝝏𝒇 ∙ 𝝏𝒇 𝝏𝒈 ∙ 𝝏𝒈 𝝏𝒉 ∙ 𝝏𝒉 𝝏𝒌 ∙ 𝝏𝒌 𝝏𝒍 ∙ 𝝏𝒍 𝝏𝒎 ∙ 𝝏𝒎 𝝏𝒏 ∙ 𝜕𝑛 𝜕𝑤 𝑥ℎ -0.00017 -0.0005 -0.297 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 1
- 78. Backward Pass * 0.078 1. 𝑊𝑥ℎ 𝑥0 𝑊ℎℎ 0.024 0.078 tanh 0.0778 * 0.00187 * 0.078 1. 𝑊𝑥ℎ 𝑥1 0.078 ℎ0 + 0.07987 ℎ1 0.0797 * 0.078 2. 𝑊𝑥ℎ 𝑥2 0.156 𝑊ℎℎ 0.024 * 0.0019 + 0.1579 0.1566 ℎ2 0.051𝑊ℎ𝑦 * 0.0080 𝑦 -3 + ** -2.99 8.95 𝐿 1-5.98 -5.98 -0.297 tanh tanh -0.297-0.0071-0.0071 -0.0071 -0.00017 0.936 -0.304 𝜕𝐿 𝜕𝑤 𝑥ℎ = 𝝏𝒇 𝝏𝒇 ∙ 𝝏𝒇 𝝏𝒈 ∙ 𝝏𝒈 𝝏𝒉 ∙ 𝝏𝒉 𝝏𝒌 ∙ 𝝏𝒌 𝝏𝒍 ∙ 𝝏𝒍 𝝏𝒎 ∙ 𝝏𝒎 𝝏𝒏 ∙ 𝝏𝒏 𝝏𝒘 𝒙𝒉 -0.00017 -0.00017 -0.0005 -0.297 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 1
- 79. Backward Pass * 0.078 1. 𝑊𝑥ℎ 𝑥0 𝑊ℎℎ 0.024 0.078 tanh 0.0778 * 0.00187 * 0.078 1. 𝑊𝑥ℎ 𝑥1 0.078 ℎ0 + 0.07987 ℎ1 0.0797 * 0.078 2. 𝑊𝑥ℎ 𝑥2 0.156 𝑊ℎℎ 0.024 * 0.0019 + 0.1579 0.1566 ℎ2 0.051𝑊ℎ𝑦 * 0.0080 𝑦 -3 + ** -2.99 8.95 𝐿 1-5.98 -5.98 -0.297 tanh tanh -0.297-0.0071-0.0071 -0.0071 -0.00017 0.936 -0.304 -0.00017 -0.00017 -0.0005 -0.297 𝑤 𝑎 ≔ 𝑤 𝑎 − 0.01 ∙ 𝜕𝐿 𝜕𝑤 𝑎 𝑤 𝑥ℎ ≔ 0.078 − 0.01 ∙ −.00017 = 0.0780017 𝑤ℎℎ ≔ 0.024 − 0.01 ∙ −.0005 = 0.024005 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 1
- 80. Backward Pass * 0.078 1. 𝑊𝑥ℎ 𝑥0 𝑊ℎℎ 0.024 0.078 tanh * * 0.078 1. 𝑊𝑥ℎ 𝑥1 0.078 ℎ0 + ℎ1 * 0.078 2. 𝑊𝑥ℎ 𝑥2 0.156 𝑊ℎℎ 0.024 * + 0.1579 0.051𝑊ℎ𝑦 * + ** 1-5.98 tanh tanh -0.297-0.0071 -0.0071 -0.00017 𝑥1𝑥0 ℎ1ℎ0 1 2 ℎ2 𝑥2 3 1
- 81. 𝜕𝐿 𝜕𝑥 = 𝑤ℎℎ … 𝑤ℎℎ … 𝑤ℎℎ … 𝑤ℎℎ = 𝑤ℎℎ 𝑛 ∙ 𝐶(𝑤) 𝑤ℎℎ𝑤ℎℎ𝑤ℎℎ 𝑤ℎℎ𝑤ℎℎ 1. 0.024 2. 0.000576 3. 1.382e-05 4. 3.318e-07 5. 7.963e-09 6. 1.911e-10 7. 4.586e-12 8. 1.101e-13 9. 2.642e-15 10. 6.340e-17 𝑊ℎℎ = 0.024 tanh tanhtanhtanhtanhtanh
- 83. W x 2n 4n 𝑖 𝑓 𝑜 𝑔 = 𝑠𝑖𝑔𝑚 𝑠𝑖𝑔𝑚 𝑠𝑖𝑔𝑚 𝑡𝑎𝑛ℎ 𝑊 𝑥 ℎ 𝑡−1 𝑐𝑡 = 𝑓 ∙ 𝑐𝑡−1 + 𝑖 ∙ 𝑔 ℎ 𝑡 = 𝑜 ∙ tanh(𝑐𝑡) i f o g x h Long Short-Term Memory (LSTM) n n n n 𝜎 𝜎 𝜎 𝜏 𝑡 − 1 𝑡 ℎ 𝑡 = (tanh) 𝑊 𝑥 ℎ 𝑡−1 - RNN
- 84. 𝑐𝑡 = 𝑓 ∙ 𝑐𝑡−1 + 𝑖 ∙ 𝑔 ℎ 𝑡 = tanh 𝑊ℎℎℎ 𝑡−1 + 𝑊𝑥ℎ 𝑥RNN: LSTM: 𝑖 𝑓 𝑜 𝑔 = 𝑠𝑖𝑔𝑚 𝑠𝑖𝑔𝑚 𝑠𝑖𝑔𝑚 𝑡𝑎𝑛ℎ 𝑊 𝑥 ℎ 𝑡−1 𝑐𝑡 = 𝑓 ∙ 𝑐𝑡−1 + 𝑖 ∙ 𝑔 ℎ 𝑡 = 𝑜 ∙ tanh(𝑐𝑡) forget gate, 0/1 input gate, 0/1
- 85. f incoming X i og + X tanh X Long Short-Term Memory (LSTM) 𝑖 𝑓 𝑜 𝑔 = 𝑠𝑖𝑔𝑚 𝑠𝑖𝑔𝑚 𝑠𝑖𝑔𝑚 𝑡𝑎𝑛ℎ 𝑊 𝑥 ℎ 𝑡−1 𝑐𝑡 = 𝑓 ∙ 𝑐𝑡−1 + 𝑖 ∙ 𝑔 ℎ 𝑡 = 𝑜 ∙ tanh(𝑐𝑡) 𝑐𝑡−1 ℎ 𝑡
- 86. 𝜕𝐿 𝜕𝑥 = 𝑤ℎℎ … 𝑤ℎℎ … 𝑤ℎℎ … 𝑤ℎℎ = 𝑤ℎℎ 𝑛 ∙ 𝐶(𝑤) 𝑤ℎℎ𝑤ℎℎ𝑤ℎℎ f f f f f f + + + RNN LSTM Flow of gradient 𝑡 − 1 𝑡 𝑡 + 1 𝑡 − 1 𝑡 𝑡 + 1
- 88. Long Short-Term Memory (LSTM) Source: https://colah.github.io/posts/2015-08-Understanding-LSTMs/
- 89. Reference 1. Long Term-Short Memory (Hochreiter, 1997), http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf 2. Learning Long Term Dependencies With Gradient Descent is Difficult (Yoshua Bengio, 1994), http://www.dsi.unifi.it/~paolo/ps/tnn-94-gradient.pdf 3. http://neuralnetworksanddeeplearning.com/chap5.html 4. Deep Learning, Ian Goodfellow et al., The MIT Press 5. Recurrent Neural Networks, LSTM, Andrej Karpathy, Stanford Lectures, https://www.youtube.com/watch?v=iX5V1WpxxkY Alex Kalinin alex@alexkalinin.com
Editor's Notes
- #5: First, we calculate new hidden state. We use both the previous hidden state and the input. Using the previous hidden states provides “memory”. Then, we use new hidden state to calculate new output, y. This is a forward pass. All operations are differentiable, so we can use vanilla back-propagation to train our network.
- #37: First, we calculate new hidden state. We use both the previous hidden state and the input. Using the previous hidden states provides “memory”. Then, we use new hidden state to calculate new output, y. This is a forward pass. All operations are differentiable, so we can use vanilla back-propagation to train our network.
- #43: We need to train our network. Calculating L is a forward pass. Updating weights is a back-propagation.
- #44: How to calculate the derivative of function.