Introduction to neural networks
0 likes124 views
The document discusses neural networks and their architectures. It describes the basic components of a neural network including perceptrons, activation functions, forward and backward propagation. It provides an example of using a neural network to learn housing prices. The network has 3 inputs (number of bedrooms, bathrooms, ground floor indicator), 2 hidden layers, and 1 output (price). It goes through the steps of forward propagation, calculation of error, then backward propagation to update the weights to minimize the error through gradient descent.
![Learning the price of a flat in Al weibdeh
• Description:
• Ground Floor? Yes
• 2 bathrooms
• 3 bedrooms
• The price is 90 K JoD.
[1,2,3]-> -> 90NN](https://image.slidesharecdn.com/neuralnetworksaipresentation-180608124433/85/Introduction-to-neural-networks-21-320.jpg)
![Inception LeNet (GoogLe Network)*
The name actually comes from the
movie Inception
*Going deeper with convolutions
[Szegedy 2014]](https://image.slidesharecdn.com/neuralnetworksaipresentation-180608124433/85/Introduction-to-neural-networks-39-320.jpg)
Introduction to neural networks
- 2. 𝑁𝑒𝑢𝑟𝑎𝑙 𝑁𝑒𝑡𝑤𝑜𝑟𝑘𝑠 ∈ 𝑆𝑢𝑝𝑒𝑟𝑣𝑖𝑠𝑒𝑑 𝑀𝑎𝑐ℎ𝑖𝑛𝑒 𝑙𝑒𝑎𝑟𝑛𝑖𝑛𝑔
- 3. History
- 4. Neural Networks Architectures Standard Artificial Neural Networks ANN Convolutional Neural Networks CNN Recurrent Neural Networks RNN
- 5. Perceptron z 𝑧 = 𝑖=0 𝑛 𝑤𝑖 𝑥𝑖 = 𝑊 ∗ 𝑋 • z = w0*1 + w1*x1+ w2*x2 + wn*xn
- 6. Perceptron z 𝑎 = 𝑔(𝑧)g(z) a
- 7. Activation functions and non linearity Tanh(z) Relu(z)σ(z)
- 9. . Relu(z) . Relu(z) 1 2 3 0.5 0.4 0.5 0.4 0.4 0.5 𝑧1 1 = 0.5 ∗ 1 + 0.5 ∗ 2 + 0.5 ∗ 3 = 3 3 2.4 2.4 𝑧2 1 = 0.4 ∗ 1 + 0.4 ∗ 2 + 0.4 ∗ 3 = 2.4 𝑎2 1 = Relu(2.4)=2.4 𝑎1 1 = Relu(3)=3 3
- 10. . Relu(z) . Relu(z) 1 2 3 0.5 0.4 0.5 0.4 0.4 0.5 3 2.4 3 2.4 Relu(z). 2.5 2.5 𝑦 0.2 0.8 𝑧1 2 = 0.2 ∗ 3 + 0.8 ∗ 2.4 = 2.52 𝑎1 2 = Relu(2.52)=2.52
- 11. Forward propagation 𝑍 = 𝑊 ∗ 𝑋 𝐴 = 𝑔(𝑍) 𝐴 𝑖𝑠 𝑡ℎ𝑒 𝑛𝑒𝑥𝑡 𝑙𝑎𝑦𝑒𝑟′ 𝑠 input 𝑍 = 𝑊 ∗ 𝐴
- 13. Learning problem • 𝑇ℎ𝑒 𝑜𝑏𝑡𝑎𝑖𝑛𝑒𝑑 𝑜𝑢𝑡𝑝𝑢𝑡 𝑖𝑠 ′𝑎′ • 𝑡ℎ𝑒 𝑐𝑜𝑟𝑟𝑒𝑐𝑡 𝑜𝑢𝑡𝑝𝑢𝑡 𝑖𝑠 ′𝑦′ • 𝑇ℎ𝑒 𝑒𝑟𝑟𝑜𝑟 ‘𝐽’ 𝑖𝑠 𝑎 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛 𝑜𝑓 ‘𝑎’ 𝑎𝑛𝑑 ‘𝑦’ 𝑓𝑜𝑟 𝑒𝑥𝑎𝑚𝑝𝑙𝑒 J(a,y) = (𝑎 − 𝑦)2 • How to minimise the error? change w • How to find w?
- 14. Optimisation: Gradient Descent? • w = w- α 𝑑𝐽 𝑑𝑤 Problem statement Error in output (J), how to change W?
- 15. How to find 𝜕𝐽 𝜕𝑤 ? Back propagation? 𝜕𝑎 𝜕𝑧 = 𝑔′ 𝑧 𝜕𝑧 𝜕𝑤 = 𝑥 𝜕𝐽 𝜕𝑤 = 𝜕𝐽 𝜕𝑎 𝜕𝑎 𝜕𝑧 𝜕𝑧 𝜕𝑤 w z a J 𝜕𝐽 𝜕𝑤 = 𝜕𝐽 𝜕𝑎 ∗ 𝑔′ 𝑧 ∗ 𝑥 x Chain Rule: g'()
- 16. How to find 𝜕𝐽 𝜕𝑥 ? Back propagation? 𝜕𝑎 𝜕𝑧 = 𝑔′ 𝑧 𝜕𝑧 𝜕𝑥 = 𝑤 𝜕𝐽 𝜕𝑥 = 𝜕𝐽 𝜕𝑎 𝜕𝑎 𝜕𝑧 𝜕𝑧 𝜕𝑥 x z a J 𝜕𝐽 𝜕𝑥 = 𝜕𝐽 𝜕𝑎 ∗ 𝑔′ 𝑧 ∗ 𝑤 w Chain Rule: g'()
- 17. Update parameters • 𝑤 𝑛𝑒𝑤 = w- α 𝑑𝐽 𝑑𝑤 • Assume α=0.01 • 𝜕𝐽 𝜕𝑤 = 𝜕𝐽 𝜕𝑎 𝜕𝑎 𝜕𝑧 𝑥 = -88.24*𝑥 • w1_new = 0.1- 0.01*(-88.24)*1= 0.1+0.9=1 • w2_new = 0.2- 0.01*(-88.24)*2.5= 2.4 • w3_new = 0.4- 0.01*(-88.24)*2.9= 2.9 0.1 0.2 1.76 0.4 1.76 Obtained Output 2.5 2.9 2.5 2.9 1.76 1 L2 Correct value is 90 Obtained value is 1.76 Error = - 88.24
- 18. Distribute the penalty to previous neurons • 𝜕𝐽 𝜕𝑥 = 𝜕𝐽 𝜕𝑎 𝜕𝑎 𝜕𝑧 𝑤 = -88.24*w • 𝜕𝐽 𝜕𝑥2 = -88.24*0.2= -17.6 • 𝜕𝐽 𝜕𝑥3 = -88.24*0.4= -35.3 𝛛𝐉 𝛛𝐱𝟐 = -17.6 𝝏𝑱 𝝏𝒙𝟑 = -35.3 Error = -88. 0.1 0.2 1.76 0.4 1.76 Obtained Output 2.5 2.9 2.5 2.9 1.76 1 L2 𝛛𝐉 𝛛𝒂 = -88.24
- 19. Feed Forward Z= W*X a= g(z) J= Cost function Feed backward dJ/da = (y-a) or calculated da/dz = g'(z) dJ/dz = (dJ/da)*(da/dz) dJ/dx = (dJ/dz)*W dJ/dw = (dJ/dz)*X dJ/dx dz/dx da/dz dJ/da wi g'(z) … w11 w12 dJ/da1 w21 w22 w31 dJ/da2 w32 z2 a2 x2 x3 x1 z1 a1 dz/dw x Summary
- 20. How about other search and optimisation methods? Forward propagation calculate error Back propagation Update parameters
- 21. Learning the price of a flat in Al weibdeh • Description: • Ground Floor? Yes • 2 bathrooms • 3 bedrooms • The price is 90 K JoD. [1,2,3]-> -> 90NN
- 22. ANN: 3 inputs, 1 output, 2 hidden Layers 0.1 0.0 0.5 0.0 0.4 0.2 0.3 0.5 0.8 0.4 0.8 -0.1 0.5 0.2 0.4 0.6 3 3.0 2.4 2.5 2.9 2.5 2.9 3.0 2.4 0.6 1 1 1 2 L2L1
- 23. 0.1 0.0 0.5 0.0 0.4 0.2 0.2 0.5 0.8 0.4 0.8 0.4 0.5 0.2 0.4 1.76 3 3.0 2.4 2.5 2.9 2.5 2.9 3.0 2.4 1.76 1 1 1 2 Initialisation Relu Relu Relu Relu Relu
- 24. . Relu(z) . Relu(z) 1 2 3 0.5 0.4 0.5 0.4 0.4 0.5 𝑧1 1 = 0.5 ∗ 1 + 0.5 ∗ 2 + 0.5 ∗ 3 = 3 3 2.4 3 2.4 𝑧2 1 = 0.4 ∗ 1 + 0.4 ∗ 2 + 0.4 ∗ 3 = 2.4 𝑎2 1 = Relu(2.4)=2.4 𝑎1 1 = Relu(3)=3
- 25. Matrix multiplication • 𝑍 = 𝑊 ∗ 𝑋 • 𝑧1 𝑧2 = 𝑤11 𝑤21 𝑤12 𝑤22 𝑤31 𝑤32 𝑥1 𝑥2 𝑥3 • 𝑧1 𝑧2 = 0.5 0.5 0.4 0.4 0.5 0.4 1 2 3 = 0.5 ∗ 1 + 0.5 ∗ 2 + 0.4 ∗ 1 + 0.4 ∗ 2 + 0.5 ∗ 3 0.4 ∗ 3 = 3 2.4 • 𝑎 = 𝑅𝑒𝑙𝑢 𝑍 = 𝑟𝑒𝑙𝑢 3 2.4 = 3 2.4
- 26. 0.1 0.0 0.5 0.0 0.4 0.2 0.2 0.5 0.8 0.4 0.8 0.4 0.5 0.2 0.4 1.76 3 3.0 2.4 2.5 2.9 2.5 2.9 3.0 2.4 1.76 1 1 1 2 L2L1 Relu Relu Relu
- 27. Second Layer and Output Layer • Second Layer • 𝑍 = 𝑊 ∗ 𝑋 • 𝑧1 𝑧2 = 0 0.2 0 0.8 0.8 0.2 1 3 2.4 = 0 ∗ 1 + 0.2 ∗ 3 + 0 ∗ 1 + 0.8 ∗ 3 + 0.8 ∗ 2.4 0.2 ∗ 2.4 = 2.52 2.88 • 𝑎 = 𝑅𝑒𝑙𝑢 2.52 2.88 = 2.52 2.88 • Output Layer • 𝑍 = 0.1 0.2 0.4 1 2.52 2.88 = 0.1 ∗ 1 + 0.2 ∗ 2.52 0.4 ∗ 2.88 = 1.76 • 𝑦 = 𝑎 = 𝑅𝑒𝑙𝑢 𝑍 = 𝑅𝑒𝑙𝑢 1.76 =1.76
- 28. 0.1 0.0 0.5 0.0 0.4 0.2 0.2 1.76 0.5 0.8 0.4 0.8 0.4 0.5 0.2 0.4 1.76 Obtained Output 3 3.0 2.4 2.5 2.9 2.5 2.9 3.0 2.4 1.76 1 1 1 2 L2L1 • 𝑜𝑏𝑡𝑎𝑖𝑛𝑒𝑑 𝑣𝑎𝑙𝑢𝑒 𝑖𝑠 1.76, correct value is 90! • Error = 1.76-90 = - 88.24
- 29. The other way around: BackProp • 𝐶𝑜𝑠𝑡 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛 𝐽 = (a− y) 2 2 • Penalty: • 𝜕𝐽 𝜕𝑎 = 𝑎 − 𝑦 = 1.76−90 = − 88.24 x z a J w
- 30. Calculate new output parameters • 𝑤 𝑛𝑒𝑤 = w- α 𝑑𝐽 𝑑𝑤 • Assume α=0.01 • 𝜕𝐽 𝜕𝑤 = 𝜕𝐽 𝜕𝑎 𝜕𝑎 𝜕𝑧 𝑥 = 𝜕𝐽 𝜕𝑧 𝑥 = -88.24*x 𝑤1_𝑛𝑒𝑤 𝑤2_𝑛𝑒𝑤 𝑤3_𝑛𝑒𝑤 = …slide 18 ……= 1 2.4 2.9 0.1 0.2 1.76 0.4 1.76 Obtained Output 2.5 2.9 2.5 2.9 1.76 1 L2 Correct value is 90 Error = - 88.24
- 31. Distribute the penalty to L2 neurons • 𝜕𝐽 𝜕𝑥 = 𝜕𝐽 𝜕𝑎 𝜕𝑎 𝜕𝑧 𝑤 = 𝜕𝐽 𝜕𝑧 𝑤 = -88.24*w • 𝜕𝐽 𝜕𝑥2 = -88.24*0.2= -17.6 • 𝜕𝐽 𝜕𝑥3 = -88.24*0.4= -35.3 𝛛𝐉 𝛛𝐱𝟐 = -17.6 𝝏𝑱 𝝏𝒙𝟑 = -35.3 Error = -88.24 0.1 0.2 1.76 0.4 1.76 Obtained Output 2.5 2.9 2.5 2.9 1.76 1 L2
- 32. Calculate L2 parameters • 𝑤 𝑛𝑒𝑤 = w− α 𝑑𝐽 𝑑𝑧 X • Weights connected to upper neuron • Weights connected to lower neuron 0.0 0.0 0.2 0.8 0.8 0.2 3.0 2.4 2.5 2.9 2.5 2.9 3.0 2.4 1 1 L2L1 𝛛𝐉 𝛛𝐚𝟐 = -17.6 𝛛𝐉 𝛛𝐚𝟑 = -35.3 𝑤11_𝑛𝑒𝑤 𝑤21_𝑛𝑒𝑤 𝑤31_𝑛𝑒𝑤 = 𝑤11 𝑤21 𝑤31 - α * 𝑑𝐽 𝑑𝑧 * 𝑥1 𝑥2 𝑥3 = 0 0.2 0.8 - 0.01*(−17.6) * 1 3 2.4 = 0.2 0.7 1.2 𝑤12_𝑛𝑒𝑤 𝑤22_𝑛𝑒𝑤 𝑤32_𝑛𝑒𝑤 = 𝑤12 𝑤22 𝑤32 - α * 𝑑𝐽 𝑑𝑧 * 𝑥1 𝑥2 𝑥3 = 0 0.8 0.2 - 0.01*(35.3) * 1 3 2.4 = 0.4 1.9 1.0
- 33. Distribute the penalty to L1 neurons • 𝜕𝐽 𝜕𝑥 = 𝜕𝐽 𝜕𝑎 𝜕𝑎 𝜕𝑧 𝑤 = 𝜕𝐽 𝜕𝑧 𝑤 • Which one should I take?! 𝛛𝐉 𝛛𝐚𝟐 = -17.6 𝝏𝑱 𝝏𝒂𝟑 = -35.3 0.0 0.0 0.2 0.8 0.8 0.2 3.0 2.4 2.5 2.9 2.5 2.9 3.0 2.4 1 1 L2L1
- 34. Distribute the penalty to L1 neurons • 𝜕𝐽 𝜕𝑥 = 𝜕𝐽 𝜕𝑎 𝜕𝑎 𝜕𝑧 𝑤 = 𝜕𝐽 𝜕𝑧 𝑤 • 𝜕𝐽 𝜕𝑥2 = -17.6*0.2 + -35.3*0.8 = = -31.8 • 𝜕𝐽 𝜕𝑥3 = -17.6*0.8+ -35.3*0.2 = 8.9 = -21.2 𝛛𝐉 𝛛𝐚𝟐 = -17.6 𝝏𝑱 𝝏𝒂𝟑 = -35.3 0.0 0.0 0.2 0.8 0.8 0.2 3.0 2.4 2.5 2.9 2.5 2.9 3.0 2.4 1 1 L2L1 𝛛𝐉 𝛛𝐱𝟐 = -31.8 𝛛𝐉 𝛛𝐱𝟐 = -21.2
- 35. Calculate new input parameters • 𝑤 𝑛𝑒𝑤 = w− α 𝑑𝐽 𝑑𝑧 X • Weights connected to upper neuron • Weights connected to lower neuron 𝛛𝐉 𝛛𝐚𝟐 = -31.8 𝛛𝐉 𝛛𝐚𝟑 = -21.2 𝑤11_𝑛𝑒𝑤 𝑤21_𝑛𝑒𝑤 𝑤31_𝑛𝑒𝑤 = 𝑤11 𝑤21 𝑤31 - α * 𝑑𝐽 𝑑𝑧 * 𝑥1 𝑥2 𝑥3 = 0.5 0.5 0.5 - 0.01*(−31.8) * 1 2 3 = 0.8 1.1 1.4 𝑤12_𝑛𝑒𝑤 𝑤22_𝑛𝑒𝑤 𝑤32_𝑛𝑒𝑤 = 𝑤12 𝑤22 𝑤32 - α * 𝑑𝐽 𝑑𝑧 * 𝑥1 𝑥2 𝑥3 = 0.4 0.4 0.4 - 0.01*(−21.2) * 1 2 3 = 0.6 0.8 1.0 0.5 0.4 0.5 0.4 0.5 0.4 3 3.0 2.4 3.0 2.4 1 1 2 L1
- 36. 1.0 0.2 0.8 0.4 0.6 0.7 2.4 86.60 1.1 1.9 0.8 1.2 2.9 1.4 1.0 1.0 2 5.4 5.4 19.3 19.3 3 7.1 7.1 12.0 12.0 86.60 86.60 L1 L2 1 1 1 Obtained Output Update parameters Output ? 86.6, Error -3.4
- 38. Neural networks and Deep Learning
- 39. Inception LeNet (GoogLe Network)* The name actually comes from the movie Inception *Going deeper with convolutions [Szegedy 2014]
- 40. Neural Networks can generate Music! • 30 seconds of Jazz generated by an RNN. • https://soundcloud.com/user-559668657/machine-generated-jazz • Do you like it?