Understanding of neural network architecture
- 1. Advanced Artificial Neural Networks Perceptron & Beyond Lecture 2 & 3 Dr. Tehseen Zia
- 3. Perceptron • Rosenblatt proposed a binary classification method • Key Idea • One weight per input • Multiple weights with respective inputs and add bias • If result larger than threshold, return 1, otherwise 0
- 4. Example Problem • Will I pass this course? Let’s start with a simple two feature model
- 5. Example Problem 𝑥1 𝑥2 Σ 𝑤1 𝑤2 • Will I pass this course? Let’s start with a simple two feature model
- 6. Example Problem 𝑥1 𝑥2 Σ 𝑤1=1 𝑤2=1 • Will I pass this course? Let’s start with a simple two feature model 𝜏=10 𝑦
- 7. Example Problem 𝑥1 𝑥2 Σ 𝑤1=1 𝑤2=1 • Will I pass this course? Let’s start with a simple two feature model 𝜏=10 𝑦 What variables of the model are known and not known ?
- 8. Example Problem 𝑥1 𝑥2 Σ 𝑤1=1 𝑤2=1 • Will I pass this course? Let’s start with a simple two feature model 𝜏=10 𝑦 What variables of the model are known and not known ? X X X
- 9. Example Problem 𝑥1 𝑥2 Σ 𝑤1=1 𝑤2=1 • Will I pass this course? Let’s start with a simple two feature model 𝜏=10 𝑦 What variables of the model are known and not known ? X X X Learning objective: To find values of the variables ()
- 10. Example Problem 𝑥1 𝑥2 Σ 𝑤1=1 𝑤2=1 • Will I pass this course? Let’s start with a simple two feature model 𝜏=0 𝑦 𝑥0=1 𝑤0=−10 How to fix threshold?
- 11. Example Problem 𝑥1 =4 𝑥2=5 Σ 𝑤1=1 𝑤2=1 • Will I pass this course? Let’s start with a simple two feature model 𝜏=10 =0
- 12. Example Problem 𝑥1 =4 =5 Σ 𝑤1=1 𝑤2=1 • Will I pass this course? Let’s start with a simple two feature model 𝜏=0 =0 𝑥0=1 𝑏¿−10 How to fix threshold?
- 13. Example Problem • How to implement the model
- 14. Training a Perceptron • Learning algorithm: • Initialize weights randomly • Take one sample and predict • For wrong predictions, update weights • If the output was = 1, increase weights • If the output was = 0, decrease weights • Repeat until no errors are made
- 15. Training a Perceptron • Learning algorithm: • Initialize weights randomly • Take one sample and predict • For wrong predictions, update weights • If the output was = 1, increase weights • If the output was = 0, decrease weights • Repeat until no errors are made But, how do the algorithm knew that the predictions are wrong?
- 16. Training a Perceptron • Learning algorithm: • Initialize weights randomly • Take one sample and predict • For wrong predictions, update weights • If the output was = 1, increase weights • If the output was = 0, decrease weights • Repeat until no errors are made But, how do the algorithm knew that the predictions are wrong? Cost function/ loss function tell it
- 17. Training a Perceptron • Learning algorithm: • Initialize weights randomly • Take one sample and predict • For wrong predictions, update weights • If the output was = 1, increase weights • If the output was = 0, decrease weights • Repeat until no errors are made What is Cost function? 𝐿(𝑊 ,𝑏)=∑ 𝑖=1 𝑚 (𝑦− ^ 𝑦) 2
- 18. Training a Perceptron • Learning algorithm: • Initialize weights randomly • Take one sample and predict • For wrong predictions, update weights • If the output was = 1, increase weights • If the output was = 0, decrease weights • Repeat until no errors are made What is Cost function? 𝐿(𝑊 ,𝑏)=∑ 𝑖=1 𝑚 (𝑦− ^ 𝑦) 2 Actual class Predicted class
- 19. Learning Problem • Problem: • Loss function: • Objective • Minimize with respect to • Algorithm: • Start with some • Keep changing to reduce • Until we (hopefully) end-up with minimum
- 20. Training a Perceptron • Learning algorithm: • Initialize weights randomly • Take one sample and predict • For wrong predictions, update weights • If the output was = 1, increase weights • If the output was = 0, decrease weights • Repeat until no errors are made How much weights to increase or decrease ?
- 21. Training a Perceptron • Learning algorithm: • Initialize weights randomly • Take one sample and predict • For wrong predictions, update weights • If the output was = 1, increase weights • If the output was = 0, decrease weights • Repeat until no errors are made Gradient Decent Algorithm
- 22. Hill-Descent 𝐿 ( 𝑊 ) 𝑊 “Climbing down from Everest in thick fog with amnesia" 𝑥 Σ 𝑤 𝜏 𝑦 𝐿(𝑊 )
- 23. Hill-Descent 𝐿 ( 𝑊 ) 𝑊 Question 1: What we need to take a step downwards ?
- 24. Hill-Descent 𝐿 ( 𝑊 ) 𝑊 Question 1: What we need to take a step downwards ?: Direction and step size
- 25. Hill-Descent 𝐿 ( 𝑊 ) 𝑊 Question 1: What we need to take a step downwards ?: Direction and step size
- 26. Hill-Descent 𝐿 ( 𝑊 ) 𝑊 Question 2: How do we find direction of decreasing ?
- 27. From Hill-Descent to Gradient-Descent 𝐿 ( 𝑊 ) 𝑊 Question 2: How do we find direction of decreasing ?:
- 28. From Hill-Descent to Gradient-Descent 𝐿 ( 𝑊 ) 𝑊 Knowing direction is not enough, we take a step towards the direction: 𝜕(𝐿 (𝑊 )) 𝜕𝑊 α
- 29. From Hill-Descent to Gradient-Descent 𝐿 ( 𝑊 ) 𝑊 Knowing direction is not enough, we take a step towards the direction: - α
- 31. Gradient-Descent Algorithm: Why It Works ? 𝐿(𝑤¿¿ 𝑗)¿ 𝑤 𝑗 𝐿(𝑤 𝑗 ) 𝑤 𝑗 - -
- 32. Gradient-Descent Algorithm: Why It Works ? 𝐿(𝑤¿¿ 𝑗)¿ 𝑤 𝑗 𝐿(𝑤 𝑗 ) 𝑤 𝑗 - - - - 𝑤 𝑗 𝑛𝑒𝑤 <𝑤 𝑗 𝑜𝑙𝑑 𝑤 𝑗 𝑛𝑒𝑤 >𝑤 𝑗 𝑜𝑙𝑑
- 33. Gradient-Descent Algorithm: Why It Works ? 𝐿(𝑤¿¿ 𝑗)¿ 𝑤 𝑗 - 𝑤 𝑗 𝑛𝑒𝑤 =𝑤 𝑗 𝑜𝑙𝑑 Why ?
- 34. Gradient Descent Algorithm: Semantics of Learning Rate 𝐿(𝑤¿¿ 𝑗)¿ 𝑤 𝑗 𝐿(𝑤¿¿ 𝑗)¿ 𝑤 𝑗 - -
- 35. Gradient Descent Algorithm: Semantics of Learning Rate 𝐿(𝑤¿¿ 𝑗)¿ 𝑤 𝑗 𝐿(𝑤¿¿ 𝑗)¿ 𝑤 𝑗 - - If is small gradient descent convergence is slow If is large gradient descent convergence fast but it can overshoot the minimum or diverge
- 36. Convergence 𝐿(𝑤¿¿ 𝑗)¿ 𝑤 𝑗 𝜕 𝐿(𝑊 ) 𝜕𝑤 𝜕 𝐿(𝑊 ) 𝜕𝑤 𝜕 𝐿(𝑊 ) 𝜕𝑤 𝜕 𝐿(𝑊 ) 𝜕𝑤 𝜕 𝐿(𝑊 ) 𝜕𝑤 𝜕 𝐿(𝑊 ) 𝜕𝑤
- 37. Convergence 𝐿(𝑤¿¿ 𝑗)¿ 𝑤 𝑗 𝜕 𝐿(𝑊 ) 𝜕𝑤 𝜕 𝐿(𝑊 ) 𝜕𝑤 𝜕 𝐿(𝑊 ) 𝜕𝑤 𝜕 𝐿(𝑊 ) 𝜕𝑤 𝜕 𝐿(𝑊 ) 𝜕𝑤 𝜕 𝐿(𝑊 ) 𝜕𝑤 As it approaches to local minimum, gradient descent automatically takes smaller steps
- 38. Derivation of • Loss function: • = = …
- 40. Perceptron Decision Boundary: Linear ∑ 𝑥1 𝑤1 𝑥1 ^ 𝑦 𝑤1 =0.5 ^ 𝑦 Single input neuron
- 41. Linearly and Nonlinearly Separable Problem
- 42. Implementing AND with Perceptron ∑=𝑤 1𝑥1+𝑤 2𝑥2 𝑤 1=1 1 𝑥 1 𝑥 2 ^ 𝑦 ¿ 2 = 1 = 0
- 43. Implementing OR with Perceptron ∑=𝑤 1𝑥1+𝑤 2𝑥2 𝑤 1=1 1 𝑥 1 𝑥 2 ^ 𝑦 ¿ 1 = 1 = 0
- 44. Implementing XOR with Perceptron
- 45. Implementing XOR with Perceptron
- 46. Implementing XOR with Perceptron We cannot because XOR is nonlinearly separable
- 47. Implementing XOR with Multi-Perceptron Perceptron # 2 Perceptron # 1
- 48. Nonlinearly Separable Problems and Multi- perceptron Perceptron # 2 Perceptron # 1 Decision rule: if ∑ of P1 < 0 -> black elseif ∑ of P2 > 0 -> black else white
- 49. Multi-perceptron Architecture ∑ 𝑥1 𝑤11 ^ 𝑦 ∑ 𝑤22 𝑥2 ∑ 𝑤12 𝑤21 𝑤11 𝑤21 Perceptron # 1 Perceptron # 2 Perceptron # 3
- 50. Multi-perceptron Architecture ∑ 𝑥1 𝑤11 ^ 𝑦 ∑ 𝑤22 𝑥2 ∑ 𝑤12 𝑤21 𝑤11 𝑤21 Perceptron # 1 Perceptron # 2 Perceptron # 3
- 51. Nonlinearity is Important • Perceptron 1: • Perceptron 2: • Perceptron 3:
- 52. Nonlinearity is Important • Perceptron 1: • Perceptron 2: • Perceptron 3: + + +
- 53. Nonlinearity is Important • Perceptron 1: • Perceptron 2: • Perceptron 3: + + + +
- 54. Nonlinearity is Important • Perceptron 1: • Perceptron 2: • Perceptron 3: + + + + Linear function
- 55. Computational Neuron 𝑧=𝑤 1𝑥 1+𝑤 2𝑥 2 𝑤 1 𝑤 2 𝑥 1 𝑥 2 𝑎=𝜎(𝑧) 𝑎= ^ 𝑦 =
- 58. Decision Boundary 𝑧=𝑤 0+𝑤 1 𝑥 1+𝑤 2 𝑥 2 = () = 0 = -3 𝑤 1=1𝑤 2=1 Suppose
- 59. Decision Boundary 𝑧=𝑤 0+𝑤 1 𝑥 1+𝑤 2 𝑥 2 = () = 0 = -3 𝑤 1=1𝑤 2=1 Suppose
- 60. Learning Task • Dataset : • Input: • Classifier • Where • How to choose parameters ? = 𝑧=𝑤 1 𝑥1 +𝑤 2 𝑥2 +…+𝑤 𝑛 𝑥𝑛
- 61. Cost Function
- 62. Cost Function Mean square loss function =
- 63. Cost Function ) Mean square loss function =
- 64. Cost Function Mean square loss function
- 65. Cost Function Mean square loss function
- 66. Cost Function Mean square loss function
- 67. Cost Function Cost = 0 if y = 1, a = 1 But as -> 0 Cost -> ∞ Mean square loss function
- 68. Cost Function Cost = 0 if y = 0, a = 0 But as -> 1 Cost -> ∞ Mean square loss function
- 69. Cost Function Mean square loss function 𝐽(𝑊 )=− 1 𝑚 ∑ 𝑖=1 𝑚 ylog(^ 𝑦)+(1− y)log(1− ^ 𝑦)
- 70. Cost Function Mean square loss function Cross-entropy loss function 𝐽(𝑊 )=− 1 𝑚 ∑ 𝑖=1 𝑚 ylog(^ 𝑦)+(1− y)log(1− ^ 𝑦)
- 71. Learning Parameters: Gradient Descent Algorithm