introduction to neural networks ANN deep
- 1. Introduction to Neural Networks John Paxton Montana State University Summer 2003
- 2. Chapter 2: Simple Neural Networks for Pattern Classification x0 x1 y xn 1 w0 w1 wn w0 is the bias f(yin) = 1 if yin >= 0 f(yin) = 0 otherwise ARCHITECTURE
- 3. Representations • Binary: 0 no, 1 yes • Bipolar: -1 no, 0 unknown, 1 yes • Bipolar is superior
- 4. Interpreting the Weights • w0 = -1, w1 = 1, w2 = 1 • 0 = -1 + x1 + x2 or x2 = 1 – x1 decision boundary x1 x2 YES NO
- 5. Modelling a Simple Problem • Should I attend this lecture? • x1 = it’s hot • x2 = it’s raining x0 x1 y x2 2.5 -2 1
- 6. Linear Separability AND OR XOR 1 0 0 0 0 0 0 1 1 1 1 1
- 7. Hebb’s Rule • 1949. Increase the weight between two neurons that are both “on”. • 1988. Increase the weight between two neurons that are both “off”. • wi(new) = wi(old) + xi*y
- 8. Algorithm 1. set wi = 0 for 0 <= i <= n 2. for each training vector 3. set xi = si for all input units 4. set y = t 5. wi(new) = wi(old) + xi*y
- 9. Example: 2 input AND s0 s1 s2 t 1 1 1 1 1 1 -1 -1 1 -1 1 -1 1 -1 -1 -1
- 10. Training Procedure w0 w1 w2 x0 x1 x2 y 0 0 0 1 1 1 1 1 1 1 1 1 -1 -1 (!) 0 0 2 1 -1 1 -1 (!) -1 1 1 1 -1 -1 -1 -2 2 2
- 11. Result Interpretation • -2 + 2x1 + 2x2 = 0 OR • x2 = -x1 + 1 • This training procedure is order dependent and not guaranteed.
- 12. Pattern Recognition Exercise • #.# .#. .#. #.# #.# .#. “X” “O”
- 13. Pattern Recognition Exercise • Architecture? • Weights? • Are the original patterns classified correctly? • Are the original patterns with 1 piece of wrong data classified correctly? • Are the original patterns with 1 piece of missing data classified correctly?
- 14. Perceptrons (1958) • Very important early neural network • Guaranteed training procedure under certain circumstances x0 x1 y xn 1 w0 w1 wn
- 15. Activation Function • f(yin) = 1 if yin > f(yin) = 0 if - <= yin <= f(yin) = -1 otherwise • Graph interpretation 1 -1
- 16. Learning Rule • wi(new) = wi(old) + *t*xi if error • is the learning rate • Typically, 0 < <= 1
- 17. Algorithm 1. set wi = 0 for 0 <= i <= n (can be random) 2. for each training exemplar do 3. xi = si 4. yin = xi*wi 5. y = f(yin) 6. wi(new) = wi(old) + *t*xi if error 7. if stopping condition not reached, go to 2
- 18. Example: AND concept • bipolar inputs • bipolar target • = 0 • = 1
- 19. Epoch 1 w0 w1 w2 x0 x1 x2 y t 0 0 0 1 1 1 0 1 1 1 1 1 1 -1 1 -1 0 0 2 1 -1 1 1 -1 -1 1 1 1 -1 -1 -1 -1
- 20. Exercise • Continue the above example until the learning algorithm is finished.
- 21. Perceptron Learning Rule Convergence Theorem • If a weight vector exists that correctly classifies all of the training examples, then the perceptron learning rule will converge to some weight vector that gives the correct response for all training patterns. This will happen in a finite number of steps.
- 22. Exercise • Show perceptron weights for the 2-of-3 concept x1 x2 x3 y 1 1 1 1 1 1 -1 1 1 -1 1 1 1 -1 -1 -1 -1 1 1 1 -1 1 -1 -1 -1 -1 1 -1 -1 -1 -1 -1
- 23. Adaline (Widrow, Huff 1960) • Adaptive Linear Network • Learning rule minimizes the mean squared error • Learns on all examples, not just ones with errors
- 25. Training Algorithm 1. set wi (small random values typical) 2. set (0.1 typical) 3. for each training exemplar do 4. xi = si 5. yin = xi*wi 6. wi(new) = wi(old) + *(t – yin)*xi 7. go to 3 if largest weight change big enough
- 26. Activation Function • f(yin) = 1 if yin >= 0 • f(yin) = -1 otherwise
- 27. Delta Rule • squared error E = (t – yin)2 • minimize error E’ = -2(t – yin)xi = (t – yin)xi
- 28. Example: AND concept • bipolar inputs • bipolar targets • w0 = -0.5, w1 = 0.5, w2 = 0.5 • minimizes E x0 x1 x2 yin t E 1 1 1 .5 1 .25 1 1 -1 -.5 -1 .25 1 -1 1 -.5 -1 .25 1 -1 -1 -1.5 -1 .25
- 29. Exercise • Demonstrate that you understand the Adaline training procedure.
- 30. Madaline • Many adaptive linear neurons xm x1 1 zk z1 1 y
- 31. Madaline • MRI (1960) – only learns weights from input layer to hidden layer • MRII (1987) – learns all weights