15 Machine Learning Multilayer Perceptron

1. Neural Networks Multilayer Perceptron Andres Mendez-Vazquez December 12, 2015 1 / 94

2. Outline 1 Introduction The XOR Problem 2 Multi-Layer Perceptron Architecture Back-propagation Gradient Descent Hidden–to-Output Weights Input-to-Hidden Weights Total Training Error About Stopping Criteria Final Basic Batch Algorithm 3 Using Matrix Operations to Simplify Using Matrix Operations to Simplify the Pseudo-Code Generating the Output zk Generating zk Generating the Weights from Hidden to Output Layer Generating the Weights from Input to Hidden Layer Activation Functions 4 Heuristic for Multilayer Perceptron Maximizing information content Activation Function Target Values Normalizing the inputs Virtues and limitations of Back-Propagation Layer 2 / 94

4. Do you remember? The Perceptron has the following problem Given that the perceptron is a linear classiﬁer It is clear that It will never be able to classify stuﬀ that is not linearly separable 4 / 94

5. Do you remember? The Perceptron has the following problem Given that the perceptron is a linear classiﬁer It is clear that It will never be able to classify stuﬀ that is not linearly separable 4 / 94

6. Example: XOR Problem The Problem 0 1 1 Class 1 Class 2 5 / 94

7. The Perceptron cannot solve it Because The perceptron is a linear classiﬁer!!! Thus Something needs to be done!!! Maybe Add an extra layer!!! 6 / 94

10. A little bit of history It was first cited by Vapnik Vapnik cites (Bryson, A.E.; W.F. Denham; S.E. Dreyfus. Optimal programming problems with inequality constraints. I: Necessary conditions for extremal solutions. AIAA J. 1, 11 (1963) 2544-2550) as the first publication of the backpropagation algorithm in his book "Support Vector Machines." It was first used by Arthur E. Bryson and Yu-Chi Ho described it as a multi-stage dynamic system optimization method in 1969. However It was not until 1974 and later, when applied in the context of neural networks and through the work of Paul Werbos, David E. Rumelhart, Geoffrey E. Hinton and Ronald J. Williams that it gained recognition. 7 / 94

13. Then Something Notable It led to a “renaissance” in the ﬁeld of artiﬁcial neural network research. Nevertheless During the 2000s it fell out of favour but has returned again in the 2010s, now able to train much larger networks using huge modern computing power such as GPUs. 8 / 94

14. Then Something Notable It led to a “renaissance” in the ﬁeld of artiﬁcial neural network research. Nevertheless During the 2000s it fell out of favour but has returned again in the 2010s, now able to train much larger networks using huge modern computing power such as GPUs. 8 / 94

16. Multi-Layer Perceptron (MLP) Multi-Layer Architecture· Output Input Target Hidden Sigmoid Activation function Identity Activation function 10 / 94

17. Information Flow We have the following information ﬂow Function Signals Error Signals 11 / 94

18. Explanation Problems with Hidden Layers 1 Increase complexity of Training 2 It is necessary to think about “Long and Narrow” network vs “Short and Fat” network. Intuition for a One Hidden Layer 1 For every input case of region, that region can be delimited by hyperplanes on all sides using hidden units on the ﬁrst hidden layer. 2 A hidden unit in the second layer than ANDs them together to bound the region. Advantages It has been proven that an MLP with one hidden layer can learn any nonlinear function of the input. 12 / 94

23. The Process We have something like this Layer 1 (0,0,1) (1,0,0) (1,0,0) Layer 2 13 / 94

25. Remember!!! The Quadratic Learning Error function Cost Function our well know error at pattern m J (m) = 1 2 e2 k (m) (1) Delta Rule or Widrow-Hoﬀ Rule ∆wkj (m) = −ηek (m) xj(m) (2) Actually this is know as Gradient Descent wkj (m + 1) = wkj (m) + ∆wkj (m) (3) 15 / 94

28. Back-propagation Setup Let tk be the k-th target (or desired) output and zk be the k-th computed output with k = 1, . . . , d and w represents all the weights of the network Training Error for a single Pattern or Sample!!! J (w) = 1 2 c k=1 (tk − zk)2 = 1 2 t − z 2 (4) 16 / 94

29. Back-propagation Setup Let tk be the k-th target (or desired) output and zk be the k-th computed output with k = 1, . . . , d and w represents all the weights of the network Training Error for a single Pattern or Sample!!! J (w) = 1 2 c k=1 (tk − zk)2 = 1 2 t − z 2 (4) 16 / 94

31. Gradient Descent Gradient Descent The back-propagation learning rule is based on gradient descent. Reducing the Error The weights are initialized with pseudo-random values and are changed in a direction that will reduce the error: ∆w = −η ∂J ∂w (5) Where η is the learning rate which indicates the relative size of the change in weights: w (m + 1) = w (m) + ∆w (m) (6) where m is the m-th pattern presented 18 / 94

35. Multilayer Architecture Multilayer Architecture: hidden–to-output weights Output Input Target Hidden 20 / 94

36. Observation about the activation function Hidden Output is equal to yj = f d i=1 wjixi Output is equal to zk = f   ynH j=1 wkjyj   21 / 94

37. Observation about the activation function Hidden Output is equal to yj = f d i=1 wjixi Output is equal to zk = f   ynH j=1 wkjyj   21 / 94

38. Hidden–to-Output Weights Error on the hidden–to-output weights ∂J ∂wkj = ∂J ∂netk · ∂netk ∂wkj = −δk · ∂netk ∂wkj (7) netk It describes how the overall error changes with the activation of the unit’s net: netk = ynH j=1 wkjyj = wT k · y (8) Now δk = − ∂J ∂netk = − ∂J ∂zk · ∂zk ∂netk = (tk − zk) f (netk) (9) 22 / 94

41. Hidden–to-Output Weights Why? zk = f (netk) (10) Thus ∂zk ∂netk = f (netk) (11) Since netk = wT k · y therefore: ∂netk ∂wkj = yj (12) 23 / 94

44. Finally The weight update (or learning rule) for the hidden-to-output weights is: wkj = ηδkyj = η (tk − zk) f (netk) yj (13) 24 / 94

46. Multi-Layer Architecture Multi-Layer Architecture: Input–to-Hidden weights Output Input Target Hidden 26 / 94

47. Input–to-Hidden Weights Error on the Input–to-Hidden weights ∂J ∂wji = ∂J ∂yj · ∂yj ∂netj · ∂netj ∂wji (14) Thus ∂J ∂yj = ∂ ∂yj 1 2 c k=1 (tk − zk)2 = − c k=1 (tk − zk) ∂zk ∂yj = − c k=1 (tk − zk) ∂zk ∂netk · ∂netk ∂yj = − c k=1 (tk − zk) ∂f (netk) ∂netk · wkj 27 / 94

48. Input–to-Hidden Weights Error on the Input–to-Hidden weights ∂J ∂wji = ∂J ∂yj · ∂yj ∂netj · ∂netj ∂wji (14) Thus ∂J ∂yj = ∂ ∂yj 1 2 c k=1 (tk − zk)2 = − c k=1 (tk − zk) ∂zk ∂yj = − c k=1 (tk − zk) ∂zk ∂netk · ∂netk ∂yj = − c k=1 (tk − zk) ∂f (netk) ∂netk · wkj 27 / 94

49. Input–to-Hidden Weights Finally ∂J ∂yj = − c k=1 (tk − zk) f (netk) · wkj (15) Remember δk = − ∂J ∂netk = (tk − zk) f (netk) (16) 28 / 94

50. What is ∂yj ∂netj ? First netj = d i=1 wjixi = wT j · x (17) Then yj = f (netj) Then ∂yj ∂netj = ∂f (netj) ∂netj = f (netj) 29 / 94

53. Then, we can deﬁne δj By defying the sensitivity for a hidden unit: δj = f (netj) c k=1 wkjδk (18) Which means that: “The sensitivity at a hidden unit is simply the sum of the individual sensitivities at the output units weighted by the hidden-to-output weights wkj; all multiplied by f (netj)” 30 / 94

56. What about ∂netj ∂wji ? We have that ∂netj ∂wji = ∂wT j · x ∂wji = ∂ d i=1 wjixi ∂wji = xi 31 / 94

57. Finally The learning rule for the input-to-hidden weights is: ∆wji = ηxiδj = η c k=1 wkjδk f (netj) xi (19) 32 / 94

58. Basically, the entire training process has the following steps Initialization Assuming that no prior information is available, pick the synaptic weights and thresholds Forward Computation Compute the induced function signals of the network by proceeding forward through the network, layer by layer. Backward Computation Compute the local gradients of the network. Finally Adjust the weights!!! 33 / 94

63. Now, Calculating Total Change We have for that The Total Training Error is the sum over the errors of N individual patterns The Total Training Error J = N p=1 Jp = 1 2 N p=1 d k=1 (tp k − zp k ) 2 = 1 2 n p=1 tp − zp 2 (20) 35 / 94

64. Now, Calculating Total Change We have for that The Total Training Error is the sum over the errors of N individual patterns The Total Training Error J = N p=1 Jp = 1 2 N p=1 d k=1 (tp k − zp k ) 2 = 1 2 n p=1 tp − zp 2 (20) 35 / 94

65. About the Total Training Error Remarks A weight update may reduce the error on the single pattern being presented but can increase the error on the full training set. However, given a large number of such individual updates, the total error of equation (20) decreases. 36 / 94

66. About the Total Training Error Remarks A weight update may reduce the error on the single pattern being presented but can increase the error on the full training set. However, given a large number of such individual updates, the total error of equation (20) decreases. 36 / 94

68. Now, we want the training to stop Therefore It is necessary to have a way to stop when the change of the weights is enough!!! A simple way to stop the training The algorithm terminates when the change in the criterion function J(w) is smaller than some preset value Θ. ∆J (w) = |J (w (t + 1)) − J (w (t))| (21) There are other stopping criteria that lead to better performance than this one. 38 / 94

72. Other Stopping Criteria Norm of the Gradient The back-propagation algorithm is considered to have converged when the Euclidean norm of the gradient vector reaches a suﬃciently small gradient threshold. wJ (m) < Θ (22) Rate of change in the average error per epoch The back-propagation algorithm is considered to have converged when the absolute rate of change in the average squared error per epoch is suﬃciently small. 1 N N p=1 Jp < Θ (23) 39 / 94

73. Other Stopping Criteria Norm of the Gradient The back-propagation algorithm is considered to have converged when the Euclidean norm of the gradient vector reaches a suﬃciently small gradient threshold. wJ (m) < Θ (22) Rate of change in the average error per epoch The back-propagation algorithm is considered to have converged when the absolute rate of change in the average squared error per epoch is suﬃciently small. 1 N N p=1 Jp < Θ (23) 39 / 94

74. About the Stopping Criteria Observations 1 Before training starts, the error on the training set is high. Through the learning process, the error becomes smaller. 2 The error per pattern depends on the amount of training data and the expressive power (such as the number of weights) in the network. 3 The average error on an independent test set is always higher than on the training set, and it can decrease as well as increase. 4 A validation set is used in order to decide when to stop training. We do not want to over-ﬁt the network and decrease the power of the classiﬁer generalization “we stop training at a minimum of the error on the validation set” 40 / 94

80. Some More Terminology Epoch As with other types of backpropagation, ’learning’ is a supervised process that occurs with each cycle or ’epoch’ through a forward activation ﬂow of outputs, and the backwards error propagation of weight adjustments. In our case I am using the batch sum of all correcting weights to deﬁne that epoch. 41 / 94

81. Some More Terminology Epoch As with other types of backpropagation, ’learning’ is a supervised process that occurs with each cycle or ’epoch’ through a forward activation ﬂow of outputs, and the backwards error propagation of weight adjustments. In our case I am using the batch sum of all correcting weights to deﬁne that epoch. 41 / 94

83. Final Basic Batch Algorithm Perceptron(X) 1 Initialize random w, number of hidden units nH , number of outputs z, stopping criterion Θ, learning rateη, epoch m = 0 2 do 3 m = m + 1 4 for s = 1 to N 5 x (m) = X (:, s) 6 for k = 1 to c 7 δk = (tk − zk ) f wT k · y 8 for j = 1 to nH 9 netj = wT j · x;yj = f netj 10 wkj (m) = wkj (m) + ηδk yj (m) 11 for j = 1 to nH 12 δj = f netj c k=1 wkj δk 13 for i = 1 to d 14 wji (m) = wji (m) + ηδj xi (m) 15 until w J (m) < Θ 16 return w (m) 43 / 94

102. Example of Architecture to be used Given the following Architecture and assuming N samples 45 / 94

104. Generating the output zk Given the input X = x1 x2 · · · xN (24) Where xi is a vector of features xi =       x1i x2i ... xdi       (25) 47 / 94

105. Generating the output zk Given the input X = x1 x2 · · · xN (24) Where xi is a vector of features xi =       x1i x2i ... xdi       (25) 47 / 94

106. Therefore We must have the following matrix for the input to hidden inputs WIH =    w11 w12 · · · w1d w21 w22 · · · w2d ... ... ... ... wnH 1 wnH 2 · · · wnH d    =     wT 1 wT 2 ... wT nH     (26) Given that wj =    wj1 wj2 ... wjd    Thus We can create the netj for all the inputs by simply netj = WIH X =     wT 1 x1 wT 1 x2 · · · wT 1 xN wT 2 x1 wT 2 x2 · · · wT 2 xN . .. . .. ... . .. wT nH x1 wT nH x2 · · · wT nH xN     (27) 48 / 94

107. Therefore We must have the following matrix for the input to hidden inputs WIH =    w11 w12 · · · w1d w21 w22 · · · w2d ... ... ... ... wnH 1 wnH 2 · · · wnH d    =     wT 1 wT 2 ... wT nH     (26) Given that wj =    wj1 wj2 ... wjd    Thus We can create the netj for all the inputs by simply netj = WIH X =     wT 1 x1 wT 1 x2 · · · wT 1 xN wT 2 x1 wT 2 x2 · · · wT 2 xN . .. . .. ... . .. wT nH x1 wT nH x2 · · · wT nH xN     (27) 48 / 94

108. Now, we need to generate the yk We apply the activation function element by element in netj y1 =         f wT 1 x1 f wT 1 x2 · · · f wT 1 xN f wT 2 x1 f wT 2 x2 · · · f wT 2 xN ... ... ... ... f wT nH x1 f wT nH x2 · · · f wT nH xN         (28) IMPORTANT about overﬂows!!! Be careful about the numeric stability of the activation function. I the case of python, we can use the ones provided by scipy.special 49 / 94

111. However, We can create a Sigmoid function It is possible to use the following pseudo-code Sigmoid(x) 1 if try 1.0 1.0+exp{−αx} catch {OVERFLOW } I will use a try-and-catch to catch the overﬂow 2 if x < 0 3 return 0 4 else 5 return 1 6 else 7 return 1.0 1.0+exp{−αx} 1.0 refers to the ﬂoating point (Rationals trying to represent Reals) 50 / 94

116. For this, we get netk For this, we obtain the WHO WHO = wo 11 wo 12 · · · wo 1nH = wT o (29) Thus netk = wo 11 wo 12 · · · wo 1nH       f wT 1 x1 f wT 1 x2 · · · f wT 1 xN f wT 2 x1 f wT 2 x2 · · · f wT 2 xN . . . . . . . . . . . . f w T nH x1 f w T nH x2 · · · f w T nH xN       yk1 yk2 · · · ykN (30) In matrix notation netk = wT o yk1 wT o yk2 · · · wT o ykN (31) 52 / 94

120. Now, we have Thus, we have zk (In our case k = 1, but it could be a range of values) zk = f wT o yk1 f wT o yk2 · · · f wT o ykN (32) Thus, we generate a vector of diﬀerences d = t − zk = t1 − f wT o yk1 t2 − f wT o yk2 · · · tN − f wT o ykN (33) where t = t1 t2 · · · tN is a row vector of desired outputs for each sample. 54 / 94

121. Now, we have Thus, we have zk (In our case k = 1, but it could be a range of values) zk = f wT o yk1 f wT o yk2 · · · f wT o ykN (32) Thus, we generate a vector of diﬀerences d = t − zk = t1 − f wT o yk1 t2 − f wT o yk2 · · · tN − f wT o ykN (33) where t = t1 t2 · · · tN is a row vector of desired outputs for each sample. 54 / 94

122. Now, we multiply element wise We have the following vector of derivatives of net Df = ηf wT o yk1 ηf wT o yk2 · · · ηf wT o ykN (34) where η is the step rate. Finally, by element wise multiplication (Hadamard Product) d = η t1 − f wT o yk1 f wT o yk1 η t2 − f wT o yk2 f wT o yk2 · · · η tN − f wT o ykN f wT o ykN 55 / 94

123. Now, we multiply element wise We have the following vector of derivatives of net Df = ηf wT o yk1 ηf wT o yk2 · · · ηf wT o ykN (34) where η is the step rate. Finally, by element wise multiplication (Hadamard Product) d = η t1 − f wT o yk1 f wT o yk1 η t2 − f wT o yk2 f wT o yk2 · · · η tN − f wT o ykN f wT o ykN 55 / 94

124. Tile d Tile downward dtile = nH rows          d d ... d       (35) Finally, we multiply element wise against y1 (Hadamard Product) ∆wtemp 1j = y1 ◦ dtile (36) 56 / 94

125. Tile d Tile downward dtile = nH rows          d d ... d       (35) Finally, we multiply element wise against y1 (Hadamard Product) ∆wtemp 1j = y1 ◦ dtile (36) 56 / 94

126. We obtain the total ∆w1j We sum along the rows of ∆wtemp 1j ∆w1j =    η t1 − f wT o yk1 f wT o yk1 y11 + η t1 − f wT o yk1 f wT o yk1 y1N . . . η t1 − f wT o yk1 f wT o yk1 ynH 1 + η t1 − f wT o yk1 f wT o yk1 ynH N    (37) where yhm = f wT h xm with h = 1, 2, ..., nH and m = 1, 2, ..., N. 57 / 94

127. Finally, we update the ﬁrst weights We have then WHO (t + 1) = WHO (t) + ∆wT 1j (t) (38) 58 / 94

129. First We multiply element wise the WHO and ∆w1j T = ∆wT 1j ◦ WT HO (39) Now, we obtain the element wise derivative of netj Dnetj =         f wT 1 x1 f wT 1 x2 · · · f wT 1 xN f wT 2 x1 f wT 2 x2 · · · f wT 2 xN ... ... ... ... f wT nH x1 f wT nH x2 · · · f wT nH xN         (40) 60 / 94

130. First We multiply element wise the WHO and ∆w1j T = ∆wT 1j ◦ WT HO (39) Now, we obtain the element wise derivative of netj Dnetj =         f wT 1 x1 f wT 1 x2 · · · f wT 1 xN f wT 2 x1 f wT 2 x2 · · · f wT 2 xN ... ... ... ... f wT nH x1 f wT nH x2 · · · f wT nH xN         (40) 60 / 94

131. Thus We tile to the right T Ttile = T T · · · T N Columns (41) Now, we multiply element wise together with η Pt = η (Dnetj ◦ Ttile) (42) where η is constant multiplied against the result the Hadamar Product (Result a nH × N matrix) 61 / 94

132. Thus We tile to the right T Ttile = T T · · · T N Columns (41) Now, we multiply element wise together with η Pt = η (Dnetj ◦ Ttile) (42) where η is constant multiplied against the result the Hadamar Product (Result a nH × N matrix) 61 / 94

133. Finally We get use the transpose of X which is a N × d matrix XT =       xT 1 xT 2 ... xT N       (43) Finally, we get a nH × d matrix ∆wij = PtXT (44) Thus, given WIH WIH (t + 1) = WHO (t) + ∆wT ij (t) (45) 62 / 94

137. We have diﬀerent activation functions The two most important 1 Sigmoid function. 2 Hyperbolic tangent function 64 / 94

138. We have diﬀerent activation functions The two most important 1 Sigmoid function. 2 Hyperbolic tangent function 64 / 94

139. Logistic Function This non-linear function has the following deﬁnition for a neuron j fj (vj (n)) = 1 1 + exp {−avj (n)} a > 0 and − ∞ < vj (n) < ∞ (46) Example 65 / 94

140. Logistic Function This non-linear function has the following deﬁnition for a neuron j fj (vj (n)) = 1 1 + exp {−avj (n)} a > 0 and − ∞ < vj (n) < ∞ (46) Example 65 / 94

141. The diﬀerential of the sigmoid function Now if we diﬀerentiate, we have fj (vj (n)) = 1 1 + exp {−avj (n)} 1 − 1 1 + exp {−avj (n)} = exp {−avj (n)} (1 + exp {−avj (n)})2 66 / 94

142. The diﬀerential of the sigmoid function Now if we diﬀerentiate, we have fj (vj (n)) = 1 1 + exp {−avj (n)} 1 − 1 1 + exp {−avj (n)} = exp {−avj (n)} (1 + exp {−avj (n)})2 66 / 94

143. The outputs ﬁnish as For the output neurons δk = (tk − zk) f (netk) = (tk − fk (vk (n))) fk (vk (n)) (1 − fk (vk (n))) For the hidden neurons δj =fj (vj (n)) (1 − fj (vj (n))) c k=1 wkjδk 67 / 94

146. Hyperbolic tangent function Another commonly used form of sigmoidal non linearity is the hyperbolic tangent function fj (vj (n)) = a tanh (bvj (n)) (47) Example 68 / 94

147. Hyperbolic tangent function Another commonly used form of sigmoidal non linearity is the hyperbolic tangent function fj (vj (n)) = a tanh (bvj (n)) (47) Example 68 / 94

148. The diﬀerential of the hyperbolic tangent We have fj (vj (n)) =absech2 (bvj (n)) =ab 1 − tanh2 (bvj (n)) BTW I leave to you to ﬁgure out the outputs. 69 / 94

152. Maximizing information content Two ways of achieving this, LeCun 1993 The use of an example that results in the largest training error. The use of an example that is radically diﬀerent from all those previously used. For this Randomized the samples presented to the multilayer perceptron when not doing batch training. Or use an emphasizing scheme By using the error identify the diﬃcult vs. easy patterns: Use them to train the neural network 71 / 94

157. However Be careful about emphasizing scheme The distribution of examples within an epoch presented to the network is distorted. The presence of an outlier or a mislabeled example can have a catastrophic consequence on the performance of the algorithm. Deﬁnition of Outlier An outlier is an observation that lies outside the overall pattern of a distribution (Moore and McCabe 1999). 72 / 94

158. However Be careful about emphasizing scheme The distribution of examples within an epoch presented to the network is distorted. The presence of an outlier or a mislabeled example can have a catastrophic consequence on the performance of the algorithm. Deﬁnition of Outlier An outlier is an observation that lies outside the overall pattern of a distribution (Moore and McCabe 1999). 72 / 94

159. However Be careful about emphasizing scheme The distribution of examples within an epoch presented to the network is distorted. The presence of an outlier or a mislabeled example can have a catastrophic consequence on the performance of the algorithm. Deﬁnition of Outlier An outlier is an observation that lies outside the overall pattern of a distribution (Moore and McCabe 1999). 0 1 2 3 Outlier 72 / 94

161. Activation Function We say that An activation function f (v) is antisymmetric if f (−v) = −f (v) It seems to be That the multilayer perceptron learns faster using an antisymmetric function. Example: The hyperbolic tangent 74 / 94

165. Target Values Important It is important that the target values be chosen within the range of the sigmoid activation function. Speciﬁcally The desired response for neuron in the output layer of the multilayer perceptron should be oﬀset by some amount 76 / 94

166. Target Values Important It is important that the target values be chosen within the range of the sigmoid activation function. Speciﬁcally The desired response for neuron in the output layer of the multilayer perceptron should be oﬀset by some amount 76 / 94

167. For example Given the a limiting value We have then If we have a limiting value +a, we set t = a − . If we have a limiting value −a, we set t = −a + . 77 / 94

171. Normalizing the inputs Something Important (LeCun, 1993) Each input variable should be preprocessed so that: The mean value, averaged over the entire training set, is close to zero. Or it is smalll compared to its standard deviation. Example 79 / 94

174. Normalizing the inputs Something Important (LeCun, 1993) Each input variable should be preprocessed so that: The mean value, averaged over the entire training set, is close to zero. Or it is smalll compared to its standard deviation. Example Mean Value 79 / 94

175. The normalization must include two other measures Uncorrelated We can use the principal component analysis Example 80 / 94

176. The normalization must include two other measures Uncorrelated We can use the principal component analysis Example 80 / 94

177. In addition Quite interesting The decorrelated input variables should be scaled so that their covariances are The decorrelated approximately equal. Why Ensuring that the diﬀerent synaptic weights in the ely the same speed network learn at approximately the same speed. 81 / 94

178. In addition Quite interesting The decorrelated input variables should be scaled so that their covariances are The decorrelated approximately equal. Why Ensuring that the diﬀerent synaptic weights in the ely the same speed network learn at approximately the same speed. 81 / 94

179. There are other heuristics As Initialization Learning form hints Learning rates etc 82 / 94

183. In addition In section 4.15, Simon Haykin We have the following techniques: Network growing You start with a small network and add neurons and layers to accomplish the learning task. Network pruning Start with a large network, then prune weights that are not necessary in an orderly fashion. 83 / 94

187. Virtues and limitations of Back-Propagation Layer Something Notable The back-propagation algorithm has emerged as the most popular algorithm for the training of multilayer perceptrons. It has two distinct properties It is simple to compute locally. It performs stochastic gradient descent in weight space when doing pattern-by-pattern training 85 / 94

190. Connectionism Back-propagation t is an example of a connectionist paradigm that relies on local computations to discover the processing capabilities of neural networks. This form of restriction It is known as the locality constraint 86 / 94

191. Connectionism Back-propagation t is an example of a connectionist paradigm that relies on local computations to discover the processing capabilities of neural networks. This form of restriction It is known as the locality constraint 86 / 94

192. Why this is advocated in Artificial Neural Networks First Artificial neural networks that perform local computations are often held up as metaphors for biological neural networks. Second The use of local computations permits a graceful degradation in performance due to hardware errors, and therefore provides the basis for a fault-tolerant network design. Third Local computations favor the use of parallel architectures as an efficient method for the implementation of artificial neural networks. 87 / 94

195. However, all this has been seriously questioned on the following grounds(Shepherd, 1990b; Crick, 1989; Stork, 1989) First The reciprocal synaptic connections between the neurons of a multilayer perceptron may assume weights that are excitatory or inhibitory. In the real nervous system, neurons usually appear to be the one or the other. Second In a multilayer perceptron, hormonal and other types of global communications are ignored. 88 / 94

198. However, all this has been seriously questioned on the following grounds(Shepherd, 1990b; Crick, 1989; Stork, 1989) Third In back-propagation learning, a synaptic weight is modiﬁed by a presynaptic activity and an error (learning) signal independent of postsynaptic activity. There is evidence from neurobiology to suggest otherwise. Fourth In a neurobiological sense, the implementation of back-propagation learning requires the rapid transmission of information backward along an axon. It appears highly unlikely that such an operation actually takes place in the brain. 89 / 94

202. However, all this has been seriously questioned on the following grounds(Shepherd, 1990b; Crick, 1989; Stork, 1989) Fifth Back-propagation learning implies the existence of a "teacher," which in the con text of the brain would presumably be another set of neurons with novel properties. The existence of such neurons is biologically implausible. 90 / 94

203. However, all this has been seriously questioned on the following grounds(Shepherd, 1990b; Crick, 1989; Stork, 1989) Fifth Back-propagation learning implies the existence of a "teacher," which in the con text of the brain would presumably be another set of neurons with novel properties. The existence of such neurons is biologically implausible. 90 / 94

204. Computational Eﬃciency Something Notable The computational complexity of an algorithm is usually measured in terms of the number of multiplications, additions, and storage involved in its implementation. This is the electrical engineering approach!!! Taking in account the total number of synapses, W including biases We have wkj = ηδkyj = η (tk − zk) f (netk) yj (Backward Pass) We have that for this step 1 We need to calculate netk linear in the number of weights. 2 We need to calculate yj = f (netj) which is linear in the number of weights. 91 / 94

207. Computational Eﬃciency Now the Forward Pass ∆wji = ηxiδj = ηf (netj) c k=1 wkjδk xi We have that for this step [ c k=1 wkjδk] takes, because of the previous calculations of δk’s, linear on the number of weights Clearly all this takes to have memory In addition the calculation of the derivatives of the activation functions, but assuming a constant time. 92 / 94

210. We have that The Complexity of the multi-layer perceptron is O (W ) Complexity 93 / 94

211. Exercises We have from NN by Haykin 4.2, 4.3, 4.6, 4.8, 4.16, 4.17, 3.7 94 / 94

15 Machine Learning Multilayer Perceptron

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to 15 Machine Learning Multilayer Perceptron (20)

More from Andres Mendez-Vazquez (20)

Recently uploaded (20)

15 Machine Learning Multilayer Perceptron