![The First Neural Neural Networks
We want the system to perceive heat if either a hot stimulus is applied or a cold
stimulus is applied (for one time step) and then removed
Y1(t) = [ X1(t – 1) ] OR [ X2(t – 3) AND NOT X2(t – 2) ]
X2(t – 3) X2(t – 2) AND NOT X1(t – 1) OR
1 1 0 1 1
1 0 1 1 1
0 1 0 1 1
0 0 0 1 1
1 1 0 0 0
1 0 1 0 1
0 1 0 0 0
0 0 0 0 0](https://image.slidesharecdn.com/neuralnetworks-230312111333-15ca95a4/85/Neural-Networks-pptx-20-320.jpg)
![The First Neural Neural Networks
The network shows
Y1(t) = X1(t – 1) OR Z1(t – 1)
Z1(t – 1) = Z2( t – 2) AND NOT X2(t – 2)
Z2(t – 2) = X2(t – 3)
Substituting, we get
Y1(t) = [ X1(t – 1) ] OR [ X2(t – 3) AND NOT X2(t – 2) ]
which is the same as our original requirements](https://image.slidesharecdn.com/neuralnetworks-230312111333-15ca95a4/85/Neural-Networks-pptx-21-320.jpg)
![Learning
While epoch produces an error
Present network with next inputs from epoch
Err = T – O
If Err <> 0 then
Wj = Wj + LR * Ij * Err
End If
End While
Epoch : Presentation of the entire training set to the neural
network.
In the case of the AND function an epoch consists of
four sets of inputs being presented to the network (i.e.
[0,0], [0,1], [1,0], [1,1])](https://image.slidesharecdn.com/neuralnetworks-230312111333-15ca95a4/85/Neural-Networks-pptx-35-320.jpg)
![Learning
While epoch produces an error
Present network with next inputs from epoch
Err = T – O
If Err <> 0 then
Wj = Wj + LR * Ij * Err
End If
End While
Training Value, T : When we are training a network we not
only present it with the input but also with a value that
we require the network to produce. For example, if we
present the network with [1,1] for the AND function
the training value will be 1](https://image.slidesharecdn.com/neuralnetworks-230312111333-15ca95a4/85/Neural-Networks-pptx-36-320.jpg)
Neural Networks.pptx
- 2. The First Neural Neural Networks McCulloch and Pitts produced the first neural network in 1943 Many of the principles can still be seen in neural networks of today
- 3. The First Neural Neural Networks -1 2 2 X1 X2 X3 Y The activation of a neuron is binary. That is, the neuron either fires (activation of one) or does not fire (activation of zero).
- 4. The First Neural Neural Networks -1 2 2 X1 X2 X3 Y For the network shown here the activation function for unit Y is f(y_in) = 1, if y_in >= θ else 0 where y_in is the total input signal received θ is the threshold for Y
- 5. The First Neural Neural Networks -1 2 2 X1 X2 X3 Y Neurons in a McCulloch-Pitts network are connected by directed, weighted paths
- 6. The First Neural Neural Networks -1 2 2 X1 X2 X3 Y If the weight on a path is positive the path is excitatory, otherwise it is inhibitory
- 7. The First Neural Neural Networks -1 2 2 X1 X2 X3 Y All excitatory connections into a particular neuron have the same weight, although different weighted connections can be input to different neurons
- 8. The First Neural Neural Networks -1 2 2 X1 X2 X3 Y Each neuron has a fixed threshold. If the net input into the neuron is greater than the threshold, the neuron fires
- 9. The First Neural Neural Networks -1 2 2 X1 X2 X3 Y The threshold is set such that any non-zero inhibitory input will prevent the neuron from firing
- 10. The First Neural Neural Networks -1 2 2 X1 X2 X3 Y It takes one time step for a signal to pass over one connection.
- 11. The First Neural Neural Networks AND Function 1 1 X1 X2 Y AND X1 X2 Y 1 1 1 1 0 0 0 1 0 0 0 0 Threshold(Y) = 2
- 12. The First Neural Neural Networks AND Function OR Function 2 2 X1 X2 Y OR X1 X2 Y 1 1 1 1 0 1 0 1 1 0 0 0 Threshold(Y) = 2
- 13. The First Neural Neural Networks AND NOT Function -1 2 X1 X2 Y AND NOT X1 X2 Y 1 1 0 1 0 1 0 1 0 0 0 0 Threshold(Y) = 2
- 14. The First Neural Neural Networks XOR Function 2 2 2 2 -1 -1 Z1 Z2 Y X1 X2 XOR X1 X2 Y 1 1 0 1 0 1 0 1 1 0 0 0 X1 XOR X2 = (X1 AND NOT X2) OR (X2 AND NOT X1)
- 15. The First Neural Neural Networks If we touch something cold we perceive heat If we keep touching something cold we will perceive cold If we touch something hot we will perceive heat
- 16. The First Neural Neural Networks To model this we will assume that time is discrete If cold is applied for one time step then heat will be perceived If a cold stimulus is applied for two time steps then cold will be perceived If heat is applied then we should perceive heat
- 17. The First Neural Neural Networks X1 X2 Z1 Z2 Y1 Y2 Heat Cold 2 2 2 1 2 -1 1 Hot Cold
- 18. The First Neural Neural Networks X1 X2 Z1 Z2 Y1 Y2 Heat Cold 2 2 2 1 2 -1 1 Hot Cold • It takes time for the stimulus (applied at X1 and X2) to make its way to Y1 and Y2 where we perceive either heat or cold • At t(0), we apply a stimulus to X1 and X2 • At t(1) we can update Z1, Z2 and Y1 • At t(2) we can perceive a stimulus at Y2 • At t(2+n) the network is fully functional
- 19. The First Neural Neural Networks We want the system to perceive cold if a cold stimulus is applied for two time steps Y2(t) = X2(t – 2) AND X2(t – 1) X2(t – 2) X2( t – 1) Y2(t) 1 1 1 1 0 0 0 1 0 0 0 0
- 20. The First Neural Neural Networks We want the system to perceive heat if either a hot stimulus is applied or a cold stimulus is applied (for one time step) and then removed Y1(t) = [ X1(t – 1) ] OR [ X2(t – 3) AND NOT X2(t – 2) ] X2(t – 3) X2(t – 2) AND NOT X1(t – 1) OR 1 1 0 1 1 1 0 1 1 1 0 1 0 1 1 0 0 0 1 1 1 1 0 0 0 1 0 1 0 1 0 1 0 0 0 0 0 0 0 0
- 21. The First Neural Neural Networks The network shows Y1(t) = X1(t – 1) OR Z1(t – 1) Z1(t – 1) = Z2( t – 2) AND NOT X2(t – 2) Z2(t – 2) = X2(t – 3) Substituting, we get Y1(t) = [ X1(t – 1) ] OR [ X2(t – 3) AND NOT X2(t – 2) ] which is the same as our original requirements
- 22. The First Neural Neural Networks You can confirm that Y2 works correctly You can also check it works on the spreadsheet Threshold 2 Time Heat (X1) Cold (X2) Z1 Z2 Hot (Y1) Cold (Y2) 0 0 1 1 0 0 0 1 2 0 0 1 0 0 0 3 0 0 1 0 Time Heat (X1) Cold (X2) Z1 Z2 Hot (Y1) Cold (Y2) X1 X2 Z1 Z2 0 0 1 Z1 -1 2 1 0 1 0 1 Z2 2 2 0 0 0 1 0 1 Y1 2 2 Y2 1 1 Time Heat (X1) Cold (X2) Z1 Z2 Hot (Y1) Cold (Y2) 0 1 0 1 1 0 0 0 2 0 0 0 0 1 0 Readacrosstoseethe inputs to eachneuron Apply coldfor onetime stepand weperceiveheat Apply coldfor twotime steps and weperceivecold See Fausett, 1994, pp31 - 35 Apply heat and weperceiveheat
- 23. Modelling a Neuron • aj :Activation value of unit j • wj,I :Weight on the link from unit j to unit i • inI :Weighted sum of inputs to unit i • aI :Activation value of unit i • g :Activation function j j i j i a W in ,
- 24. Activation Functions • Stept(x) = 1 if x >= t, else 0 • Sign(x) = +1 if x >= 0, else –1 • Sigmoid(x) = 1/(1+e-x) • Identity Function
- 25. Simple Networks AND OR NOT Input 1 0 0 1 1 0 0 1 1 0 1 Input 2 0 1 0 1 0 1 0 1 Output 0 0 0 1 0 1 1 1 1 0
- 26. Simple Networks t = 0.0 y x W = 1.5 W = 1 -1
- 27. Perceptron • Synonym for Single- Layer, Feed-Forward Network • First Studied in the 50’s • Other networks were known about but the perceptron was the only one capable of learning and thus all research was concentrated in this area
- 28. Perceptron • A single weight only affects one output so we can restrict our investigations to a model as shown on the right • Notation can be simpler, i.e. j WjIj Step O 0
- 29. What can perceptrons represent? AND XOR Input 1 0 0 1 1 0 0 1 1 Input 2 0 1 0 1 0 1 0 1 Output 0 0 0 1 0 1 1 0
- 30. What can perceptrons represent? 0,0 0,1 1,0 1,1 0,0 0,1 1,0 1,1 AND XOR • Functions which can be separated in this way are called Linearly Separable • Only linearly Separable functions can be represented by a perceptron
- 31. What can perceptrons represent? Linear Separability is also possible in more than 3 dimensions – but it is harder to visualise
- 32. Training a perceptron AND Input 1 0 0 1 1 Input 2 0 1 0 1 Output 0 0 0 1 Aim
- 33. Training a perceptrons t = 0.0 y x -1 W = 0.3 W = -0.4 W = 0.5 I1 I2 I3 Summation Output -1 0 0 (-1*0.3) + (0*0.5) + (0*-0.4) = -0.3 0 -1 0 1 (-1*0.3) + (0*0.5) + (1*-0.4) = -0.7 0 -1 1 0 (-1*0.3) + (1*0.5) + (0*-0.4) = 0.2 1 -1 1 1 (-1*0.3) + (1*0.5) + (1*-0.4) = -0.2 0
- 34. Learning While epoch produces an error Present network with next inputs from epoch Err = T – O If Err <> 0 then Wj = Wj + LR * Ij * Err End If End While
- 35. Learning While epoch produces an error Present network with next inputs from epoch Err = T – O If Err <> 0 then Wj = Wj + LR * Ij * Err End If End While Epoch : Presentation of the entire training set to the neural network. In the case of the AND function an epoch consists of four sets of inputs being presented to the network (i.e. [0,0], [0,1], [1,0], [1,1])
- 36. Learning While epoch produces an error Present network with next inputs from epoch Err = T – O If Err <> 0 then Wj = Wj + LR * Ij * Err End If End While Training Value, T : When we are training a network we not only present it with the input but also with a value that we require the network to produce. For example, if we present the network with [1,1] for the AND function the training value will be 1
- 37. Learning While epoch produces an error Present network with next inputs from epoch Err = T – O If Err <> 0 then Wj = Wj + LR * Ij * Err End If End While Error, Err : The error value is the amount by which the value output by the network differs from the training value. For example, if we required the network to output 0 and it output a 1, then Err = -1
- 38. Learning While epoch produces an error Present network with next inputs from epoch Err = T – O If Err <> 0 then Wj = Wj + LR * Ij * Err End If End While Output from Neuron, O : The output value from the neuron Ij : Inputs being presented to the neuron Wj : Weight from input neuron (Ij) to the output neuron LR : The learning rate. This dictates how quickly the network converges. It is set by a matter of experimentation. It is typically 0.1
- 39. Learning 0,0 0,1 1,0 1,1 I1 I2 After First Epoch 0,0 0,1 1,0 1,1 I1 I2 At Convergence Note I1 point = W0/W1 I2 point = W0/W2