Neural networks
Download as PPTX, PDF9 likes6,942 views
This document discusses neural networks and their biological and technical underpinnings. It covers how natural neural networks operate using electrochemical signals and thresholds. It also discusses early artificial neural network models like McCulloch-Pitts networks and perceptrons. Perceptrons are defined as single-layer feedforward networks and can only represent linearly separable functions. The document introduces the concept of adding hidden layers to networks to increase their computational power and ability to represent more complex functions like XOR.
Neural networks
- 1. Neural Networks V.Saranya AP/CSE Sri Vidya College of Engineering and Technology, Virudhunagar
- 3. Natural Neural Networks • Signals “move” via electrochemical signals • The synapses release a chemical transmitter – the sum of which can cause a threshold to be reached – causing the neuron to “fire” • Synapses can be inhibitory or excitatory 3
- 4. Natural Neural Networks • We are born with about 100 billion neurons • A neuron may connect to as many as 100,000 other neurons 4
- 5. Natural Neural Networks • Many of their ideas still used today e.g. – many simple units, “neurons” combine to give increased computational power – the idea of a threshold 5
- 6. Modelling a Neuron • aj :Activation value of unit j • wj,i :Weight on link from unit j to unit i • ini :Weighted sum of inputs to unit i • ai :Activation value of unit i • g :Activation function j jiji aWin , 6
- 7. Activation Functions • Stept(x) = 1 if x ≥ t, else 0 threshold=t • Sign(x) = +1 if x ≥ 0, else –1 • Sigmoid(x) = 1/(1+e-x) 7
- 8. Building a Neural Network 1. “Select Structure”: Design the way that the neurons are interconnected 2. “Select weights” – decide the strengths with which the neurons are interconnected – weights are selected so get a “good match” to a “training set” – “training set”: set of inputs and desired outputs – often use a “learning algorithm” 8
- 9. Basic Neural Networks • Will first look at simplest networks • “Feed-forward” – Signals travel in one direction through net – Net computes a function of the inputs 9
- 10. The First Neural Neural Networks Neurons in a McCulloch-Pitts network are connected by directed, weighted paths -1 2 2X1 X2 X3 Y 10
- 11. The First Neural Neural Networks If the on weight on a path is positive the path is excitatory, otherwise it is inhibitory -1 2 2X1 X2 X3 Y 11
- 12. The First Neural Neural Networks The activation of a neuron is binary. That is, the neuron either fires (activation of one) or does not fire (activation of zero). -1 2 2X1 X2 X3 Y 12
- 13. The First Neural Neural Networks For the network shown here the activation function for unit Y is f(y_in) = 1, if y_in >= θ else 0 where y_in is the total input signal received θ is the threshold for Y -1 2 2X1 X2 X3 Y 13
- 14. The First Neural Neural Networks Originally, all excitatory connections into a particular neuron have the same weight, although different weighted connections can be input to different neurons Later weights allowed to be arbitrary -1 2 2X1 X2 X3 Y 14
- 15. The First Neural Neural Networks Each neuron has a fixed threshold. If the net input into the neuron is greater than or equal to the threshold, the neuron fires -1 2 2X1 X2 X3 Y 15
- 16. The First Neural Neural Networks The threshold is set such that any non-zero inhibitory input will prevent the neuron from firing -1 2 2X1 X2 X3 Y 16
- 17. Building Logic Gates • Computers are built out of “logic gates” • Use threshold (step) function for activation function – all activation values are 0 (false) or 1 (true) 17
- 18. The First Neural Neural Networks AND Function 1 1 X1 X2 Y AND X1 X2 Y 1 1 1 1 0 0 0 1 0 0 0 0 Threshold(Y) = 2 18
- 19. The First Neural Networks AND FunctionOR Function 2 2X1 X2 Y OR X1 X2 Y 1 1 1 1 0 1 0 1 1 0 0 0 Threshold(Y) = 2 19
- 20. Perceptron • Synonym for Single-Layer, Feed-Forward Network • First Studied in the 50’s • Other networks were known about but the perceptron was the only one capable of learning and thus all research was concentrated in this area 20
- 21. Perceptron • A single weight only affects one output so we can restrict our investigations to a model as shown on the right • Notation can be simpler, i.e. j WjIjStepO 0 21
- 22. What can perceptrons represent? AND XOR Input 1 0 0 1 1 0 0 1 1 Input 2 0 1 0 1 0 1 0 1 Output 0 0 0 1 0 1 1 0 22
- 23. What can perceptrons represent? 0,0 0,1 1,0 1,1 0,0 0,1 1,0 1,1 AND XOR • Functions which can be separated in this way are called Linearly Separable • Only linearly separable functions can be represented by a perceptron • XOR cannot be represented by a perceptron 23
- 24. XOR • XOR is not “linearly separable” – Cannot be represented by a perceptron • What can we do instead? 1. Convert to logic gates that can be represented by perceptrons 2. Chain together the gates 24
- 25. Single- vs. Multiple-Layers • Once we chain together the gates then we have “hidden layers” – layers that are “hidden” from the output lines • Have just seen that hidden layers allow us to represent XOR – Perceptron is single-layer – Multiple layers increase the representational power, so e.g. can represent XOR • Generally useful nets have multiple-layers – typically 2-4 layers 25