![Error Correction Learning
• Error signal: ek(n) = dk(n) – yk(n)
• Control mechanism to apply a series of corrective
adjustments
• Index of performance or instantaneous value of
Error Energy: E(n) = ½ ek2(n)
• Delta rule or Widrow-Hopf rule
– Thus Δwkj(n) = ηek(n)xj(n)
• And wkj(n+1) = wkj(n) + Δwkj(n)
• Using unit delay operator: wkj(n) = z-1[wkj(n+1)]](https://image.slidesharecdn.com/neuralnetworks20120320-120321121443-phpapp01/85/Neural-networks-31-320.jpg)
![Perceptron Convergence Theorem
• 1: Initialization : set w(0) = 0
• 2: Activation: at time step n, activate the perceptron by applying
continuous valued input vector x(n) and desired response d(n)
• 3: Computation of Actual Response: Compute the actual response
of the perceptron
– y(n) = sgn(wT(n)x(n))
• 4: Adaptation of weight vector: Update the weight vector of the
perceptron:
– w(n+1) = w(n) + η[d(n) – y(n)]x(n)
– Where
– D(n) = +1 if x(n) belongs to class C1
– = -1 if x(n) belongs to class C2
• Continuation: Increment time step n by one and go back to step 2](https://image.slidesharecdn.com/neuralnetworks20120320-120321121443-phpapp01/85/Neural-networks-56-320.jpg)
Neural networks...
- 1. Neural Networks (TEC-833) B.Tech (EC – VIII Sem) – Spring 2012 dcpande@gmail.com 9997756323
- 2. Mid Term Syllabus • Introduction: – Brain and Machine, Biological neurons and its mathematical model, Artificial Neural Networks, Benefits and Applications, Architectures, Learning Process (paradigms and algorithms), Correlation Matrix Memory, Adaptation. • Supervised learning – I: – Pattern space and weight space, Linearly and non-linearly separable classes, decision boundary, Hebbian learning and limitation, Perceptron, Perceptron convergence theorem, Logic Functions implementations • LMS Algorithm: – LMS Algorithm, • Supervised Learning – II: – Multilayer Perceptrons, XOR problem,
- 3. End Sem Syllabus • Introduction: Brain and Machine, Biological neurons and its mathematical model, Artificial Neural Networks, Benefits and Applications, Architectures, Learning Process (paradigms and algorithms), Correlation Matrix Memory, Adaptation. • Supervised learning – I: Pattern space and weight space, Linearly and non-linearly separable classes, decision boundary, Hebbian learning and limitation, Perceptron, Perceptron convergence theorem, Logic Functions implementations • LMS Algorithm: Wiener-Hopf equations, Steepest Descent search method, LMS Algorithm, Convergence consideration in mean and mean square, Adaline, Learning curve, Learning rate annealing schedules • Supervised Learning – II: Multilayer Perceptrons, Backpropagation algorithm, XOR problem, Training modes, Optimum learning, Local minima, Network pruning techniques • Unsupervised learning: Clustering, Hamming networks, Maxnet, Simple competitive learning, Winner-take-all networks, Learning Vector Quantizers, Counterpropagation Networks, Self Organizing Maps (Kohonen Networks), Adaptive Resonance Theory • Associative Models: Hopfield Networks (Discrete and Continuous), Storage Capacity, Energy function and minimization, Brain-state-in-a-box neural network • Applications of ANN and MATLAB Simulation: Character Recognition, Control Applications, Data Compression, Self Organizing Semantic Maps
- 4. References • Neural Networks: A Comprehensive Foundation – Simon Haykin (Pearson Education) • Neural Networks: A Classroom Approach – Satish Kumar (Tata McGraw Hill) • Fundamentals of Neural Networks – Laurene Fausett (Pearson Education) • ftp://ftp.sas.com/pub/neural/FAQ.html • MATLAB neural network toolbox and related help notes
- 5. Inputs to Neural Networks • Biology • Graph Theory • Algorithms • Artificial Intelligence • Control Systems • Signal Theory
- 6. Minsky’s challenge (adapted from Minsky, Singh and Sloman (2004)) Few Number of Causes Many Symbolic Logical Case based Many Intractable Reasoning Reasoning Ordinary Analogy Based Number of Effects Qualitative Classical AI Reasoning Reasoning Connectionist, Few Easy Linear, Statistical Neural Network, Fuzzy Logic
- 7. Who uses Neural Networks Area Use Computer Scientists To understand properties of non-symbolic information processing; Learning systems Engineers In many areas including signal processing and automatic control Statisticians As flexible, non-linear regression and classification models Physicists To model phenomenon in statistical mechanics and other tasks Cognitive Scientists To describe models of thinking and consciousness and other high level brain functions Neuro-physiologists To describe and explore memory, sensory functions, motor functions and other mid-level brain functions Biologists To interpret nucleotide sequences Philosophers etc. For their own reasons
- 8. Brain vs the computer http://scienceblogs.com/developingintelligence/2007/03/why_the_brain_is_not_like_a_co.php Brain Computer Brains are analogue (neuronal firing rate, Computers are digital asynchronous, leakiness) Brain uses content-addressable memory Computers use byte addressable memory Brain is a massively parallel machine Computers are modular and serial Processing speed is not fixed in the brain; there is Processing speed is fixed; there is a system clock no system clock Short-term memory only holds pointers to long RAM has isomorphic data term memory No hardware/software distinction can be made Computers have a clear distinction between with respect to the brain or mind hardware and software Synapses are far more complex than electrical Electrical gates are simpler in function and logic gates mechanism Processing and memory are performed by the Processing and memory are performed by same components in the brain different components in the computer The brain is a self-organizing system Computers are usually not self organizing Brains have bodies and use them Computers do not usually use their bodies The brain capacity is much larger than any Computer capacities though large are still not computer comparable with those of the brain
- 9. Neuro products and application areas • Academia Research • Market Segmentation • Automotive Industry • Medical Diagnosis • Bio Informatics • Meteorological Research • Cancer Detection • Optical Character Recognition • Computer Gaming • Pattern Recognition • Credit Ratings • Predicting Business Expenses • Drug Interaction Prediction • Real Estate Evaluations • Electrical Load Balancing • Robotics • Financial Forecasting • Sales Forecasting • Fraud Detection • Search Engines • Human Resources • Software Security • Image Recognition • Speech Recognition • Industrial Plant Modeling • Sports Betting • Machine Control • Sports Handicap Predictions • Machine Diagnostics
- 10. Applications of ANN (Sample Examples) • Non linear statistical data modeling tools • Function Approximation/ Mapping • Pattern recognition in data • Noise Cancellation (LMS) in signaling systems • Time Series Predictions • Control and Steering of Autonomous Vehicles (Feedforward) • Protein structure prediction and RNA splice junction identification • Sonar/radar/image/astronomy/handwriting target recognition/ classification • Call admission control for improving QOS in telecommunications (ATM) networks • Software engineering project management • Reinforcement Learning in Robotics (Backpropagation) • Pattern Completion (Hopfield) • Object recognition (Hopfield) • Clustering and Character Recognition (ART) • Neural Information Retrieval System(Machine parts retrieval at Boeing) (ART) • Neural Phonetic Typewriter (SOM) • Control of Robot Arms (SOM) • Vector Quantization (SOM) • Radar based classification (SOM) • Brain Modeling (SOM) • Feature mapping of language data (SOM) • Organization of massive document collection (SOM)
- 11. Neuroscience basics I • 100 B (10**11) neurons in brain • Each neuron has 10K (10**4) synapses on average • Thus 10**15 connections • A lifetime of 80 years is 2.5B seconds.
- 12. Structural organization of levels in brain Central Nervous System Interregional Circuits (Systems) Local Circuits (Maps/Networks) Neurons Dendritic Trees Neural microcircuits Synapses Molecules
- 13. Structural organization of levels in brain (Churchland)
- 14. Neuroscience Basics II • Brain structures – Cerebrum • Frontal Lobe • Temporal Lobe • Parietal Lobe • Occipital Lobe • Central Sulcus • Sylvian fissure – Cerebellum – Brain Stem • Corpus callosum • Thalamus • Hypothalamus • Midbrain • Pons • Medulla
- 15. Brain Anatomy
- 16. Brain Areas
- 17. Homunculus
- 18. Nervous System
- 19. Sympathetic and Parasympathetic nerves
- 20. Neuron - I
- 21. Neuron - II
- 22. Sensory and Motor pathways
- 23. Wiring the Brain
- 25. Neurotransmitters • Neurotransmitters are endogenous chemicals which transmit signals from a neuron to a target cell across a synapse. Excitatory Inhibitory Glutamate (memory storage) Gamma Amino Butyric Acid (GABA) (brain) Acetylcholine (neuro muscular Glycine (spinal cord) junction) Dopamine (brain reward system) Norepinephrine Serotonin (regulation of appetite, sleep, memory, learning, mood, behaviour) Substance P
- 26. Action Potential
- 27. Action Potential
- 28. Types of Neural Networks Based on Learning Algorithms Supervised and Unsupervised Associativity in Supervised Learning Auto Associative and Hetro Associative Based on Network Topology Feed forward and feedback / recurrent Based on kind of data accepted Categorical variables, Quantitative variables Based on transfer function used Linear, Non-linear Based on number of layers Single Layer, Multilayer
- 29. ANN Architecture Taxonomy Linear Hebbian, Perceptron, Adaline, Higher Order, Functional Link MLP (Multilayer Back Propagation, Cascade Correlation, Quick Prop, RPROP Perceptron) Feed RBF Networks Orthogonal Least Squares forward CMAC Cerebellar Model Articulation Controller LVQ (Learning Vector Quantization), PNN (Probabilistic Classification Only Neural Network) Regression Only GNN (General Regression Neural Network) Supervised BAM (Binary Associative Memory) Boltzmann Machine Feedback Back Propagation through time, Elman, FIR, Jordan, Real time Recurrent Time recurrent network, Recurrent Back propagation, TDNN (Time Series Delay Neural Nets) Competitive ARTMAP, Fuzzy ARTMAP, Gaussian ARTMAP, Counter propagation, Neocognitron Vector Quantization Grossberg, Kohonen, Conscience ANN Self Organizing Map Kohonen, GTM, Local Linear Competitive Adaptive Resonance Theory ART1, ART2, ART2A, ART3, Fuzzy ART DCL (Differential Competitive Learning) Unsupervised Dimension Reduction Hebbian, Oja, Sanger, Differential Hebbian Auto Association Linear Auto Associator, BSB (Brain State in a Box), Hopfield Non learning Hopfield, various networks for optimization
- 30. Learning Rules • Error correction learning • Memory based learning • Hebbian learning • Competitive learning • Boltzmann learning
- 31. Error Correction Learning • Error signal: ek(n) = dk(n) – yk(n) • Control mechanism to apply a series of corrective adjustments • Index of performance or instantaneous value of Error Energy: E(n) = ½ ek2(n) • Delta rule or Widrow-Hopf rule – Thus Δwkj(n) = ηek(n)xj(n) • And wkj(n+1) = wkj(n) + Δwkj(n) • Using unit delay operator: wkj(n) = z-1[wkj(n+1)]
- 32. Euclidean Distance • Ordinary distance between two points that can be measured with a ruler. • In multi dimensional case it is the distance between two vectors.
- 33. Memory based learning • Binary pattern classification : – with input output pairs {(xi,di)}Ni=1 • Nearest Neighbor Rule – xN’ є {x1, x2, …, xN} – If mini d(xi, xtest) = d(xN’, xtest) – Where d(xi, xtest) is the Euclidean distance between the vectors xi and x test. • Cover and Hart (1967): nearest neighbor rule for pattern classification. Assumptions are: – The classified examples (xi, di) are independently and identically distributed according to the joint probability distribution of the example – The sample size N is infinitely large Then the probability of classification error is bounded by twice the Bayes probability of error. , the minimum probability of error over all decision rules. • Radial basis function network for curve fitting (approximation problem in higher dimensional space)
- 34. Hebbian Learning • Repeated or persistent firing changes synaptic weight due to increased efficiency • Associative learning at cellular level – Time dependent mechanism – Local mechanism – Interactive mechanism – Conjunctional or correlational mechanism – Here Δwkj(n) = F(yk(n), xj(n)) – Hebb’s hypothesis : Δwkj(n) = η yk(n)xj(n) – Covariance hypothesis: Δwkj(n) = η (yk – yav)(xj(n)-xav) • Synaptic modifications can be Hebbian, Anti-Hebbian, or non- Hebbian. • Evidence for Hebbian learning in the Hippocampus which plays an important role in learning and memory
- 35. Competitive Learning • The O/P neurons compete among themselves to become active • Elements of competitive learning rule (Rumelhart and Zisper (1985)) – Sets of neurons are same except randomly distributed synaptic weights – Limit on strength of each neuron – Winner takes all mechanism • Use as feature detectors • Has feed forward (excitatory connections) • Has lateral (inhibitory) connections • Here Δwkj(n) = η(xj – wkj) if neuron k wins • = 0 if neuron k loses
- 36. Boltzmann Learning • Stochastic model of a neuron – x = +1 with probability P(v) – = -1 with probability 1- P(v) – P = 1/(1+ exp(-v/T) – T is pseudo temperature use to control uncertainty in firing (noise level) • Stochastic learning algorithm for statistical mechanics • Neurons in recurrent structure • Operate in binary manner • Energy function – Here E= -1/2 Σ Σ wkjxkxj • Flip a random neuron from state xk to state –xk at some temperature with probability • P(xk -> -xk) = 1/(1+exp(- ΔEk/T))
- 37. Credit Assignment Problem in Distributed Systems • Assignment of credit or blame for overall outcome to internal decisions • Credit assignment problem has two parts: – Temporal Credit Assignment Problem – Structural Credit Assignment Problem • Credit Assignment problem becomes more complex in multilayer feed forward neural nets.
- 38. Supervised Learning • Knowledge is represented by a series of input-output examples • Environment provides training vector to both teacher and Neural Network • Teacher or Trainer provides Desired response • Neural Network provides Actual response • Error Signal = Desired response – Actual response • Adjustment is carried out iteratively to make the neural network emulate the teacher. • The mean square error function can be visualized as a multidimensional error-performance surface with the free parameters as coordinates. • Identification of local or global minimum is done using steepest gradient descent method.
- 39. Reinforcement learning/ Neuro-dynamic Programming (Learning with a Critic) • Critic converts a primary reinforcement signal from environment to a heuristic reinforcement signal • system learns under delayed reinforcement after observation of temporal sequences • goal is to minimize the cumulative cost of actions over a sequence of steps • Problems: – No teacher to provide desired response – Learning machine must solve temporal credit assignment problem • Reinforcement learning is related to Dynamic Programming
- 40. Unsupervised Learning (Self Organized Learning) • No external teacher or critic • Provision for task independent measure of quality of learning • Free parameters are optimized with respect to that measure • Network becomes tuned to statistical regularities in data • It develops ability to form internal representations for encoding features of input and create new classes automatically • Competitive Learning rule is used for Unsupervised learning • Two layers: input layer and competitive layer
- 41. Learning Applications • Pattern Association • Pattern Recognition • Function Approximation • Control • Filtering • Beam forming
- 42. Pattern Association • Cognition uses association in distributed memory : – xk -> yk ; key pattern -> memorized pattern – Two phases: • storage phase (training) • recall phase (noisy or distorted version of key pattern presented) • y= yj (Perfect recall) • y ≠ yj for x =xj (error) • Two types: – Auto associative memory: • Output set of patterns is the same as input set: yk = xk • Used for pattern retrieval • Input and output spaces have same dimensionality • Uses unsupervised learning – Hetero associative memory: • Output set of patterns is the different from input set: yk ≠ xk • Used in other Pattern Association • Input and output spaces may or may not have same dimensionality • Uses supervised learning
- 43. Pattern Recognition • Process whereby a received pattern is assigned to a prescribed number of classes (categories) • Two stages: – Training Session – New patterns • Patterns can be considered as points in multidimensional decision space (MDS) • MDS is divided into regions, each associated with a class • Decision boundaries are determined by the training process • Boundary definition is by a statistical mechanism due to variability between classes • Machine has two parts: – Feature Extraction (Unsupervised network) – Classification (Supervised network) – m-dimensional observation (data) space -> q-dimensional feature space -> r dimensional decision space • Approaches: – Single layered feed forward network using a supervised learning algorithm – Feature extraction is done in the hidden layer
- 44. Function Approximation • I/O mapping: d=f(x) • Function f(.) is unknown • Set of labeled examples are available – T= {(xi, di)}N i=1 • ||F(x) –f(x)|| < ε for all x • Used in – System model identification – Inverse system model identification
- 45. Control • Ref signal is compared with feedback signal • Error signal e is fed to neural network controller • O/P of NNC u is fed to plant as input • Plant output is y (part of which is sent as feedback) • J={dyk/duj} (partial differential) • Two approaches: – Indirect Learning – Direct Learning
- 46. Filtering • To extract information from noisy data • Filter used for: – Filtering (for getting current data based on past data) – Smoothing (for getting current data based on future data) – Prediction (for forecasting future data based on current and past data) • In filtering – Cocktail party problem – Blind signal separation – Here x(n) = A u(n), were A = mixing matrix – Need a de mixing W to recover the original signal • In prediction – Error correction learning – x(n) provides the desired response and used for training – A form of model building, where network acts as model – When prediction is non-linear; NNs are a powerful method because non-linear processing units can be used for its construction – However if dynamic range of the time series is unknown, linear output unit is the most reasonable choice
- 47. Beam forming • Spatial form of filtering • To provide attentional selectivity in the presence of noise • Used in radar and sonar systems • Detect and track a target of interest in the presence of receiver noise and interfering signals (e.g. from jammers) • Task is complicated by: – Target signal can be from an unknown direction – No prior information about interfering signals • Generalized Side Lobe Canceller (GSLC) consisting of: – Array of antenna elements: which samples the observed signals – A linear combiner: acts as a spatial filter and provides the desired response (i.e. for main lobe) – A signal blocking matrix: to cancel leakage from side lobes – A neural network : to accommodate variations in interfering signals • Neural network adjusts its free parameters and acts as an attentional neurocomputer.
- 48. Associative Memory • Memory is relatively enduring neural alterations induced by the interaction of an organism with its environment. • Activity must be stored in memory through a learning process • Memory may be short term or long term • Associative memory – Distributed – Stimulus (key) pattern and response (stored) pattern vectors – Information is stored in memory by setting up a spatial pattern of neural activities across a large number of neurons – Information in stimulus also contains storage location and address for retrieval – High degree of resistance to noise and damage of a diffusive kind – May be interactions between different patterns stored in memory and thus errors in recall process
- 49. Memory and noise • For a linear network yk = W(k)xk • Total experience gained M = Σk=1..q W(k) • Memory matrix Mk = Mk-1 + W(k); k = 1..q • Estimate of memory matrix Me = Σk=1..q ykxkT • Correlation matrix memory Me= YXT • X = key matrix; Y = memorized matrix • Recall : y= Mxj • y = yj + vj ; vj = noise vector is due to cross talk between key vector xj and all other key vectors stored in memory • For a linear signal space cosine of angle between vectors xj and xk cos(xk,xj) = xkTxj/(|xk|.|xj|) • Noise vector vj = Σk=1..m cos(xk,xj)yk
- 50. Orthogonality, Community and Errors • The memory associates perfectly (noise vector is zero) when the key vectors are orthogonal, i.e. xkTxj = {1 when k=j and 0 when k≠j} • If key patterns are not orthogonal or highly separated it leads to confusion and errors • Community of set of patterns {xkey } can be such that xkTxj >= ᵞ for k≠j • If the lower bound ᵞis large enough, the memory may fail to distinguish the response y from any other key pattern contained in the set {xkey}
- 51. Adaptation • Spatiotemporal nature of learning • Temporal structure of experience from insects to humans, thus animal can adapt its behavior • In time-stationary environment, – supervised learning possible, – synaptic weights can be frozen after learning – learning system relies on memory • In non-time-stationary environments – supervised learning inadequate – network needs a way to track the statistical variations in environment with time – desirable for neural network to continually adapt its free parameters to respond in real time – this requires continuous learning – Linear adaptive filters perform continuous learning • Used in radar, sonar, communications, seismology, biomedical signal processsing • In a mature state of development • Nonlinear adaptive filters, development not yet mature.
- 52. Pseudo stationary process • Neural network requires stable time for computation • How can it adapt to signals varying in time? • Many non stationary processes change slowly enough for the process to be considered pseudo stationary over a window of short enough duration. – Speech signal: 10 – 30 ms – Radar returns from ocean surface: few seconds – Long range weather forecasting: few minutes – Long range stock market trends: few days • Retrain network at regular intervals, dynamic approach – Select a window short enough for data to be considered pseudo stationary – Use the sampled data to train the network – Keep data samples in a FIFO, add new sample and drop oldest data sample – Use updated data window to retrain and repeat • Network undergoes continual training with time ordered examples • Non linear filter : a generalization of linear adaptive filters • Resources available must be fast enough to complete the compute in one sampling period.
- 53. Rosenblatt’s perceptron • Type: feed forward • Neuron layers: 1 I/P, 1 O/P • Input value types: binary • Activation function: Hard Limiter • Learning method: Supervised • Learning Algorithm: Hebb’s learning rule • Used in: Simple logic operations; pattern classification
- 55. Perceptron
- 56. Perceptron Convergence Theorem • 1: Initialization : set w(0) = 0 • 2: Activation: at time step n, activate the perceptron by applying continuous valued input vector x(n) and desired response d(n) • 3: Computation of Actual Response: Compute the actual response of the perceptron – y(n) = sgn(wT(n)x(n)) • 4: Adaptation of weight vector: Update the weight vector of the perceptron: – w(n+1) = w(n) + η[d(n) – y(n)]x(n) – Where – D(n) = +1 if x(n) belongs to class C1 – = -1 if x(n) belongs to class C2 • Continuation: Increment time step n by one and go back to step 2
- 57. LMS Rule • Also known as: – Delta rule – Adaline rule – Widrow Hopf rule
- 58. Neural Network Hardware • Hardware runs orders of magnitude faster than software • Two approaches: – General, but probably expensive, system that can be reprogrammed for many kinds of tasks • e.g. Adaptive Solutions CNAPS – Specialized but cheap chip to do one thing very quickly and efficiently. • e.g. IBM ZISC • Number of neurons vary from 10 to 10**6 • Precision is mostly limited to 16 bit fixed point for weights and 8 bit fixed point for outputs • Recurrent NNs may require output of >16 bits • Performance is measured in – number of multiply and accumulate operations in unit time (MCPS: millions of connections per second) – Rate of weight updates (MCUPS: millions of connections update per second)
- 59. NN Hardware categories • Neurocomputers – Standard chips • Sequential + Accelerator • Multiprocessor – Neuro chips • Analog • Digital • Hybrid
- 60. Hardware Implementation (Accelerator Boards) • Accelerator boards – Most frequently used neural commercial hardware • Relatively cheap • Widely available • Simple to connect to PCs, workstations • Have user friendly software tools • However usually specialized for certain tasks and may lack flexibility – Based on neural network chips • IBM ZISC036 : 36 neurons; RBF network; RCE (or ROI algorithm) • PCI card: 19 chips, 684 prototypes, • Can process 165,000 patterns per second; where patterns are 64 8-bit element vectors. • SAIC Sigma-1 • Neuro Turbo • HNC – Some use just fast DSPs
- 61. Hardware Implementation (General Purpose Processors) • Neuro computers built from general purpose Processors – BSP400 – COKOS – RAP (Ring Array Processor) • Used for development of connectionist algorithms for speech recognition • 4 to 40 TMS320C20 DSPs • Connected via ring of Xilinx FPGAs • VME bus to connect to host computer • 57 MCPS in feed forward mode • 13.2 MCPS in back propagation training