![Training Data Set:
It's the set of data used to train the model.
During each epoch, our model will be trained over and over again on this same data
in our training set, and it will continue to learn about the features of this data.
A set of data used for learning is to fit the parameters [i.e., weights] of the network.
Validation Set:
The validation set is a set of data, separate from the training set, that is used to
validate our model during training.
This validation process helps give information that may assist us with adjusting our
hyperparameters.
A set of data used to tune the parameters [i.e., architecture, number of hidden
units, Storage capacity] of the network.
Test set:
The test set is a set of data that is used to test the model after the model has
already been trained.
The test set is separate from both the training set and validation set.
A set of data is used only to assess the performance [generalization] of a fully
specified network or apply successfully to predict output whose input is known.](https://image.slidesharecdn.com/ann-unit3-240127100139-677744bb/85/Artificial-Neural-Networks-Recurrent-networks-Perceptron-s-19-320.jpg)
![Weights should be initializedwith small, randomly chosenvalues:
The initialization of weights with 0, there will be no changein weights
at all.
If they are all initialized by the same value, they will all change equally
during training.
The simple solution of this problem is called symmetry breaking,
which is the initialization of weights with small random values.
The range of random values could be the interval [-0.5 to 0.5] not
including 0 or values very close to 0.
This random initializationhas a nice effect.](https://image.slidesharecdn.com/ann-unit3-240127100139-677744bb/85/Artificial-Neural-Networks-Recurrent-networks-Perceptron-s-129-320.jpg)
Artificial Neural Networks , Recurrent networks , Perceptron's
- 1. ARTIFICIAL NEURAL NETWORKS UNIT - III Department: AIML-A,D Staff Name: Ms.L.Sasikala SRM INSTITUTE OF SCIENCE AND TECHNOLOGY, CHENNAI
- 2. Unit - III Fundamentals on Learning and training samples Paradigms of Learning Using training samples Gradient Optimization Procedure Hebbian learning rule Supervised learning network paradigms: The perceptron, back propagation and its Variants The single-layer perceptron Linear Separability The multilayer perceptron Backpropagation of error Selecting learning rate Resilient Backpropagation Adaption of weights Further variations and extensions to Backpropagation Initial configuration of a multilayer perceptron
- 3. Fundamentals on Learning and training samples Introduction to Learning: The primary significance for a neural network is the ability of the network to learn from its environment and to improve its performance through learning. The network becomes more knowledgeable about its environment after each iteration of the learning process. The type of learning is determined by the way the parameter changes take place. The learning process implies the following sequence of events: The neural network is stimulated by an environment. The neural network undergoes changes in its parameters as a result of this stimulation. The neural network responds in a new way to the environment because of the changes that have occurred in its internal structure.
- 4. Paradigms of learning: The most interesting characteristic of neural networks is their capability to familiarize with problems by means of training and, after sufficient training, to be able to solve unknown problems of the same class. This approach is referred to as generalization. Learning is a comprehensive term and learning system changes itself in order to adapt to environmental changes. A neural network could learn by • Developing new connections, • Deleting existing connections, • Changing connecting weights, • Changing the threshold values of neurons, • Varying one or more of the three neuron functions (activation function, propagation function and output function), • Developing new neurons, • Deleting existing neurons.
- 5. Our task is to find the weights that most accurately map our input data to the correct output class. This mapping is what the network must learn. After passing all of our data through our model, we're going to continue passing the same data over and over again. This process of repeatedly sending the same data through the network is considered training.
- 6. Epochs: One Epoch is when an ENTIRE dataset is passed forward and backward through the neural network only ONCE. Since one epoch is too big to feed to the computer at once we divide it in several smaller batches. We need to pass the full dataset multiple times to the same neural network. So, updating the weights with single pass or one epoch is not enough. Batch Size: Total number of training examples present in a single batch. Batch size and number of batches are two different things. We can’t pass the entire dataset into the neural net at once. So, we divide dataset into Number of Batches or sets or parts.
- 7. Iterations: Iterations is the number of batches needed to complete one epoch. The number of batches is equal to number of iterations for one epoch. Let’s say we have 2000 training examples that we are going to use . We can divide the dataset of 2000 examples into batches of 500 then it will take 4 iterations to complete 1 epoch. Where Batch Size is 500 and Iterations is 4, for 1 complete epoch.
- 8. Training set, Training patterns and Teaching input: A learning procedure is always an algorithm that can easily be implemented by means of a programming language. A training set consisting of set of training patterns. A training pattern is a set of pairs of input patterns with corresponding output pattern used to begin training a neural network. The teaching input tj is the desired and correct value j should output after the input of a certain training pattern. For a neuron j with the incorrect output oj ,tj is the teaching input, which means it is the correct or desired output for a training pattern p.
- 9. Summary: There is the input vector x, which can be entered into the neural network. Depending on the type of network being used the neural network will output an output vector y. Basically, the training sample p is nothing more than an input vector. We only use it for training purposes because we know the corresponding teaching input t which is nothing more than the desired output vector to the training sample. The error vector / difference vector Ep is the difference between the teaching input t and the actual output y under a training input p
- 10. Offline Learning: The weight vector adjustment and threshold adjustment depend overall (training) dataset, defining a global cost. The learning algorithm updates its parameters after consuming the whole batch. It is also called Batch Learning. Online Learning: The adjustment of the weight and threshold is made after presenting each training sample to the network. The learning algorithm updates its parameters after learning from 1 training instance. It is also called Incremental learning.
- 11. Epoch: Suppose that we need to train a machine learning model with some data, that data you call training data. Huge sets of training data cannot feed the whole bunch to the model at once due to limitations in computer memory. So, we break up the whole training data set into sizeable batches which can fit into computer’s memory at once. We then feed these batches one by one to the model for training. One forward pass and one backward pass of all batches exactly once , we call it has an epoch. Basically, it is equivalent to showing the model, the whole training data bunch once. Now, we must carry this one multiple times for successful training, hence, multiple epochs. For example, if there are 20,000 images of data (training set) into 500 batch size and number of batches is 4, so the iteration is 4 to complete 1 epoch.
- 12. Underfit, Overfit and Best fit:
- 13. Overfitting is the situation where any given model is performing too well on the training data but the performance drops significantly over the test set is called an overfitting model. On the other hand, underfitting is the situation if the model is performing poorly over the test and the train set, then we call that an underfitting model.
- 15. Visualization of training results of the same training set on networks with a capacities
- 16. Using training samples: Following successful learning, it's especially interesting to see if the network has just memorized. If it can utilize our training examples to create the correct output but, give incorrect answers for all other problems in the same class. Suppose that we want the network to train a mapping The network has sufficient storage capacity to concentrate on the six training samples with the output 1 and exactly mark the areas around the training samples (Image on top). On the other hand, a network could have insufficient capacity this rough presentation of input data does not correspond to the good generalization performance we desire (Image on bottom). Thus, we must find the balance (Image on middle).
- 17. An often-proposed solution for these problems is to divide, the training set (70% for training data) into one training set really used to train and the verification set (30% for verification data) to test our progress if there are enough training samples. We can finish the training when the network provides good results on the training data as well as on the verification data. If the verification data is poor, don't change the network structure until the verification data is positive; otherwise, you risk customizing the network to the verification data. The solution is a third set of validation data used only for validation after a supposably successful training. Order of pattern representation: There is no guarantee that model will be learned equally well if patterns are presented in a random order. When employing recurrent networks, on the other hand, the same sequence of patterns causes the patterns to be memorized. A random permutation would solve both concerns, however calculating such a permutation takes a long time.
- 18. Learning Data Sets in Artificial Neural Networks:
- 19. Training Data Set: It's the set of data used to train the model. During each epoch, our model will be trained over and over again on this same data in our training set, and it will continue to learn about the features of this data. A set of data used for learning is to fit the parameters [i.e., weights] of the network. Validation Set: The validation set is a set of data, separate from the training set, that is used to validate our model during training. This validation process helps give information that may assist us with adjusting our hyperparameters. A set of data used to tune the parameters [i.e., architecture, number of hidden units, Storage capacity] of the network. Test set: The test set is a set of data that is used to test the model after the model has already been trained. The test set is separate from both the training set and validation set. A set of data is used only to assess the performance [generalization] of a fully specified network or apply successfully to predict output whose input is known.
- 20. Supervised learning: Supervised learning is a process of providing labelled input data as well as correct output data to the machine learning model. The aim of a supervised learning algorithm is to find a mapping function to map the input variable(x) with the output variable(y). In supervised learning, the training set consists of input patterns as well as their correct results in the form of the precise activation of all output neurons. Thus, for each training set that is fed into the network the output, for instance, can directly be compared with the correct solution and the network weights can be changed according to their difference. The training data provided to the machines work as the supervisor that teaches the machines to predict the output correctly. It applies the same concept as a student learns in the supervision of the teacher.
- 21. How Supervised Learning Works? In supervised learning, models are trained using labelled dataset, where the model learns about each type of data. Once the training process is completed, the model is tested on the basis of test data (a subset of the training set), and then it predicts the output. Suppose we have a dataset of different types of shapes which includes square, rectangle, triangle, and Polygon. We need to train the model for each shape: o If the given shape has four sides, and all the sides are equal, then it will be labelled as a square. o If the given shape has three sides, then it will be labelled as a triangle. o If the given shape has six equal sides, then it will be labelled as hexagon. Now, after training, we test our model using the test set, and the task of the model is to identify the shape. The machine is already trained on all types of shapes, and when it finds a new shape, it classifies the shape on the basis of a number of sides, and predicts the output.
- 24. Types of supervised machine learning algorithms: Supervised learning can be further divided into two types of problems:
- 25. Regression: Regression algorithms are used if there is a relationship between the input variable and the output variable. Below are some popular Regression algorithms which come under supervised learning: • Linear Regression • Logistic Regression • Ridge Regression • Lasso Regression • Polynomial Regression
- 26. Classification: Classification algorithms are used when the output variable is categorical, which means there are two classes such as Yes-No, True- false, etc. Below are some popular Classification algorithms which come under supervised learning: • K-nearest neighbor (KNN) • Decision Trees • Naive Bayes • Support vector Machines (SVM)
- 27. Applications of supervised learning: • Text categorization • Face Detection • Signature recognition • Customer discovery • Spam detection • Weather forecasting • Predicting housing prices based on the prevailing market price
- 28. Advantages of Supervised learning: • Supervised learning in Machine Learning allows you to collect data or produce a data output from the previous experience. • Supervised machine learning helps you to solve various types of real- world computation problems. Disadvantages of supervised learning: • Supervised learning models are not suitable for handling the complex tasks. • Supervised learning cannot predict the correct output if the test data is different from the training dataset. • Training required lots of computation times.
- 29. Unsupervised learning: Unsupervised learning is a machine learning technique in which models are not supervised using training dataset. Instead, models itself find the hidden patterns and insights from the given data. Unsupervised learning works on unlabelled and uncategorized data which make unsupervised learning more important. The training set only consists of input patterns, the network tries by itself to detect similarities and to generate pattern classes. The goal of unsupervised learning is to find the underlying structure of dataset, group that data according to similarities, and represent that dataset in a compressed format. In real-world, we do not always have input data with the corresponding output so to solve such cases, we need unsupervised learning. It can be compared to learning which takes place in the human brain while learning new things.
- 30. How Unsupervised Learning works: We have taken an unlabelled input data, which means it is not categorized and corresponding outputs are also not given. Now, this unlabelled input data is fed to the machine learning model in order to train it. Firstly, it will interpret the raw data to find the hidden patterns from the data and then will apply suitable algorithms. The algorithm is never trained upon the given dataset, divides the data objects into groups according to the similarities and difference between the objects. Suppose the unsupervised learning algorithm is given an input dataset containing images of different types of cats and dogs. The task of the unsupervised learning algorithm is to identify the image features on their own.
- 32. Types of Unsupervised machine learning algorithm: The unsupervised learning algorithm can be further categorized into two types of problems:
- 33. Clustering: • Clustering is a method of grouping the objects into clusters such that objects with most similarities remains into a group and has less or no similarities with the objects of another group. • Cluster analysis finds the commonalities between the data objects and categorizes them as per the presence and absence of those commonalities. • Below are some popular Clustering algorithms which come under unsupervised learning: • Centroid-based Clustering • Density-based Clustering • Distribution-based Clustering • Hierarchical Clustering
- 34. Association: • An association rule is an unsupervised learning method which is used for finding the relationships between variables in the large database. • It determines the set of items that occurs together in the dataset. • People who buy X item (suppose a bread) are also tend to purchase Y (Butter/Jam) item. • Below is the popular Association algorithm which come under unsupervised learning: • Apriori algorithm
- 35. Applications of unsupervised learning algorithms: • Fraud detection • Malware detection • Identification of human errors during data entry • Conducting accurate basket analysis, etc.
- 36. Advantages of Unsupervised Learning: • Unsupervised learning solves the problem by learning the data and classifying it without any labels. • This type of learning is like human intelligence in some way as the model learns slowly and then calculates the result. Disadvantages of Unsupervised Learning: • The result of the unsupervised learning algorithm might be less accurate as input data is not labelled, and algorithms do not know the exact output in advance. • The more the features, the more the complexity increases. • The learning phase of the algorithm might take a lot of time, as it analyses and calculates all possibilities.
- 37. Reinforcement learning: In reinforcement learning, the network receives a logical or a real value after completion of a sequence, which defines whether the result is right or wrong. It is clear that this procedure should be more effective than unsupervised learning since the network receives specific criteria for problem-solving. The training set consists of input patterns, after completion a value is returned to the network indicating whether the result was right or wrong. Reinforcement Learning is defined as a Machine Learning method that is concerned with how software agents should take actions in an environment.
- 39. Some important terms used in Reinforcement AI: • Agent(): An entity that can perceive/explore the environment and act upon it. • Environment(): A situation in which an agent is present or surrounded by. In RL, we assume the stochastic environment, which means it is random in nature. • Action(): Actions are the moves taken by an agent within the environment. • State(): State is a situation returned by the environment after each action taken by the agent. • Reward(): A feedback returned to the agent from the environment to evaluate the action of the agent. • Policy(): Policy is a strategy applied by the agent for the next action based on the current state. • Value(): It is expected long-term retuned with the discount factor and opposite to the short-term reward.
- 40. How Reinforcement Learning works? Consider the scenario of teaching new tricks to your cat As cat doesn’t understand English or any other human language, we can’t tell her directly what to do. Your cat is an agent that is exposed to the environment, it is your house. An example of a state could be your cat sitting, and you use a specific word in for cat to walk. Our agent reacts by performing an action transition from one “state” to another “state.” For example, your cat goes from sitting to walking. The reaction of an agent is an action, and the policy is a method of selecting an action given a state in expectation of better outcomes. After the transition, they may get a reward (fish) or penalty in return.
- 42. Types of Reinforcement machine learning algorithms: Two kinds of reinforcement learning methods are: Positive: It is defined as an event, that occurs because of specific behaviour. It increases the strength and the frequency of the behaviour and impacts positively on the action taken by the agent. This type of Reinforcement helps you to maximize performance and sustain change for a more extended period. Negative: Negative Reinforcement is defined as behaviour that occurs because of a negative condition which should have stopped or avoided.
- 43. Reinforcement Learning Algorithms: The main used algorithms are: Q-Learning State-Action-Reward-State-Action algorithm (SARSA) Monte carlo Deep Q network
- 44. Applications of Reinforcement learning: Traffic Light Control Robotics Games Healthcare Finance Image processing Marketing
- 45. Advantages of Reinforcement learning: Reinforcement learning can be used to solve very complex problems that cannot be solved by conventional techniques. Once an error is corrected by the model, the chances of occurring the same error are very less. In the absence of a training dataset, it is bound to learn from its experience. Disadvantages of Reinforcement learning: • Too much reinforcement learning can lead to an overload of states, which can diminish the results. • Reinforcement learning is not preferable to use for solving simple problems. Reinforcement learning needs a lot of data and a lot of computation.
- 46. Gradient optimization procedures: • Gradient descent is an optimization algorithm which is commonly-used to train machine learning models and neural networks, for finding a local minimum of a differentiable function. • Until the function is close to or equal to zero, the model will continue to adjust its parameters to yield the smallest possible error or cost function.
- 47. How does gradient descent work? The starting point is just an arbitrary point for us to evaluate the performance. From that starting point, we will find the derivative (or slope), and from there, we can use a tangent line to observe the steepness of the slope. The slope will inform the updates to the parameters—i.e. the weights and bias. The slope at the starting point will be steeper, but as new parameters are generated, the steepness should gradually reduce until it reaches the lowest point on the curve, known as the point of convergence. The goal of gradient descent is to minimize the cost function, or the error between predicted and actual value.
- 49. Learning rate or step size - is the size of the steps that are taken to reach the minimum. This is typically a small value, and it is evaluated and updated based on the behaviour of the cost function. High learning rates result in larger steps but risks overshooting the minimum. A low learning rate has small step sizes. While it has the advantage of more precision, the number of iterations compromises overall efficiency as this takes more time and computations to reach the minimum. The cost function or loss function - measures the difference or error, between actual value and predicted value at its current position. This improves the machine learning model's efficacy by providing feedback to the model so that it can adjust the parameters to minimize the error and find the local or global minimum.
- 51. The gradient is a vector g that is defined for any differentiable point of a function, steepest ascent is g. The gradient is a generalization of the derivative for multi- dimensional functions. The negative gradient -g exactly points towards the steepest descent. The gradient is referred to as nabla operator ∇ The overall notation of the gradient g of the point (x, y) of a two- dimensional function f being g(x, y) =∇f (x, y). Let g be a gradient, then g is a vector with n components that is defined for any point ofa differential n-dimensionalfunction f(x1, x2, . . . , xn). The gradient operator notation is defined as g(x1, x2, ... ,xn)=∇f(x1,x2,. ..,xn)
- 52. How to calculate Gradient Descent? Here are the steps of finding minimum of the function using gradient descent: Calculate the gradient by taking the derivative of the function with respect to the specific parameter. In case, there are multiple parameters, take the partial derivatives with respect to different parameters. Calculate the descent value for different parameters by multiplying the value of derivatives with learning rate (step size) and -1. Update the value of parameter by adding up the existing value of parameter and the descent value.
- 54. The below represents the updating of parameter θ with the value of gradient in the opposite direction while taking small steps.
- 55. The gradient of a scalar-valued multivariable function g(x,y,…) denoted by ∇f(x,y,…)
- 56. As the name suggests minimum is the lowest value in a set and maximum is the highest value. Global means it is true for the entire set and local means it is true in some vicinity. A function can have multiple local maxima and minima. However, there can be only one global maximum as well as minimum.
- 58. Possible errors during gradient descent: (a) - Every gradient descent procedure can get stuck within local minimum. This problem is increasing proportionally to the size of the error surface, and there is no universal solution. (b) - Flat plateau on the error surface has less slope may cause training slowness. (c) – Steep canyons in the error surface may cause oscillation. A sudden alternation from very strong negative gradient to a very positive one, result in oscillation. Such error does not occur often. (d) – The gradient is very large at a steep slope, so that the large steps can be made, and a good minimum can be possibly be missed.
- 61. Hebbian learning rule: In 1949, Donald O. Hebb formulated the Hebbian rule, which is the basis for most of the complicated learning rules. Hebbian rule "If neuron j receives an input from neuron i and if both neurons are strongly active at the same time, then increase the weight wi,j”. The rule is: with being the change in weight from i to j, which is proportional to the following factors: The output oi of the predecessor neuron i, as well as, The activation aj of the successor neuron j A constant η, i.e. the learning rate
- 63. The changes in weight are simply added to the weight wi,j Hebb proposed that If two interconnected neurons on either side of a synapse are both on or fired or activated are at the same time(synchronously), then the synaptic weight between them should be increased. If two interconnected neurons on either side of a synapse are both on or fired or activated are at the different time(asynchronously), then the synaptic weight between them should be decreased. Such a synapse is called Hebbian Synapse. The generalized form of the Hebbian Rule specifies the proportionality of the change in weight to the product of two undefined functions, but with defined input values. Thus, the product of the functions. Changes in weight = Learning rate . Pre synaptic signal . Post synaptic signal
- 64. The perceptron, backpropagation and its variants Introduction: Perceptron was described by Frank Rosenblatt in 1958. Rosenblatt defined that the weighted sum and a non-linear activation function as components of the perceptron.
- 65. Architecture of perceptron with one layer of variable connections
- 66. An input neuron is an identity neuron. It exactly forwards the information received. The input neuron is represented by the symbol Information processing neuron processes the input information, i.e., do not represent the identity function. A binary neuron sums up all inputs by using the weighted sum as propagation function, which we want to illustrate by the sign Σ. Then the activation function of the neuron is the binary threshold function, which can be illustrated by Other neurons that use the weighted sum as propagation function but the activation functions hyperbolic tangent or Fermi function, or with a separately defined activation function fact, are similarly represented by
- 67. The perceptron is a feed forward network containing a retina that is used only for data acquisition and which has fixed-weighted connections with the first neuron layer (input layer). The fixed-weight layer is followed by at least one trainable weight layer. One neuron layer is completely linked with the following layer. The first layer of the perceptron consists of the input neurons. The first neuron layer is often understood as input layer, because this layer only forwards the input values. The retina itself and the static weights behind it are no longer mentioned or displayed, since they do not process information in any case. So, the depiction of a perceptron starts with the input neurons
- 68. A single layer perceptron Introduction: A single layer perceptron (SLP) is a perceptron having only one layer of variable weights and one layer of output neurons Ω. Connections with trainable weights go from input layer to an output neuron Ω, which returns the information whether the pattern entered at the input neurons was recognized or not. Certainly, the existence of several output neurons Ω 1, Ω2. . . Ω n does not considerably change the concept of the perceptron. A perceptron with several output neurons can also be regarded as several different perceptron with the same input.
- 71. Perceptron learning algorithm: The original perceptron learning algorithm with binary neuron activation function is described in algorithm. It has been proven that the algorithm converges in finite time, so in finite time the perceptron can learn anything. Suppose that we have a single layer perceptron with randomly set weights which we want to teach a function by means of training samples. The set of these training samples is called P. It contains, as already defined, the pairs (p, t) of the training samples p and the associated teaching input t.
- 73. x is the input vector y is the output vector of a neural network Output neurons are referred to as i is the input value of a neuron o is the output value of a neuron The error vector Ep represents the difference (t−y) under a certain training sample p. O be the set of output neurons I be the set of input neurons. Our learning target will be certainly be, that for all training samples the output y of the network is approximately the desired output t,
- 74. Learning in neural network: Learn values of weights from I/O pairs Start with random weights Load training example’s input Observe computed input Modify weights to reduce difference Iterate over all training examples Terminate when weights stop changing OR when error is very small
- 75. The error function regards the set of weights W as a vector and maps the values onto the normalized output error. It is obvious that a specific error function can analogously be generated for a single pattern p. Err(W) is defined on the set of all weights which we here regard as the vector W. Change in all weights is referred to as ΔW. ΔW is calculated by the gradient ∇Err(W) of the error function Err(W):
- 76. We derive the error function according to a weight wi,Ω and obtain the value ∆wi,Ω of how to change this weight. The squared distance between the output vector y and the teaching input t appears adequate to our needs. It provides the error Errp that is specific for a training sample p over the output of all outputneurons Ω Thus, we calculate the squared difference of the components of the vectors t and y, given the pattern p, and sum up these squares.
- 77. The summation of the specific errors Errp(W ) of all patterns p then yields the definition of the error Err and there fore the definition of the error functionErr(W ): We tweak the individual weights wi,Ω a bit and see how the error Err(W ) is changing – which corresponds to the derivative of the error function Err(W ) according to the very same weight wi,Ω. This derivative corresponds to the sum of the derivativesof all specific errors Errp according to this weight (since the total error Err(W ) results from the sum of the specific errors):
- 78. Basically, the data is only transferred through a function, the result of the function is sent through another one, and so on. The path of the neuron outputs oi1 and oi2 , which the neurons i1 and i2 entered into aneuron Ω, initially is the propagation function (here weighted sum), from which thenetwork input is going to be received. This is then sent through the activation function of the neuron Ω so that we receive the output of this neuron which is at the same time a component of the output vector y:
- 79. Propagation function & Network input
- 80. Let I = i1, i2, . . . , in be the set of neurons, such that Then the network input of j, called netj, is calculated by the propagation function fprop as follows: The multiplication of the output of each neuron i by wi,j, and the summation of the results represents netj
- 81. netΩ → fact = fact(netΩ) = oΩ = yΩ As we can see, this output results from many nested functions: oΩ = fact(netΩ) = fact(oi1 · wi1,Ω + oi2 ·wi2,Ω) We want to calculate the derivatives ofequation and due to the nested functions we can apply the chain rule to factorize the derivative
- 82. The examination of Errp clearly shows that this change is exactly the difference between teaching input and output (tp,Ω - op,Ω) Since Ω is anoutput neuron, op,Ω = yp,Ω. The closer the output is to the teaching input, the smaller is the specific error. This difference is also called δp,Ω . We know, op,Ω = yp,Ω
- 84. The second multiplicative factor of equation and of the following one is the derivative of the output specific to the pattern p of the neuron Ω according to the weight wi,Ω. Due to the requirement at the beginning of the derivation, we only have a linear activation function fact, therefore we can just as well look at the change of the network input when wi,Ω is changing:
- 86. We insert this in equation, which results in our modification rule for a weight wi,Ω
- 87. From the very beginning the derivation has been intended as an “offline rule” by means of the question of how to add the errors of all patterns and how to learn them after all patterns have been represented. Although this approach is mathematically correct, the implementation is far more time-consuming. The "online-learning version" of the delta rule simply omits the summation and learning is realized immediately after the presentation of each pattern.
- 88. Delta rule: If we determine, analogously to the aforementioned derivation, that the function h of the Hebbian theory only provides the output oi of the predecessor neuron i and if the function g is the difference between the desired activation tΩ and the actual activation aΩ or oΩ, we will receive the delta rule, also known as Widrow-Hoff rule: Apparently the delta rule only applies for SLPs, since the formula is always related to the teaching input, and there is no teaching input for the inner processing layers of neurons.
- 89. Linear Separability: Let f be the XOR function which expects two binary inputs and generates a binary output. Let us try to represent the XOR function by means of an SLP with two input neurons i1, i2 and one output neuron. We use the weighted sum as propagation function, a binary activation function with the threshold value and the identity as output function. Depending on i1 and i2has to output the value 1 if the following holds:
- 92. With a constant threshold value ƟΩ, the right part of in equation is a straight line through a coordinate system defined by the possible outputs oi1 und oi2 of the input neurons i1 and i2 For a positive wi2,Ω the output neuron fires for input combinations lying above the generated straight line. For a negative wi2,Ω it would fire for all input combinations lying below the straight line. A SLP is only capable of representing linearly separable data. Only sets that can be separated by a hyperplane, i.e. which are linearly separable, can be classified by an SLP. Thus, for more difficult tasks with more inputs we need something more powerful than SLP. The XOR problem itself is one of these tasks, since a perceptron that is supposed to represent the XOR function already needs a hidden layer.
- 102. Assume a simple SLP model with 3 neurons and inputs= 2, 2 and 2. The weights to the input neurons are 4, 4 and 4 respectively. Assume the activation function is a linear constant value of 3. What will be the output?
- 103. It is given that inputs of 3 neurons are 2, 2 and 2 corresponding weights are 4, 4 and 4. activation function (2 * 4 + 2 * 4 + 2 * 4) activation function is a linear constant value of 3. 3 (2 * 4 + 2 * 4 + 2 * 4) 3 (24) The output will be 72
- 104. A multilayer perceptron: A perceptron with two or more trainable weight layers is called multilayer perceptron or MLP. It is more powerful than an SLP. A single layer perceptron can divide the input space by means of a hyper plane (in a two-dimensional input space by means of a straight line).
- 105. A two stage perceptron (two trainable weight layers, three neuron layers) can classify convex polygons by further processing these straight lines. A multilayer perceptron represents an universal function approximator. Perceptron with more than one layer of variably weighted connections are referred to as multilayer perceptron (MLP). An n-layer or n-stage perceptron has thereby exactly n variable weight layers and n +1 neuron layers with neuron layer 1 being the input layer.
- 106. 3 weights layer or 3-stage perceptron and 4 neuron layers
- 109. Backpropagation of error: Backpropagation of error generalizes the delta rule to allow for MLP training. Backpropagation is a gradient descent procedure with the error function Err(W) receiving all n weights as arguments and assigning them to the output error, i.e. being n-dimensional. On Err(W) a point of small error or even a point of the smallest error is sought by means of the gradient descent. Thus, in analogy to the delta rule, backpropagation trains the weights of the neural network. And it is exactly the delta rule or its variable i for a neuron i which is expanded from one trainable weight layer to several ones by backpropagation.
- 110. Selecting learning rate: The selection of the learning rate has heavy influence on the learning process. The change in weight is proportional to the learning rate. Speed and accuracy of a learning procedure can always be controlled by and are always proportional to a learning rate which is written as η. If the value of the learning rate is too large, the jumps on the error surface are also too large. Additionally, the movements across the error surface would be very uncontrolled. A small η is the desired input, which, however, can cost a huge, often unacceptable amount of time. Experience shows that good learning rate values are in the range of
- 111. Variation of the learning rate over time: The selection of η significantly depends on the problem, the network and the training data. But for instance, it is popular to start with a relatively large , e.g. 0.9 and to slowly decrease it down to 0.1 For simpler problems can often be kept constant. Variable learning rate: In the beginning, a large learning rate leads to good results, but later it results in inaccurate learning. A smaller learning rate is more time-consuming, but the result is more precise. Thus, during the learning process the learning rate needs to be decreased by one order of magnitude once or repeatedly.
- 112. Different layers – Different learning rates: The farer we move away from the output layer during the learning process, the slower backpropagation is learning. Thus, it is a good idea to select a larger learning rate for the weight layers close to the input layer than for the weight layers close to the output layer.
- 113. Resilient backpropagation: Resilient backpropagation is an extension to backpropagation of error. We have two backpropagation specific properties that can occasionally be a problem: 1. Users of backpropagation can choose a bad learning rate η. 2. The further the weights are from the output layer; the slower backpropagation learns. MARTIN RIEDMILLER et al. enhanced backpropagation and called their version resilient backpropagation (short Rprop).
- 114. Learning rates: Backpropagation uses default learning rate η, which is selected by the user, and applies to the entire network. It remains static until it is manually changed. Here, Rprop pursues a completely different approach: Thereis no global learning rate. First, eachweight wi,j has its own learning rate ηi,j second, these learning ratesare not chosen by the user, but are automatically set by Rprop itself. Third,the weight changes are not static but are adapted for each time step of Rprop. To account for the temporal change, we must correctly call it ηi,j(t). This not only enables more focused learning, also the problem of an increasingly slowed down learning throughout the layers is solved in anelegant way.
- 115. Weight change: When using backpropagation, weights are changed proportionally to the gradient of the error function. Here, Rprop takes other waysas well: The amount of weight change ∆wi,j directly corresponds to the automatically adjusted learning rate ηi,j. Thus, the change in weight is not proportional to the gradient, it is only influenced by the sign of the gradient. The weight specific learning rates directly serve as absolute values for the changes of the respective weights. As with the derivation of backpropagation, we derive the error function Err(W ) by the individual weights wi,j and obtain gradients
- 116. We shorten the gradient to : If the sign of the gradient is positive, we must decrease the weight wi,j, so the weight is reduced by ηi,j. If the sign of the gradient is negative, the weightneeds to be increased, so ηi,j is added to it. If the gradient is exactly 0, nothing happens at all. The corresponding terms are affixed with a (t)to show that everything happens at the same time step.
- 117. Variations in Backpropagation: Backpropagation has often been extended: Backpropagation has often been extended and altered besides Rprop. Many of these extensions can simply be implemented as optional features of backpropagation in order to have a larger scope for testing. Adding momentum to learning: Let us assume to descent a steep slope on skis, what prevents us from immediately stopping at the edge of the slopeto the plateau? Exactly - our momentum. With backpropagation the momentum term is responsible for the fact that a kind of moment of inertia (momentum) is added to every step size by always adding a fraction of the previous change to every new change in weight:
- 119. The concept of time - when referring to the current cycle as (t), then the previous cycle is identified by (t - 1), The variation of backpropagation by means of the momentum term is defined as: We accelerate on plateaus (avoiding standstill on plateaus) and slow down on craggy surfaces (preventing oscillations). Moreover, the effect of inertia can be varied via the pre-factor α, common values are between 0.6 and 0.9. The momentum enables the positive effect that our skier swings back and forth several times in a minimum, and finally lands in the minimum.
- 121. Flat spot elimination prevents neurons from getting stuck: It must be pointed out that with the hyperbolic tangent as well as with the Fermi function the derivative outside of the close proximity of Θ is nearly 0. f(x) = This results in the fact that it becomes very difficult to move neurons away from the limits of the activation (flat spots), which could extremely extend the learning time. f'(x) = sigmoid(x)*(1-sigmoid(x)) This problem can be dealt with by modifying the derivative, for example by adding a constant (e.g. 0.1), which is called flat spot elimination or fudging.
- 122. Fermi activation function and hyperbolic tangent activation function
- 123. The second derivative can beused, too: According to DAVID PARKER, Second order backpropagation also uses the second gradient, i.e. the second multi-dimensional derivative of the error function, to obtain more precise estimates of the correct ∆wi,j. Even higher derivatives rarely improve the estimations. Thus, less training cycles are needed but those require much more computational effort. In general, we use further derivatives for higher order methods. As expected, the procedures reduce the number of learning epochs, but significantly increase the computational effort of the individual epochs. So in the end theseprocedures often need more learning time than backpropagation. If you have “n” weights in the neural network, one iteration of a second-order optimization algorithm will reduce the loss function at approximately the same rate as “n” iterations of a standard first-order optimization algorithm.
- 124. Weight decay - Punishment oflarge weights: The weight decay according to PAUL WERBOS is a modification that extends the error by a term punishing large weights. So, the error under weight decay does not only increase proportionally to the actual error but also proportionally tothe square of the weights. As a result, the network is keeping the weights small during learning. Additionally, due to these small weights, the error function often shows weaker fluctuations, allowing easier and more controlled learning. The pre-factor again resulted from simple pragmatics. The factor strength of punishment: Values from 0.001 to 0.02 are often used here.
- 125. Cutting networks down: Pruning and Optimal BrainDamage: If we have executed the weight decay longenough and notice that for a neuron in the input layer all successor weights are 0 or close to 0, we can remove the neuron, hence losing this neuron and some weightsand thereby reduce the possibility that the network will memorize. This procedure is called pruning. A method to detect and delete unnecessary weights and neurons is referred to as optimal brain damage (reducing the size of a learning network by selectively deleting weights) Two competing terms make up the mean error per output neuron: The first term is, if a weight is required to minimize the error, as we evaluate the difference between output and teaching input which is a customary. If this does not happen, the second term attempts to "push" a weight towards 0 (Weight decay). Neurons which only have zero weights can be pruned again in the end.
- 126. Getting started – Initial configuration of a multilayer perceptron: Number of layers: A network should have one layer of input neurons and one layer of output neurons, which results in at least two layers. If our problem is not linearly separable, then we need at least one hidden layer ofneurons. MLP with one hidden neuron layer is already capable of approximating arbitrary functions withany accuracy. Representability means that a perceptron can theoretically realize a mapping. Learnability indicates that we can train a perceptron to realize a mapping. Experience shows that two hidden neuron layers (or three trainable weight layers) can be very useful to solve a problem.
- 127. The number of neurons hasto be tested: The number of neurons principally corresponds to the number of free parameters (Neuron) of the problem to be represented. Since we have already discussed the network capacity with respect to memorizing, it is clear that our goal is to have as few free parameters as possible but as many as necessary. But we also know that there is no standard solution for the question of how many neurons should be used. Thus, the most useful approach is to initially train with only a few neurons and to repeatedly train new networks with more neurons until the result significantly improves and, particularly, the generalization performance is not affected.
- 128. Selecting an activationfunction: Another very important parameter for the way of information processing of a neural network is the selection of an activation function. The activation function for input neurons is fixed to the identity function, since they do not process information. The first question to be asked is whether we actually want to use the same activation function in the hidden layer and in the output layer – no one prevents us from choosing different functions. Generally, the activation function is the same for all hidden neurons as well as for the output neurons respectively. For tasks of function approximation it has been found reasonable to use the hyperbolic tangent as activation function of the hidden neurons, while a linear activation function is used in the output. However, linear activation functions in the output can also cause huge learning steps and jumping over good minima in the error surface. This can be avoided by setting the learning rate to very small values in the output layer.
- 129. Weights should be initializedwith small, randomly chosenvalues: The initialization of weights with 0, there will be no changein weights at all. If they are all initialized by the same value, they will all change equally during training. The simple solution of this problem is called symmetry breaking, which is the initialization of weights with small random values. The range of random values could be the interval [-0.5 to 0.5] not including 0 or values very close to 0. This random initializationhas a nice effect.
- 130. Dropout: Deep neural nets with a large number of parameters are very powerful machine learning systems. However, overfitting is a serious problem in such networks. Large networks are also slow to use, making it difficult to deal with overfitting by combining the predictions of many different large neural nets at test time. Dropout is a technique for addressing this problem. The key idea is to randomly drop units (along with their connections) from the neural network during training. This prevents units from co-adapting too much. We show that dropout improves the performance of neural networks on supervised learning tasks in vision, speech recognition, document classification and computational biology.