![Rectified liner unit Range 0 , x
Differentiations range 0 or 1
[tan 45 = 1]
Liki Rectified liner unit Range -x , x
Differentiations range -x or 1
Continued………..](https://image.slidesharecdn.com/activationfunction-191015105652/85/Activation-function-12-320.jpg)
![Weight update function
• Since we random initialize the weights we end up with some error.
• Now adjust your weights by differentiating your pervious output
• until you reach minimum error
Example : simple weight update
F(x) = x2 -3x + 2
df/dx = 0
df/dx = 2x – 3 + 0 = 0
x = 1.5 is this maxima or minima ?
put 1.5 in x2 – 3x + 2
f(1.5) = -0.25
f(1) = 0
Standard weight update functions
• SGD
• Min batch SGD
• SGD with momentum
• Ada grad
• Ada delta and RMS prop
• Adam [Adaptive momentum estimate ]Heart of gradient decent is chain rule in differentiation
When tangent is horizontal to x axis you may be at minima or maxima](https://image.slidesharecdn.com/activationfunction-191015105652/85/Activation-function-14-320.jpg)
![Vanishing Gradient in Sigmoid and Tanh functions
Dated from 1980 - 2012 people tried 2 - 3 layer neural networks
• Biggest problem faced is vanishing gradient [mathematical problem]
• Too little labelled data
• computation powers
History of activation function
We can get stuck in local minima or at saddle points](https://image.slidesharecdn.com/activationfunction-191015105652/85/Activation-function-15-320.jpg)
Activation function
- 2. Walk Through • Mathematical Functions • Types of functions • Activation function • Laws of activation function • Types of Activation functions • Limitations of activation function
- 3. What is Function ? Which takes some input and munch on it and generate some output relates an input to an output f(x) = 𝑥2 f(q) = 1 - q + q2 h(A) = 1 - A + A2 w(θ) = 1 - θ + θ2 Function
- 4. special rules: •It must work for every possible input value •And it has only one relationship for each input value x 1 3 8 𝑥2 1 9 16 1 3 8 1 9 16 The relationship x → x2
- 5. Kinds of Functions A. Function With Discontinuities Consider the function Factoring the denominator gives: We observe that the function is not defined for x=0 and x=1. Here is the graph of the function. We see that small changes in x near 0 (and near 1) produce large changes in the value of the function. We say the function is discontinuous when x = 0 and x = 1. plot(2/(x(x-1))
- 6. B. Continuous Functions • Consider the graph of f(x) = x3 − 6x2 − x + 30 a continuous graph. • We can see that there are no "gaps" in the curve. Any value of x will give us a corresponding value of y. We could continue the graph in the negative and positive directions, and we would never need to take the pencil off the paper. • Such functions are called continuous functions.
- 7. Activation function | Transfer Function
- 8. NN without activation function ! • A Neural Network without Activation function would simply be a Linear regression Model • Linear function is simple polynomial of one degree • Linear regression are easy to solve but they have less power to learn complex functional mappings from data. • We want our Neural Network to not just learn and compute a linear function but something more complicated than that • i.e complicated kinds of data such as images, videos , audio , speech etc.
- 9. Why activation function ? • we need to apply a Activation function f(x) so as to make the network more powerful • Add ability to it to learn something complex and complicated from the data • And represent non-linear complex arbitrary functional mappings between inputs and outputs. Output of one function is fed as input to next function so it can be approximated to function composition Picture 1 Picture 2
- 10. LAWS of Activation function Monotonic : A function is said to be monotonically increasing if X it’s corresponding Y also increases by some unit or remain constant for some time and then increases , monotonically decreasing function is vice versa it Monotonic increasing Monotonic decreasing Non monotonic Continues : A continues curve without any gape Easily differentiable : It should be smooth “continues” curve and differentiable at every point . Non continues .Curves can not be differentiated
- 11. Popular Activation functions Sigmoid Function Range 0 -1 Differentiation Range 0 -0.25 Tanh Function Range -1 + 1 Differentiation Range 0 -1
- 12. Rectified liner unit Range 0 , x Differentiations range 0 or 1 [tan 45 = 1] Liki Rectified liner unit Range -x , x Differentiations range -x or 1 Continued………..
- 13. Additional info : https://ml-cheatsheet.readthedocs.io/en/latest/activation_functions.html Continued………..
- 14. Weight update function • Since we random initialize the weights we end up with some error. • Now adjust your weights by differentiating your pervious output • until you reach minimum error Example : simple weight update F(x) = x2 -3x + 2 df/dx = 0 df/dx = 2x – 3 + 0 = 0 x = 1.5 is this maxima or minima ? put 1.5 in x2 – 3x + 2 f(1.5) = -0.25 f(1) = 0 Standard weight update functions • SGD • Min batch SGD • SGD with momentum • Ada grad • Ada delta and RMS prop • Adam [Adaptive momentum estimate ]Heart of gradient decent is chain rule in differentiation When tangent is horizontal to x axis you may be at minima or maxima
- 15. Vanishing Gradient in Sigmoid and Tanh functions Dated from 1980 - 2012 people tried 2 - 3 layer neural networks • Biggest problem faced is vanishing gradient [mathematical problem] • Too little labelled data • computation powers History of activation function We can get stuck in local minima or at saddle points
- 16. How to avoid vanishing gradient ? -- Rectified linear units ReLu • Partial derivation of relu is either 0 or 1 • Since max value is 1 there is no problem of exploding gradient and again values are not in between 0 -1 there is no vanishing gradient • Relu function converge faster then other function because value is either 1 or 0 • But it may lead into Dead activation • If z is negative f(z) = 0 • df/dz = 0 • If one of the derivative is zero then complete = Dead activation • Sigmoid and tanh may subjected to dead activation when you have very high negative values