What is activation function in the context of neural network?
- 1. Neural networks Feedforward neural network - activation function
- 2. ARTIFICIAL NEURON 2 Topics: connection weights, bias, activation function • Neuron pre-activation (or input activation): • Neuron (output) activation • are the connection weights • is the neuron bias • is called the activation function ... 1 September 6 Abstrac Math for my slides “Feedforward neural n • a(x) = b + P i wixi = b + w> x • h(x) = g(a(x)) = g(b + P i wixi) • x1 xd • w • { Septe Math for my slides “Feedforwar • a(x) = b + P i wixi = b + w> • h(x) = g(a(x)) = g(b + P i wi • x1 xd • w • { September 6, 2012 Abstract Math for my slides “Feedforward neural network”. • a(x) = b + P i wixi = b + w> x • h(x) = g(a(x)) = g(b + P i wixi) • x1 xd b w1 wd • w • { • g(·) b September 6, 2012 Abstract Math for my slides “Feedforward neural network”. • a(x) = b + P i wixi = b + w> x • h(x) = g(a(x)) = g(b + P i wixi) • x1 xd b w1 wd • w • { • g(·) b Abst Math for my slides “Feedforward neu • a(x) = b + P i wixi = b + w> x • h(x) = g(a(x)) = g(b + P i wixi) • x1 xd b w1 wd • w • { • g(·) b September 6, 2012 Abstract Math for my slides “Feedforward neural network”. • a(x) = b + P i wixi = b + w> x • h(x) = g(a(x)) = g(b + P i wixi) • x1 xd b w1 wd • w • { • g(·) b hugo.larochelle@usherbrooke.ca September 6, 2012 Abstract Math for my slides “Feedforward neural network”. • a(x) = b + P i wixi = b + w> x • h(x) = g(a(x)) = g(b + P i wixi) • x1 xd b w1 wd • w • { Abstract Math for my slides “Feedforward neural network”. • a(x) = b + P i wixi = b + w> x • h(x) = g(a(x)) = g(b + P i wixi) • x1 xd b w1 wd • w • { • g(·) b Abstract Math for my slides “Feedforward neural network”. x) = b + P i wixi = b + w> x (x) = g(a(x)) = g(b + P i wixi) 1 xd b w1 wd ·) b (x) = g(a(x)) iversité de Sherbrooke arochelle@usherbrooke.ca September 6, 2012 Abstract dforward neural network”. + w> x P i wixi)
- 3. ACTIVATION FUNCTION 3 Topics: linear activation function • Performs no input squashing • Not very interesting... Abstract Math for my slides “Feedforward neural n • a(x) = b + P i wixi = b + w> x • h(x) = g(a(x)) = g(b + P i wixi) • x1 xd b w1 wd • w • { • g(a) = a 1
- 4. ACTIVATION FUNCTION 4 Topics: sigmoid activation function • Squashes the neuron’s pre-activation between 0 and 1 • Always positive • Bounded • Strictly increasing Abstract Math for my slides “Feedforward neural network”. • a(x) = b + P i wixi = b + w> x • h(x) = g(a(x)) = g(b + P i wixi) • x1 xd b w1 wd • w • { • g(a) = a • g(a) = sigm(a) = 1 1+exp( a) exp(a) exp( a) exp(2a) 1
- 5. ACTIVATION FUNCTION 5 Topics: hyperbolic tangent (‘‘tanh’’) activation function • Squashes the neuron’s pre-activation between -1 and 1 • Can be positive or negative • Bounded • Strictly increasing Math for my slides “Feedforward neural network”. • a(x) = b + P i wixi = b + w> x • h(x) = g(a(x)) = g(b + P i wixi) • x1 xd b w1 wd • w • { • g(a) = a • g(a) = sigm(a) = 1 1+exp( a) • g(a) = tanh(a) = exp(a) exp( a) exp(a)+exp( a) = exp(2a) 1 exp(2a)+1
- 6. ACTIVATION FUNCTION 6 Topics: rectified linear activation function • Bounded below by 0 (always non-negative) • Not upper bounded • Strictly increasing • Tends to give neurons with sparse activities • x1 xd b w1 wd • w • { • g(a) = a • g(a) = sigm(a) = 1 1+exp( a) • g(a) = tanh(a) = exp(a) exp( a) exp(a)+exp( a) = exp(2a) 1 exp(2a)+1 • g(a) = max(0, a) • g(a) = reclin(a) = max(0, a)