Sigmoid function machine learning made simple
- 1. Sigmoid Function: Machine Learning Made Simple Devansh
- 2. Overview Actually a family of functions. All functions have a characteristic "S"- shaped curve or sigmoid curve. The most famous example is the logistic function. Other big ones are tanh and arc tan. By their nature, they can be used to “squish” input, making them useful in Machine Learning
- 3. Why and When we use the Sigmoid As mentioned earlier, sigmoids can “squish” input. They can take input with very high values and map it to a much more manageable range. In neural networks, we see them being used as activation functions. We can also put them in the last layer the network, converting model output into probabilities. This is done to make the output easier to understand and interpret. Sigmoid functions are often taught hand in hand with logistic regression. The logistic function is a sigmoid.
- 4. Example of some sigmoid functions
- 5. Sigmoid and the Vanishing Gradient(why we have a ReLU) Since the value of a sigmoid function converges to a limit, we know that the derivative of the function converges to 0. This can lead to the vanishing gradient problem (remember what it is?). Also, 0 as a derivative makes training very slow. How can we work around this? In modern artificial neural networks, it is common to see in place of the sigmoid function, the rectifier, also known as the rectified linear unit, or ReLU, being used as the activation function. The ReLU is defined as: f(x)= max(0,x)
- 6. Some more about ReLU Supposed to be closer to biological reality (doesn’t get triggered for values below a threshold). Simple function->Fast Calculation. Since the gradient is always 1, we avoid both the vanishing gradient and the learning rate slowdown. Depending on the context, we can add a slight bias to the gradient for our needs.
- 7. Reach out to me Check out my other articles on Medium. : https://machine-learning-made- simple.medium.com/ My YouTube: https://rb.gy/88iwdd Reach out to me on LinkedIn: https://www.linkedin.com/in/devansh-devansh-516004168/ My Instagram: https://rb.gy/gmvuy9 My Twitter: https://twitter.com/Machine01776819 My Substack: https://devanshacc.substack.com/ Live conversations at twitch here: https://rb.gy/zlhk9y Get a free stock on Robinhood: https://join.robinhood.com/fnud75