Neural Networks Basic Concepts and Deep Learning
- 1. Course Outcomes After completion of this course, students will be able to Understand machine-learning concepts. Understand and implement Classification concepts. Understand and analyse the different Regression algorithms. Apply the concept of Unsupervised Learning. Apply the concepts ofArtificial Neural Networks.
- 2. Topics Biological Neurons and Biological Neural Networks Artificial Neural Networks Perceptron Activation Functions Applications ofArtificial Neural Networks (ANNs) Neural Network Types of Artificial Neural Networks Feedforward Neural Networks (FNN) / Multi-Layer Perceptron (MLP) Convolutional Neural Networks (CNN) Recurrent Neural Networks (RNN) Transformer Neural Networks Autoencoders Generative Adversarial Networks (GANs) Competitive Neural Networks
- 3. Biological Neurons and Biological Neural Networks
- 4. Biological Neurons These are the nerve cells in your brain and nervous system. Each neuron has dendrites (which receive signals), a cell body (which processes the signals), and an axon (which sends signals to other neurons). Neurons communicate with each other through electrical impulses, forming complex networks to help us think, move, feel, and learn.
- 5. Biological Neural Networks A biological neural network is a collection of interconnected neurons in your brain. These networks are responsible for everything you do, like recognizing faces, remembering things, or solving problems. The neurons are connected by synapses. When you learn something new, the connections between certain neurons strengthen, which helps you remember or perform tasks better.
- 8. ANN ANNs are computer systems designed to mimic how biological neurons work, but they’re made up of math, not cells. An artificial neuron takes in information, processes it, and sends an output, much like how a biological neuron works. When many artificial neurons are connected together, they form an artificial neural network, which can learn to do things like recognizing objects in pictures, predicting outcomes, or playing video games.
- 9. ANN The term "Artificial neural network" refers to a biologically inspired sub-field of artificial intelligence modeled after the brain. An Artificial neural network is usually a computational network based on biological neural networks that construct the structure of the human brain. Similar to a human brain has neurons interconnected to each other, artificial neural networks also have neurons that are linked to each other in various layers of the networks.These neurons are known as nodes.
- 10. The given figure illustrates the typical diagram of Biological Neural Network. The typical Artificial Neural Network looks something like the given figure.
- 11. The given figure illustrates the typical diagram of Biological Neural Network. Biological Neural Network Artificial Neural Network Dendrites Inputs Cell nucleus Nodes Synapse Weights Axon Output The typical Artificial Neural Network looks something like the given figure.
- 12. Perceptron A ‘Perceptron’ is the basic building block, or single node, of a neural network inspired from the neurons that are found in the brain. It operates by taking in a set of inputs, calculating a weighted sum, adding a bias term, and then applying an activation function to this sum to produce an output.
- 14. The inner working of a perceptron is as follows:
- 15. Perceptron Learning Perceptron learning refers to how a perceptron adjusts its weights to improve accuracy. When the perceptron makes a wrong prediction, it learns by changing the weights to get closer to the correct answer next time. Over time, through repeated adjustments, the perceptron learns to make better predictions.
- 17. Activation Function Activation : In biological neurons, activation is the firing rate of the neuron which happens when the impulses are strong enough to reach the threshold. In artificial neural networks, A mathematical function known as an activation function maps the input to the output, and executes activations.
- 19. Activation Functions The purpose of an activation function is to introduce non-linearity into the model, allowing the network to learn and represent complex patterns in the data. The activation function decides whether a neuron should be activated or not by calculating the weighted sum and further adding bias to it. The purpose of the activation function is to introduce non-linearity into the output of a neuron.
- 23. Linear Function Equation : Linear function has the equation similar to as of a straight line i.e. y = x No matter how many layers we have, if all are linear in nature, the final activation function of last layer is nothing but just a linear function of the input of first layer. Range : -inf to +inf Uses : Linear activation function is used at just one place i.e. output layer. Issues : If we will differentiate linear function to bring non-linearity, result will no more depend on input “x” and function will become constant, it won’t introduce any ground-breaking behavior to our algorithm. For example : Calculation of price of a house is a regression problem. House price may have any big/small value, so we can apply linear activation at output layer. Even in this case neural net must have any non-linear function at hidden layers.
- 25. RELU Function It Stands for Rectified linear unit. It is the most widely used activation function. Chiefly implemented in hidden layers of Neural network. Equation :- A(x) = max(0,x). It gives an output x if x is positive and 0 otherwise. Value Range :- [0, inf) Nature :- non-linear, which means we can easily backpropagate the errors and have multiple layers of neurons being activated by the ReLU function. Uses :- ReLu is less computationally expensive than tanh and sigmoid because it involves simpler mathematical operations. At a time only a few neurons are activated making the network sparse making it efficient and easy for computation. In simple words, RELU learns much faster than sigmoid and Tanh function.
- 28. Tanh Function The activation that works almost always better than sigmoid function is Tanh function also known as Tangent Hyperbolic function. It’s actually mathematically shifted version of the sigmoid function. Both are similar and can be derived from each other. Equation :- f(x) = tanh(x) = 2/(1 + e-2x) – 1 OR tanh(x) = 2 * sigmoid(2x) – 1 Value Range :- -1 to +1 Nature :- non-linear Uses :- Usually used in hidden layers of a neural network as it’s values lies between -1 to 1 hence the mean for the hidden layer comes out be 0 or very close to it, hence helps in centering the data by bringing mean close to 0.This makes learning for the next layer much easier.
- 30. Sigmoid Function It is a function which is plotted as ‘S’ shaped graph. Equation : A = 1/(1 + e-x) Nature : Non-linear. Notice that X values lies between -2 to 2, Y values are very steep. This means, small changes in x would also bring about large changes in the value ofY. Value Range : 0 to 1 Uses : Usually used in output layer of a binary classification, where result is either 0 or 1, as value for sigmoid function lies between 0 and 1 only so, result can be predicted easily to be 1 if value is greater than 0.5 and 0 otherwise.
- 32. Softmax Function The softmax function is also a type of sigmoid function but is handy when we are trying to handle multi- class classification problems. Nature :- non-linear Uses :- Usually used when trying to handle multiple classes. the softmax function was commonly found in the output layer of image classification problems.The softmax function would squeeze the outputs for each class between 0 and 1 and would also divide by the sum of the outputs. Output:- The softmax function is ideally used in the output layer of the classifier where we are actually trying to attain the probabilities to define the class of each input.
- 35. Applications of Artificial Neural Networks (ANNs): Image Recognition (e.g., face detection, object classification) Speech Recognition (e.g., virtual assistants like Siri and Alexa) Natural Language Processing (NLP) (e.g., language translation, text generation) Medical Diagnosis (e.g., detecting diseases from medical images or records) Financial Predictions (e.g., stock market forecasting, fraud detection) AutonomousVehicles (e.g., self-driving cars, traffic sign recognition) Recommender Systems (e.g., Netflix, Amazon,YouTube recommendations) Robotics (e.g., robot vision, control systems) Customer Support Chatbots (e.g., automating responses to queries) Game AI (e.g.,AI playing video games or board games like Go) Time Series Forecasting (e.g., weather prediction, sales forecasting) Anomaly Detection (e.g., cybersecurity, equipment failure detection) Art Generation (e.g., creating artwork, music composition) Social Media Monitoring (e.g., sentiment analysis, spam detection) Personalized Marketing (e.g., targeted advertising, customer behavior prediction)
- 36. Neural Network
- 37. Perceptron is a single layer neural network and a multi-layer perceptron is called Neural Networks.
- 38. Neural Network
- 39. Neural Network This Neural Network or Artificial Neural Network has multiple hidden layers that make it a multilayer neural Network and it is feed-forward because it is a network that follows a top-down approach to train the network. In this network there are the following layers: Input Layer Hidden Layer Output Layer
- 41. Neural Network
- 42. Neural Network
- 43. Neural Network
- 44. Neural Network
- 45. Neural Network
- 46. Neural Network
- 47. Neural Network The basic rule of thumb is if you really don’t know what activation function to use, then simply use RELU as it is a general activation function in hidden layers and is used in most cases these days. If your output is for binary classification then, sigmoid function is very natural choice for output layer. If your output is for multi-class classification then, Softmax is very useful to predict the probabilities of each classes.
- 49. Types of Artificial Neural Networks Feedforward Neural Networks (FNN) / Multi-Layer Perceptron (MLP) Convolutional Neural Networks (CNN) Recurrent Neural Networks (RNN) Transformer Neural Networks Autoencoders Generative Adversarial Networks (GANs) Competitive Neural Networks
- 50. Feedforward Neural Networks (FNN) / Multi- Layer Perceptron (MLP) These are the simplest neural networks where information flows in one direction: from the input to the output. Think of it like a funnel—you give some input at the top (like numbers), and the network processes the input layer by layer until it reaches a final decision at the output (like yes/no, or classifying an image). Example: You give it an image, and it tells you whether it’s a cat or a dog.
- 51. Convolutional Neural Networks (CNN) CNNs are special neural networks designed for image data. They have layers that scan the image piece by piece to find patterns like edges, colors, or shapes, which helps the network understand the content of the image. Example: CNNs are used in applications like facial recognition or identifying objects in photos.
- 52. Recurrent Neural Networks (RNN) RNNs are used when you deal with sequences of data (like sentences, time series, or speech). These networks remember what they processed earlier in the sequence, allowing them to make decisions based on both current input and past inputs (like having a short-term memory). Example: RNNs can be used to predict the next word in a sentence or recognize spoken words in speech.
- 53. Transformer Neural Networks: Transformers are powerful networks for processing language or sequential data. Unlike RNNs, they look at the whole sentence at once instead of one word at a time, which makes them faster and more accurate. They use something called self-attention to focus on important parts of the input sequence. Example: GPT-3 (the model behind chatbots like this) is a transformer, used for generating text, answering questions, or translating languages.
- 54. Autoencoders Autoencoders are used for tasks like compressing data or finding patterns. They take input data, reduce it to a simpler version (compression), and then try to rebuild it back to its original form. Example: Autoencoders are used to compress images into smaller files or clean noisy data.
- 55. Generative Adversarial Networks (GANs) GANs involve two neural networks working together. One tries to create fake data (like fake images), and the other tries to detect which data is real and which is fake. Over time, the generator network gets better at making realistic fake data, while the discriminator gets better at spotting fakes. Example: GANs are used to create realistic fake images, like generating pictures of people who don’t exist.
- 56. Competitive Neural Networks In competitive networks, neurons compete with each other, and only the most active one is “activated.” These networks are often used in clustering tasks where similar data points are grouped together. Example: Self-Organizing Maps (SOMs) are a type of competitive network used to cluster data into groups, like finding similar patterns in large datasets.
- 57. Type of Neural Network Purpose Structure Best For How ItWorks Feedforward Neural Networks (FNN) / Multi-Layer Perceptron (MLP) Basic neural network for general tasks like classification and regression. Data moves in one direction, layer by layer (input → hidden → output). Classifying simple data like numbers or basic images. Takes input, processes it through layers, and gives a prediction (e.g., "cat" or "dog"). Convolutional Neural Networks (CNN) Specialized for image and video data. Uses layers that look at parts of an image (like scanning) to detect patterns (edges, shapes). Recognizing objects in images, like face detection or medical image analysis. Scans parts of the image to learn what it contains (e.g., looks for edges, colors, textures). Recurrent Neural Networks (RNN) Processes sequences of data, where order matters. Includes loops to remember past data (short-term memory). Time-series data, speech recognition, text prediction (e.g., what comes next). Remembers previous data (like words in a sentence or past events in time) to make a decision. Transformer Neural Networks Fast and efficient for language and sequential tasks. Processes whole sequences at once (no need for memory loops). Natural language processing (NLP) like translation, text generation. Looks at all words in a sentence together and figures out their relationships using "self- attention." Autoencoders Compresses data, finds patterns, or reduces noise. Has an encoder to shrink the input and a decoder to reconstruct it. Data compression, noise reduction (like removing background noise). Compresses the input into a smaller form and then tries to rebuild it to match the original. GenerativeAdversarial Networks (GANs) Generates new, realistic data like images or videos. Has two networks: a generator (creates fake data) and a discriminator (detects fake data). Creating realistic images, videos, or even music (like deepfakes). The generator makes fake data, and the discriminator tries to catch the fakes, making the generator improve over time. Competitive Neural Networks Groups similar data into clusters without supervision. Neurons compete to be the most active; only one "wins" and is activated. Clustering similar patterns in data, like organizing large datasets into groups. Neurons compete, and the "winner" learns from the data, helping to group similar data points together.