Deep learning crash course
- 1. Deep Learning Crash Course By : Vishwas Narayan
- 2. Deep Learning is Everywhere
- 6. What you should Learn is to ● Build a groundbreaking intelligence just like humans through Deep Learning ● Build Neural Network with your approach and also make them make some sustainable Decision ● Understand and give your own approach to give a effective training for the deep learning model
- 7. What is the Difference between the Ai and others Like 1. Deep Learning with ML and AI 2. Machine Learning with AI Here never forget somehow you train to get the model
- 8. What you will learn? ● Loss Function and Optimizers ● Gradient Descent Algorithm ● Neural Network Architecture
- 9. What is Deep Learning? Microsoft Word - Turing Test.doc (umbc.edu)
- 11. Machine learning is basically Teaching machine learn patterns in data
- 18. What is Deep Learning A Machine learning Technique that learn features and tasks directly from data. Inputs are run through “Neural Networks” Neural Network have hidden layers
- 19. Why Deep Learning? ● Machines never get Fatigue. ● They need to get trained from the Human intelligence - ● They just fetch patterns.
- 20. In Deep Learning Features can be learnt from Raw Data
- 21. What they Really mean to us? Black Box Neuron X Y Some functional Output X Y Some functional Output that is inspired by brain
- 26. Why do we need now? ● Data is Prevalent? ● Improved Hardware Architecture ● New Software Architecture
- 27. Neural Networks Inspired by the Neurons on the Brain.
- 28. Building Block of the Neural Network is Neuron
- 29. Neural Network ● Take data as the input ● Train themselves to understand the patterns in the data
- 30. A simple Neural Network
- 31. Learning Process of the Neural Network 1. Learning Process of the Neural Network 2. Forward Propagation 3. Back Propagation
- 33. Weights and Biases ● Weights - How important information can this neural network can get ● Bias - allow the right decision to be taken into consideration
- 35. In Backpropagation Loss function helps the Neural Network Quantify the deviation front he expected output.
- 36. Randomly Initialize the Parameters
- 37. Back Propagation ● Use of the Loss Function ● Go Backwards and self tune the initial weights and biases ● Values adjusted to better fit prediction of the model that is trained from the data.
- 38. Learning Algorithm for the Neural Network ● Initialize the parameters with some tuned and calculated value. ● Feed Input data to the Network ● Compare the Predicted value with the expected data and calculate loss ● Perform Backpropagation to propagate this loss back through the network. ● Update parameters based on the loss ● Iterate the previous steps till the loss is minimized.
- 39. Terms used in the Neural Network Activation Function ● Helps to decide whether a Neuron can be a drop out or can contribute to the next layer based on the dataset that it is trained on. ● Introduce non linearity into the Neural Network.
- 43. Which Activation Function to use? ● For the Binary Classification: Sigmoid or the Relu is Used for the best results. ● In the case of classifiers, sigmoid functions and their combinations often perform better. ● Because of the vanishing gradient issue, sigmoids and tanh functions are sometimes avoided. ● The ReLU function is a generic activation function that is employed in the majority of applications these days.
- 44. Activation Function Condition ● If we have dead neurons in our networks, the leaky ReLU function is the best option. ● Remember that the ReLU function should only be utilised in the hidden layers. ● As a general guideline, you should start with the ReLU function and then go on to other activation functions if the ReLU function does not produce the best results.
- 45. The king here is the data nad queen also No matter what you are training a model using a dataset that is available for you,Neural Network is as it becomes.
- 46. Loss Function We know that from the Random Weights and Biases the Neural Network Makes decision - Expected Output ,Weights and Biases are calculated. Thus they Quantify the deviation of the predicted output by the NEural NEtwork to the expected output.
- 47. The loss functions in Regression are ● Absolute Error Loss ● Huber Loss ● Squared Error Loss
- 48. The loss Function in Binary Classification Binary Cross Entropy Hinge Loss
- 49. Multi Class Classification loss functions Multi-Class Cross Entropy Loss KL(Kullback–Leibler) Divergence
- 50. Optimizers During the Training process we will adjust the parameters to minimize the loss function and make our model as optimized as possible for the Use.
- 51. Optimizers are basically A function that combines to the loss function and model parameters by updating the Neural Network based on the output of the Loss Function.
- 52. Gradient Descent Iterative function that starts off at a random Point on the loss function and travel down its slope in steps(learning rate -from user) until it reaches the lowest point of the function. 1. Again this depends on the data.but they are 2. Most popular optimizer 3. Fast,Robust,Flexible.
- 53. Algorithm in the Lay man Terms 1. Calculate what a small change in the each individual weights would do to the loss function 2. Adjust each parameter based int eh its gradient(differential) 3. Repeat Steps one and Two until lower loss function is calculated by the Neural Network.
- 55. To avoid getting stuck in the local minima We use Learning Rate ● Usually a small number that is multiplied to the scale of the gradients,which is any changes made to the weights are quite small. ● If we take large steps as learning rate then algorithm will tend to overshoot the global minimum ● Where we also don't want the algorithm to take forever to train and converge to the Global minimum.
- 57. They are more robust ,why? ● Like Gradient Descent,Except uses a subset of training example rather than the entire lot. ● SGD is Gradient descent that uses batch on each training. ● Use of the Momentum to Accumulate gradients. ● Less intensive computation as they are batched.
- 58. Backpropagation A simple implementation of the Gradient descent on the neural network.
- 59. AdaGrad ● Adaptive learning rate to individual features. ● Some weights will have different learning rates ● Ideal for the Sparse datasets with many input examples missing ● Learning rate tends to get lower accordingly.
- 60. Parameters and Hyperparameters ● What are model parameters? ● Variable from the neural Network whose values can be estimated from the data. ● Required by the model to make prediction ● Value define the learnt parameters from the data. ● Not set manually.saved as the Neural Network is trained. Example - Weights and Biases.
- 61. What are model Hyper parameters? ● They are configured externally to the neural network : Value cannot be estimated until we train the dataset on the neural network ● No clear way to find the best value ● When the DL algorithm is tuned ,you are really tuning the hyperparameters. ● This is tuned manually Example - Learning Rate,C and Alpha in SVM,Epochs,k in the kNEes
- 62. Summary Model parameters -> Estimated front the data Model Hyperparameters -> Can't be estimated from the data HyperParameters are often called as the parameters as they are a part of the Machine Learning that must set manually and tuned.
- 63. Epochs,Batches,Batch Size and Iterations Need to learn to do this to your Neural Network when the dataset is too Big. Break the dataset into smaller chunks and feed those chunks to the Neural Network One by One.
- 64. Epochs When the Entire dataset is passed forward to the Neural Network and only once they get trained in the Network. We use more than one epoch to help model generalize better and accurate. There is no absolute count for the dataset as its different for different datasets.
- 65. Batch and Batch size We divide large dataset into the smaller batches and feed those batches to the Neural Network Batch Size - Total number of the training examples in the Batches.
- 66. Iterations Number of Batches needed to complete one epoch Number f batches = Number of iterations in one epochs
- 67. Let's have some more insights Suppose we have 1 million Number if dataset as the Training Example and you divide the dataset into the batches of 500 ,to Complete 1 Epoch ,it would take 20000 iterations.
- 68. Conclusion for the terms used in NN’s How to design an Architecture? Which Activation Function to use? The only thing is to
- 69. Types of Learning There are Three Main Types - ● Supervised Learning ● Unsupervised Learning ● Reinforcement Learning
- 70. Supervised Learning ● Algorithms designed to learn from Examples. ● Models are trained on well-labelled data ● Each example has ● Input Object - Typically a Vector ● Desired Output Value Supervised Signal
- 71. During Training Searches the pattern and correlate with the desired output.
- 72. After Training Takes the unseen inputs and determine which label to classify it to.
- 73. Objective of a Supervised learning model Is to predict the correct label for the unseen data.
- 75. Supervised learning is of two types ● Classification ● Regression
- 76. Classification ● Take Input data and assign it to a class/category. ● Models finds features in the data that correlates to the class and creates a mapping function ● This mapping function will be used to classify unseen data from testing and the validation set from the cross validation of the data
- 77. Binary and Multiclass classification Definition and Example
- 78. Popular Classification Algorithms ● Logistic Regression. ● Naïve Bayes. ● Stochastic Gradient Descent. ● K-Nearest Neighbours. ● Decision Tree. ● Random Forest. ● Support Vector Machine
- 79. Regression Model tries to find a relationship between dependent and independent variable. Goal is always to predict continuous values such as a test score.
- 80. Equation is always continuous
- 82. Different Regression Algorithm ● Linear Regression. ● Logistic Regression. ● Ridge Regression. ● Lasso Regression. ● Polynomial Regression. ● Bayesian Linear Regression.
- 83. Application of Supervised Learning ● Text categorization ● Face Detection ● Signature recognition ● Customer discovery ● Spam detection ● Weather forecasting ● Predicting housing prices based on the prevailing market price ● Stock price predictions, among others
- 84. Unsupervised Learning ● Uses to manifest underlying pattern in data ● Used in Exploratory Data Analysis ● Need no labelled data,they use the feature from the Data
- 85. Unsupervised Learning is of ● Clustering ● Association
- 86. Clustering -Partitional Clustering ● Partitional Clustering ● Each Data point can belong to a single cluster
- 87. Clustering - Hierarchical Clustering ● Clusters within the clusters ● Datapoint may belong to different clusters
- 88. Association Attempts to find different relationship between the different entities. Example - Market Basket Analysis
- 89. Some Clustering Algorithm 1. Clustering Dataset 2. Affinity Propagation 3. Agglomerative Clustering 4. BIRCH 5. DBSCAN 6. K-Means 7. Mini-Batch K-Means 8. Mean Shift 9. OPTICS 10.Spectral Clustering 11.Gaussian Mixture Model
- 90. Application of the Unsupervised Learning ● Fraud detection ● Malware detection ● Identification of human errors during data entry ● Conducting accurate basket analysis, etc.
- 91. Reinforcement Learning Enable the intelligent entity to learn in an interactive environment by trial and error(by Policy and reward network) based on its own actions and experience. This is a very new way of getting the things learnt. If your Neural Network doesn't work well then you have to use the Reinforcement :Learning.
- 92. Reward and Punishment is the key here Uses the positive and negative signals as the behavior to understand what has been learnt.
- 93. Goal of Reinforcement learning is to ● Find a Suitable model that would maximize the total cumulative reward and make a very approximate result that might help in making some more discussion. ● Maximize the points won in a training over many examples ● Penalize when they make wrong decisions ● Reward where they make Right decision Usually modelled as a “Markov Decision Process”
- 95. Application of the Robotics ● Robotics ● Business strategy ● Traffic Light Control ● Web system configuration ● NLP ○ to personalize suggestions ○ deliver more meaningful notifications to users ○ optimize video streaming quality. ● Gaming ● Bidding
- 96. Some core and Canonical Problems in Deep Learning Basically we find this as the situation: Model should perform well on training data and new test data Most common problem faced will always be overfitting
- 97. So the data points as the example is here ● Data is Skew ● Data is Random ● They don't care anything and anybody they are just generated ● They are collected to make sense and make a model ● They are collected to make the right decision from the model
- 98. Underfitting
- 99. Over-fitting
- 100. Tackling Overfitting is 1. Hold-out 2. Cross-validation 3. Data augmentation 4. Feature selection 5. L1 / L2 regularization 6. Remove layers / number of units per layer 7. Dropout 8. Early stopping
- 102. Data Augmentation Just create some fake data as much as possible from the data itself.
- 104. Early Stopping Use Early Stopping to Halt the Training of Neural Networks At the Right Time (machinelearningmastery.com)
- 105. When do you need to do this? Training error decreases steadily but the validation error increases after a certain point.
- 106. Neural Network are in plenty So go the sources that I am saying in this stream.
- 107. So we talked a lot about the models So now let's get to know how we can build a model.
- 108. Gathering Data Picking the right data is very important,Good way to start is you need to make assumption about the data that you need.
- 109. Size of the data set also matters No one size fits all Amount of the data needed = 10 times the model parameters
- 110. Quality of the data also matters Data has to be more Accurate and Reliable with no Adversaries. Noiseless Features.
- 111. Some Dataset Repositories are I will list out here.
- 112. Pre- Processing dataset Split the dataset into the subset. Training data set Testing Data Set Validation Data Set We can randomly split the dataset
- 113. This process depends on ● Number of the samples in eh data ● Model Being Trained
- 114. Simple rule of thumb ● Few Hyperparameters means small validation set ● Many hyperparameters means large validation set
- 115. The ratio in which you split the dataset is specific to your Use Case
- 118. Look for the missing data ● Nan or Null ● Eliminated Features or the Missing Value ● Impute the Missing data
- 119. Sampling Use a sample of the dataset
- 120. Why we need this ● Faster Convergence ● Reduces the Disk Space
- 121. Preprocessing is Required for the FEature Scaling ● Crucial Step for the Model TRaining: ● Normalization ● Standardization Then obviously train and Evaluate.
- 122. Optimization Will be Continued ...