An Introduction to Deep Learning (May 2018)
1 like472 views
This document provides an introduction to deep learning, including common network architectures and use cases. It defines artificial intelligence, machine learning, and deep learning. It discusses how neural networks are trained using stochastic gradient descent and backpropagation to minimize loss and optimize weights. Common network types are described, such as convolutional neural networks for image recognition and LSTM networks for sequence prediction. Examples of deep learning applications include machine translation, object detection, segmentation, and generation of images, text, and video. Resources for learning more about deep learning are provided.
An Introduction to Deep Learning (May 2018)
- 1. An Introduction to Deep Learning Julien Simon Principal Evangelist, Artificial Intelligence & Machine Learning @julsimon May 2018
- 2. What to expect • An introduction to Deep Learning • Common network architectures and use cases • Resources
- 3. • Artificial Intelligence: design software applications which exhibit human-like behavior, e.g. speech, natural language processing, reasoning or intuition • Machine Learning: teach machines to learn without being explicitly programmed • Deep Learning: using neural networks, teach machines to learn from complex data where features cannot be explicitly expressed
- 4. Myth: AI is dark magic aka « You’re not smart enough »
- 5. Fact: AI is math, code and chips A bit of Science, a lot of Engineering
- 6. An introduction to Deep Learning
- 7. Activation functionsThe neuron ! "#$ % xi ∗ wi = u ”Multiply and Accumulate” Source: Wikipedia
- 8. x = x11, x12, …. x1I x21, x22, …. x2I … … … xm1, xm2, …. xmI I features m samples y = 2 0 … 4 m labels, N2 categories 0,0,1,0,0,…,0 1,0,0,0,0,…,0 … 0,0,0,0,1,…,0 One-hot encoding Neural networks
- 9. x = x11, x12, …. x1I x21, x22, …. x2I … … … xm1, xm2, …. xmI I features m samples y = 2 0 … 4 m labels, N2 categories Total number of predictions Accuracy = Number of correct predictions 0,0,1,0,0,…,0 1,0,0,0,0,…,0 … 0,0,0,0,1,…,0 One-hot encoding Neural networks
- 10. Neural networks Initially, the network will not predict correctly f(X1) = Y’1 A loss function measures the difference between the real label Y1 and the predicted label Y’1 error = loss(Y1, Y’1) For a batch of samples: ! "#$ %&'() *"+, loss(Yi, Y’i) = batch error The purpose of the training process is to minimize error by gradually adjusting weights.
- 11. Training Training data set Training Trained neural network Batch size Learning rate Number of epochs Hyper parameters Backpropagation
- 12. Stochastic Gradient Descent (1951) Imagine you stand on top of a mountain with skis strapped to your feet. You want to get down to the valley as quickly as possible, but there is fog and you can only see your immediate surroundings. How can you get down the mountain as quickly as possible? You look around and identify the steepest path down, go down that path for a bit, again look around and find the new steepest path, go down that path, and repeat—this is exactly what gradient descent does. Tim Dettmers University of Lugano 2015 https://devblogs.nvidia.com/parallelforall/deep-learning-nutshell-history-training/ The « step size » depends on the learning rate z=f(x,y)
- 13. Local minima and saddle points « Do neural networks enter and escape a series of local minima? Do they move at varying speed as they approach and then pass a variety of saddle points? Answering these questions definitively is difficult, but we present evidence strongly suggesting that the answer to all of these questions is no. » « Qualitatively characterizing neural network optimization problems », Goodfellow et al, 2015 https://arxiv.org/abs/1412.6544
- 14. Optimizers https://medium.com/@julsimon/tumbling-down-the-sgd-rabbit-hole-part-1-740fa402f0d7 SGD works remarkably well and is still widely used. Adaptative optimizers use a variable learning rate. Some even use a learning rate per dimension (Adam).
- 15. Validation Validation data set (also called dev set) Neural network in training Validation accuracy Prediction at the end of each epoch This data set must have the same distribution as real-life samples, or else validation accuracy won’t reflect real-life accuracy.
- 16. Test Test data set Fully trained neural network Test accuracy Prediction at the end of experimentation This data set must have the same distribution as real-life samples, or else test accuracy won’t reflect real-life accuracy.
- 17. Early stopping Training accuracy Loss function Accuracy 100% Epochs Validation accuracy Loss Best epoch OVERFITTING « Deep Learning ultimately is about finding a minimum that generalizes well, with bonus points for finding one fast and reliably », Sebastian Ruder
- 18. Common network architectures and use cases
- 19. Convolutional Neural Networks (CNN) Le Cun, 1998: handwritten digit recognition, 32x32 pixels https://devblogs.nvidia.com/parallelforall/deep-learning-nutshell-core-concepts/
- 20. Source: http://timdettmers.com Extracting features with convolution Convolution extracts features automatically. Kernel parameters are learned during the training process.
- 23. Text Detection and Recognition https://github.com/Bartzi/stn-ocr
- 26. Long Short Term Memory Networks (LSTM) • A LSTM neuron computes the output based on the input and a previous state • LSTM networks have memory • They’re great at predicting sequences, e.g. machine translation
- 28. GAN: Welcome to the (un)real world, Neo Generating new ”celebrity” faces https://github.com/tkarras/progressive_growing_of_gans From semantic map to 2048x1024 picture https://tcwang0509.github.io/pix2pixHD/
- 29. Wait! There’s more! Models can also generate text from text, text from images, text from video, images from text, sound from video, 3D models from 2D images, etc.
- 31. https://aws.amazon.com/machine-learning https://aws.amazon.com/blogs/ai https://mxnet.incubator.apache.org | https://github.com/apache/incubator-mxnet https://gluon.mxnet.io | https://github.com/gluon-api https://medium.com/@julsimon https://youtube.com/juliensimonfr https://github.com/juliensimon/dlnotebooks Getting started
- 32. Thank you! Julien Simon Principal Evangelist, Artificial Intelligence & Machine Learning @julsimon