An Introduction to Deep Learning (April 2018)
Download as PPTX, PDF6 likes711 views
The document provides an introduction to deep learning, including its definitions, common architectures, training processes, and resources. It emphasizes that AI is a mathematical and engineering field, debunking myths around its complexity. Key concepts cover neural networks, training optimization techniques, and practical applications like convolutional and recurrent networks for tasks such as image recognition and natural language processing.
An Introduction to Deep Learning (April 2018)
- 1. An Introduction to Deep Learning Julien Simon Principal Evangelist, Artificial Intelligence & Machine Learning @julsimon
- 2. What to expect • An introduction to Deep Learning • Common network architectures and use cases • Resources
- 3. • Artificial Intelligence: design software applications which exhibit human-like behavior, e.g. speech, natural language processing, reasoning or intuition • Machine Learning: teach machines to learn without being explicitly programmed • Deep Learning: using neural networks, teach machines to learn from complex data where features cannot be explicitly expressed
- 4. Myth: AI is dark magic aka « You’re not smart enough »
- 5. Fact: AI is math, code and chips A bit of Science, a lot of Engineering
- 6. An introduction to Deep Learning
- 7. Activation functions The neuron i=1 l xi ∗ wi = u ”Multiply and Accumulate” Source: Wikipedia
- 8. x = x11, x12, …. x1I x21, x22, …. x2I … … … xm1, xm2, …. xmI I features m samples y = 2 0 … 4 m labels, N2 categories 0,0,1,0,0,…,0 1,0,0,0,0,…,0 … 0,0,0,0,1,…,0 One-hot encoding Neural networks
- 9. x = x11, x12, …. x1I x21, x22, …. x2I … … … xm1, xm2, …. xmI I features m samples y = 2 0 … 4 m labels, N2 categories Total number of predictions Accuracy = Number of correct predictions 0,0,1,0,0,…,0 1,0,0,0,0,…,0 … 0,0,0,0,1,…,0 One-hot encoding Neural networks
- 10. Neural networks Initially, the network will not predict correctly f(X1) = Y’1 A loss function measures the difference between the real label Y1 and the predicted label Y’1 error = loss(Y1, Y’1) For a batch of samples: 𝑖=1 𝑏𝑎𝑡𝑐ℎ 𝑠𝑖𝑧𝑒 loss(Yi, Y’i) = batch error The purpose of the training process is to minimize error by gradually adjusting weights
- 11. Training Training data set Training Trained neural network Batch size Learning rate Number of epochs Hyper parameters Backpropagation
- 12. Stochastic Gradient Descent (SGD) Imagine you stand on top of a mountain with skis strapped to your feet. You want to get down to the valley as quickly as possible, but there is fog and you can only see your immediate surroundings. How can you get down the mountain as quickly as possible? You look around and identify the steepest path down, go down that path for a bit, again look around and find the new steepest path, go down that path, and repeat—this is exactly what gradient descent does. Tim Dettmers University of Lugano 2015 https://devblogs.nvidia.com/parallelforall/deep-learning-nutshell-history-training/ The « step size » is called the learning rate z=f(x,y)
- 13. Local minima and saddle points « Do neural networks enter and escape a series of local minima? Do they move at varying speed as they approach and then pass a variety of saddle points? Answering these questions definitively is difficult, but we present evidence strongly suggesting that the answer to all of these questions is no. » « Qualitatively characterizing neural network optimization problems », Goodfellow et al, 2015 https://arxiv.org/abs/1412.6544
- 15. Validation Validation data set Trained neural network Validation accuracy Prediction at the end of each epoch
- 16. Early stopping Training accuracy Loss function Accuracy 100% Epochs Validation accuracy Loss Best epoch OVERFITTING « Deep Learning ultimately is about finding a minimum that generalizes well, with bonus points for finding one fast and reliably », Sebastian Ruder
- 17. Common network architectures and use cases
- 18. Convolutional Neural Networks (CNN) Le Cun, 1998: handwritten digit recognition, 32x32 pixels Convolution and pooling reduce dimensionality https://devblogs.nvidia.com/parallelforall/deep-learning-nutshell-core-concepts/
- 19. Source: http://timdettmers.com Extracting features with convolution Kernel parameters are discovered during the training process.
- 20. Downsampling images with pooling Source: Stanford
- 23. Text Detection and Recognition https://github.com/Bartzi/stn-ocr
- 26. Long Short Term Memory Networks (LSTM) • A LSTM neuron computes the output based on the input and a previous state • LSTM networks have memory • They’re great at predicting sequences, e.g. machine translation
- 28. GAN: Welcome to the (un)real world, Neo Generating new ”celebrity” faces https://github.com/tkarras/progressive_growing_of_gans From semantic map to 2048x1024 picture https://tcwang0509.github.io/pix2pixHD/
- 29. Wait! There’s more! Models can also generate text from text, text from images, text from video, images from text, sound from video, 3D models from 2D images, etc.
- 30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Resources https://aws.amazon.com/machine-learning https://aws.amazon.com/blogs/ai https://gluon.mxnet.io http://ruder.io/optimizing-gradient-descent/ http://ruder.io/deep-learning-optimization-2017/ https://medium.com/@julsimon
- 31. Thank you! Julien Simon Principal Evangelist,Artificial Intelligence & Machine Learning @julsimon