Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)
2 likes1,258 views
The document provides an overview of deep neural networks, including concepts like perceptrons, multilayer perceptrons, and convolutional neural networks (CNNs). It discusses the limitations of linear decision boundaries, the importance of activation functions, and the universal approximation theorem that supports the use of deep networks in complex function approximation. Additionally, it presents examples from the MNIST digit classification and highlights key features of CNNs such as filtering and pooling layers.
![Convnets
●
●
LeNet-5 [LeCun et al. 1998] 32](https://image.slidesharecdn.com/dlmmdcud1l02deepneuralnetworks-170427160932/85/Deep-Neural-Networks-D1L2-Insight-DCU-Machine-Learning-Workshop-2017-32-320.jpg)
Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)
- 1. Eva Mohedano eva.mohedano@insight-centre.org PhD Student Insight Centre for Data Analytics Dublin City University Deep Neural Networks
- 4. Linear decision decision boundary Class 0 Class 1 x
- 5. Linear decision decision boundary Class 0 Class 1 Parameters of the line. They are find based on training data - Learning Stage. x
- 6. Limitations: XOR problem Input 1 Input 2 Desired Output 0 0 0 0 1 1 1 0 1 1 1 0 XOR logic table 1 0 0 1 Input 1 Input 2 ?
- 9. Principle of deep learning ● ○ ○ ○ ● ○ ○ ○ ● ○ ○ Data Pixels Deep learning Convolutional Neural Network Data Pixels Low-level features SIFT Representation BoW/VLAD/Fisher Classifier SVM EngineeredUnsupervisedSupervised Supervised 9
- 10. Example: feature engineering in computer vision 10 ϕ(x)
- 11. Neural networks: single neuron We already seen the single neuron. This is just a linear classifier (or regressor) Inputs: ● x1 , x2 Parameters ● w1 , w2 , b
- 12. Neural networks A composition of these simple neurons into several layers Each neuron simply computes a linear combination of its inputs, adds a bias, and passes the result through an activation function g(x) The network can contain one or more hidden layers. The outputs of these hidden layers can be thought of as a new representation of the data (new features). The final output is the target variable (y = f(x))
- 13. Activation functions ● ● ● ● ● ● Sigmoid Tanh ReLU 13 If we only use linear layers we are only able to learn linear transformations of our input.
- 14. Multilayer perceptrons 14 x1 x2 x3 x4 y1 Layer 1 Layer 2 Layer 3 Layer 0 y2 When each node in each layer is a linear combination of all inputs from the previous layer then the network is called a multilayer perceptron (MLP) Weights can be organized into matrices. Fully connected network Forward pass computes h0 h1 h2 h3
- 15. Multilayer perceptrons 15 Forward pass computes w11 w12 w13 w14 w21 w22 w23 w24 w31 w32 w33 w34 w41 w42 w43 w44 W1 x1 x2 x3 x4 b1 b2 b3 b4 b1 h0 x1 x2 x3 x4 y1 Layer 1 Layer 2 Layer 3 Layer 0 y2 h0 h1 h2 h3
- 16. Multilayer perceptrons 16 Forward pass computes w11 w12 w13 w14 w21 w22 w23 w24 w31 w32 w33 w34 w41 w42 w43 w44 W1 x1 x2 x3 x4 b1 b2 b3 b4 b1 h0 x1 x2 x3 x4 y1 Layer 1 Layer 2 Layer 3 Layer 0 y2 h0 h1 h2 h3
- 17. Universal approximation theorem Universal approximation theorem states that “the standard multilayer feed-forward network with a single hidden layer, which contains finite number of hidden neurons, is a universal approximator among continuous functions on compact subsets of Rn , under mild assumptions on the activation function.” If a 2 layer NN is a universal approximator, then why do we need deep nets?? The universal approximation theorem: ● Says nothing about the how easy/difficult it is to fit such approximators ● Needs a “finite number of hidden neurons”: finite may be extremely large In practice, deep nets can usually represent more complex functions with less total neurons (and therefore, less parameters)
- 18. Example: MNIST digit classification ● ● ● ● ● ● ● ● 18
- 19. Example: MNIST digit classification ● ● ● ● ● 28 28 19 Layer #Weights #Biases Total 1 784 x 512 512 401,920 2 512 x 512 512 262,656 3 512 x 10 10 5,130 669,706
- 20. Example: MNIST digit classification ● ● ● ● ● ● ● ● ● ● ● ● ● ● 20
- 22. Convolutional neural networks (CNNs, convnets) ● ● 22
- 23. 1D convolution 23 0 1 5 0 2 8 1 0 -1 2 -1 Signal Kernel/filter dot product -3 = .
- 24. 1D convolution 24 0 1 5 0 2 8 1 0 -1 2 -1 Signal Kernel/filter -3 9 . .
- 25. 1D convolution 25 0 1 5 0 2 8 1 0 -1 2 -1 Signal Kernel/filter -3 9 -7 . . .
- 26. 1D convolution 26 0 1 5 0 2 8 1 0 -1 2 -1 Signal Kernel/filter -3 9 -7 -4 13 -6 . . . . . .
- 27. 1D convolution 27 0 1 5 0 2 8 1 0 -1 2 -1 Signal Kernel/filter -3 -7 13 . . . Hyperparameters
- 28. Convolution on a grid 28
- 29. Convolution on a volume 32 32 5 5 3 A 5x5 convolution on a volume of depth 3 (e.g. an image) needs a filter (kernel) with 5x5x3 elements (weights) + a bias Andrej Karpathy's demo: http://cs231n.github.io/convolutional-networks/# conv
- 30. Convolution with multiple filters Stack results into a 3D volume (depth 4) Apply filter 1 Apply filter 2 Apply filter 3 Apply filter 4
- 31. Pooling layers 1 5 0 2 8 1 10 2 4 9 0 3 8 9 3 7 8 2 3 8 9 6 0 5 16 7 2 2 7 3 6 3 0 5 2 2 10 9 16 7
- 32. Convnets ● ● LeNet-5 [LeCun et al. 1998] 32
- 33. Alexnet ● 8 parameter layers (5 convolution, 3 fully connected) ● Softmax output ● 650,000 units ● 60 million free parameters ● Trained on two GPUs (two streams) for a week ● Ensemble of 7 nets used in ILSVRC challenge Krizhevsky et al. ImageNet classification with deep convolutional neural networks. NIPS, 2012.
- 34. Features of Alexnet: Convolutions
- 35. Features of Alexnet: max pooling
- 36. Features of Alexnet: ReLu Sigmoid ReLU
- 37. Filters learnt by Alexnet Visualization of the 96 11 x 11 filters learned by bottom layer Krizhevsky et al. ImageNet classification with deep convolutional neural networks. NIPS, 2012.
- 38. Example: a convnet on MNIST ● ● ● ● ● ● ● ● ● ● https://github.com/fchollet/keras/blob/master/examples/mnist_cnn.py 38
- 41. Questions? 41
- 42. ● ● ● ● ● Layer #Weights #Biases Total 1 784 x 512 512 401,920 2 512 x 512 512 262,656 3 512 x 10 10 5,130 669,706 ● ● ● ● ● ● ● Layer #Weights #Biases Total 1 3 x 3x 32 1 10 2 3x3x32 32 320 3 14X14x128 128 25,216 4 128X10 10 1,290 35,526