Intro to Deep Learning
4 likes1,189 views
The document provides an overview of neural networks and Theano, emphasizing their significance in improving state-of-the-art performance in tasks like speech recognition, natural language processing, and object recognition. It discusses key concepts like learning representations, the role of logistic regression and multilayer perceptrons, and the use of neural networks in various applications, including autoencoders and deep learning techniques. Theano is introduced as a Python library that aids in efficient mathematical expression evaluation and computation for neural networks.
![Refrences
● [Bengio07] Bengio, P. Lamblin, D. Popovici and H. Larochelle, Greedy
LayerWise Training of Deep Networks, in Advances in Neural Information
Processing Systems 19 (NIPS‘06), pages 153160, MIT Press 2007.
● [Bengio09] Bengio, Learning deep architectures for AI, Foundations and
Trends in Machine Learning 1(2) pages 1127.
● [Xavier10] Bengio, X. Glorot, Understanding the difficulty of training
deep feedforward neuralnetworks, AISTATS 2010
● Bengio, Yoshua, Aaron Courville, and Pascal Vincent. "Representation
learning: A review and new perspectives." Pattern Analysis and Machine
Intelligence, IEEE Transactions on 35.8 (2013): 17981828.
● Deep learning for NLP (http://nlp.stanford.edu/courses/NAACL2013/)](https://image.slidesharecdn.com/introtodeeplearning-150418202842-conversion-gate02/85/Intro-to-Deep-Learning-25-320.jpg)
Intro to Deep Learning
- 1. Introduction to Neural Networks and Theano Kushal Arora Karora@cise.ufl.edu
- 2. Outline 1. Why Care? 2. Why it works? 3. Logistic Regression to Neural Networks 4. Deep Networks and Issues 5. Autoencoders and Stacked Autoencoders 6. Why Deep Learning Works 7. Theano Overview 8. Code Hands On
- 3. Why Care? ● Has bettered state of art of various tasks – Speech Recognition ● Microsoft Audio Video Indexing Service speech system uses Deep Learning ● Brought down word error rate by 30% compared to SOTA GMM – Natural Language Understanding/Processing ● Neural net language models are current state of art in language modeling, Sentiment Analysis, Paraphrase Detection and many other NLP task ● SENNA system uses neural embedding for various NLP tasks like POS tagging, NER, chunking etc. ● SENNA is not only better than SOTA but also much faster – Object Recognition ● Breakthrough started in 2006 with MNIST dataset. Still the best (0..27% error ) ● SOTA error rate over ImageNet dataset down to 15.3% compared to 26.1%
- 4. Why Care? ● MultiTask and Transfer Learning – Ability to exploit commonalities of different learning task by transferring learned knowledge – Example Google Image Search (Woman in a red dress in a garden) – Other example is Google's Image Caption (http://googleresearch.blogspot.com/2014/11/a pictureisworththousandcoherent.html)
- 5. Why it works? ● Learning Representations – Time to move over hand crafted features ● Distributed Representation – More robust and expressive features ● Learning Multiple Level of Representation – Learning multiple level of abstraction
- 6. #1 Learning Representation • Handcrafting features is inefficient and time consuming • Must be done again for each task and domain • The features are often incomplete and highly correlated
- 7. #2 Distributed Representation ● Features can be nonmutually exclusive example language. ● Need to move beyond onehot representations such as from clustering algorithms, knearest neighbors etc ● O(n) parameters/examples for O(N) input regions but we can do better O(k) parameters/inputs for O(2k ) input regions ● Similar to multiclustering, where multiple clustering algorithms are applied in parallel or same clustering algorithm is applied to different input region
- 8. #3 Learning multiple level of representation ● Deep architectures promote reuse of features ● Deep architecture can potentially lead to progressively more abstract features at higher layer ● Good intermediate representation can be shared across tasks ● Insufficient depth can be exponentially inefficient
- 9. The Basics ● Building block of the network is a neuron. ● The basic terminologies – Input – Activation layer – Output ● Activation function is usually of the form where input activation function output h(w,b)=f (w T .x+b) f ( z)= 1 1+e −z
- 10. Logistic Regression ● Logistic regression is a probabilistic, linear classifier ● Parameterized by W and b ● Each output is probability of input belonging to class yi ● Prob of x being member of class Yi is calculated as where and P(Y=Yi∣x)=softmaxi(Wx+b) softmaxi (Wx +b)= e W i Tx + bi ∑ j e W j Tx + bi ypred=argmaxYi P(Y=Yi∣x)
- 11. Logistic Regression – Training ● The loss function for logistic regression is negative log likelihood, defined as ● The loss function minimization is done using stochastic gradient descent or mini batch stochastic gradient descent. ● The function is convex and will always reach global minima which is not true for other architectures we will discuss. ● Cannot solve famous XOR problem.
- 12. Multilayer preceptron ● An MLP can be viewed as a logistic regression classifier where the input is first transformed using a learned non-linear transformation. ● The non linear transformation can be a sigmoid or a tanh function. ● Due to addition of non linear hidden layer, the loss function now is non convex ● Though there is no sure way to avoid minima, some empirical value initializations of the weight matrix helps.
- 13. Multilayer preceptron – Training ● Let D be the size of input vector and there be L output labels. The output vector (of size L) will be given as where and are activation functions. ● The hard part of training is calculating the gradient for stochastic gradient descent and of course avoid the local minima. ● Backpropagation used as an optimization technique to avoid calculation gradient at each step. (It is like Dynamic programming for derivative chain rule recursion) G s
- 14. Deep Neural Nets and issues ● Generally networks with more than 23 hidden layers are called deep nets. ● Pre 2006 they performed worse than shallow networks. ● Though hard to analyze the exact cause the experimental results suggested that gradientbased training for deep nets got stuck in local minima due to gradient diffusion ● It was difficult to get good generalization. ● Random initialization which worked for shallow network couldn't be used deep networks. ● The issue was solved using a unsupervised pretraining of hidden layers followed by a fine tuning or supervised training.
- 15. AutoEncoders ● Multilayer neural nets with target output = input. ● Reconstruction = decoder(encoder(input)) ● Objective is to minimize the reconstruction error. ● PCA can be seen as a auto-encoder with and ● So autoencoder could be seen as an non-linear PCA which tries of learn latent representation of input. a=tanh(Wx+b) x '=tanh(W Tx +c) L=‖x '−x‖2 a=Wx x'=W T a
- 18. PreTraining
- 19. Why PreTraining Works ● Hard to know exactly as deep nets are hard to analyze ● Regularization Hypothesis – PreTraining acts as adding regularization term leading to better generalization. – It can be seen as good representation for P(X) leads to good representation of P(Y|X) ● Optimization Hypothesis – PreTraining leads to weight initialization that restricts the parameter space near better local minima – These minimas are not achievable via random initialization
- 20. Other Type of Neural Networks ● Recurrent Neural Nets ● Recursive Neural Networks ● Long Short Term Memory Architecture ● Convolutional Neural Networks
- 21. Libraries to use ● Theano/Pylearn2 (U Motreal, Python) ● Caffe (UCB, C++, Fast, CUDA based) ● Torch7 (NYU, Lua, supported by Facebook)
- 22. Introduction to Theano What is Theano? Theano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multidimensional arrays efficiently. Why should it be used? – Tighter integration with numpy – Transparent use of GPU – Efficient symbolic representation – Does lots of heavy lifting like gradient calculation for you – Uses simple abstractions for tensors and matrices so you don't have to deal with it
- 23. Theano Basics (Tutorial) ● How To: – Declare Expression – Compute Gradient ● How expression is evaluated. (link) ● Debugging
- 24. Implementations ● Logistic Regression ● Multilayer preceptron ● AutoEncoders ● Stacked AutoEncoders
- 25. Refrences ● [Bengio07] Bengio, P. Lamblin, D. Popovici and H. Larochelle, Greedy LayerWise Training of Deep Networks, in Advances in Neural Information Processing Systems 19 (NIPS‘06), pages 153160, MIT Press 2007. ● [Bengio09] Bengio, Learning deep architectures for AI, Foundations and Trends in Machine Learning 1(2) pages 1127. ● [Xavier10] Bengio, X. Glorot, Understanding the difficulty of training deep feedforward neuralnetworks, AISTATS 2010 ● Bengio, Yoshua, Aaron Courville, and Pascal Vincent. "Representation learning: A review and new perspectives." Pattern Analysis and Machine Intelligence, IEEE Transactions on 35.8 (2013): 17981828. ● Deep learning for NLP (http://nlp.stanford.edu/courses/NAACL2013/)
- 26. Thank You!