Introduction to Deep Learning with Python
154 likes73,648 views
This document discusses the application of Theano in machine learning, focusing on tasks such as handwritten digit recognition using the MNIST dataset. It explains different models including linear regression, logistic regression, and convolutional networks, along with practical coding and training methods. Key takeaways highlight the importance of noise for regularization, the advantages of rectifiers, and the necessity of respecting data structure in model design.
![from arXiv:1409.4842v1 [cs.CV] 17 Sep 2014
A ⌠model? - GoogLeNet](https://image.slidesharecdn.com/tuftspreso-141029155034-conversion-gate01/85/Introduction-to-Deep-Learning-with-Python-12-320.jpg)
Introduction to Deep Learning with Python
- 1. From multiplication to convolutional networks How do ML with Theano
- 2. Todayâs Talk â A motivating problem â Understanding a model based framework â Theano â Linear Regression â Logistic Regression â Net â Modern Net â Convolutional Net
- 3. Follow along Tutorial code at: https://github.com/Newmu/Theano-Tutorials Data at: http://yann.lecun.com/exdb/mnist/ Slides at: http://goo.gl/vuBQfe
- 4. A motivating problem How do we program a computer to recognize a picture of a handwritten digit as a 0-9? What could we do?
- 5. A dataset - MNIST What if we have 60,000 of these images and their label? X = images Y = labels X = (60000 x 784) #matrix (list of lists) Y = (60000) #vector (list) Given X as input, predict Y
- 6. An idea For each image, find the âmost similarâ image and guess that as the label.
- 7. An idea For each image, find the âmost similarâ image and guess that as the label. KNearestNeighbors ~95% accuracy
- 8. Trying things Make some functions computing relevant information for solving the problem
- 9. What we can code Make some functions computing relevant information for solving the problem feature engineering
- 10. What we can code Hard coded rules are brittle and often arenât obvious or apparent for many problems.
- 11. Model A Machine Learning Framework 8 Inputs Computation Outputs
- 12. from arXiv:1409.4842v1 [cs.CV] 17 Sep 2014 A ⌠model? - GoogLeNet
- 13. 3 mult Input Computation Output A very simple model by x 12
- 14. Theano intro
- 16. Theano intro imports theano symbolic variable initialization
- 17. Theano intro imports theano symbolic variable initialization our model
- 18. Theano intro imports theano symbolic variable initialization our model compiling to a python function
- 19. Theano intro imports theano symbolic variable initialization our model compiling to a python function usage
- 20. Theano
- 21. Theano imports
- 22. Theano imports training data generation
- 23. Theano imports training data generation symbolic variable initialization
- 24. Theano imports training data generation symbolic variable initialization our model
- 25. Theano imports training data generation symbolic variable initialization our model model parameter initialization
- 26. Theano imports training data generation symbolic variable initialization our model model parameter initialization metric to be optimized by model
- 27. Theano imports training data generation symbolic variable initialization our model model parameter initialization metric to be optimized by model learning signal for parameter(s)
- 28. Theano imports training data generation symbolic variable initialization our model model parameter initialization metric to be optimized by model learning signal for parameter(s) how to change parameter based on learning signal
- 29. Theano imports training data generation symbolic variable initialization our model model parameter initialization metric to be optimized by model learning signal for parameter(s) how to change parameter based on learning signal compiling to a python function
- 30. Theano imports training data generation symbolic variable initialization our model model parameter initialization metric to be optimized by model learning signal for parameter(s) how to change parameter based on learning signal compiling to a python function iterate through data 100 times and train model on each example of input, output pairs
- 31. Theano doing its thing
- 32. Zero One Two Three Four Five Six Seven Eight Nine 0.1 0. 0. 0.1 0. 0. 0. 0. 0.7 0.1 Logistic Regression softmax(X) T.dot(X, w)
- 33. Back to Theano
- 34. Back to Theano convert to correct dtype
- 35. Back to Theano convert to correct dtype initialize model parameters
- 36. Back to Theano convert to correct dtype initialize model parameters our model in matrix format
- 37. Back to Theano convert to correct dtype initialize model parameters our model in matrix format loading data matrices
- 38. Back to Theano convert to correct dtype initialize model parameters our model in matrix format loading data matrices now matrix types
- 39. Back to Theano convert to correct dtype initialize model parameters our model in matrix format loading data matrices now matrix types probability outputs and maxima predictions
- 40. Back to Theano convert to correct dtype initialize model parameters our model in matrix format loading data matrices now matrix types probability outputs and maxima predictions classification metric to optimize
- 41. Back to Theano convert to correct dtype initialize model parameters our model in matrix format loading data matrices now matrix types probability outputs and maxima predictions classification metric to optimize compile prediction function
- 42. Back to Theano convert to correct dtype initialize model parameters our model in matrix format loading data matrices now matrix types probability outputs and maxima predictions classification metric to optimize compile prediction function train on mini-batches of 128 examples
- 43. 0 1 2 3 4 5 6 7 8 9 What it learns
- 44. 0 1 2 3 4 5 6 7 8 9 What it learns Test Accuracy: 92.5%
- 45. Zero One Two Three Four Five Six Seven Eight Nine 0.0 0. 0. 0.1 0. 0. 0. 0. 0.9 0. y = softmax(T.dot(h, wo)) h = T.nnet.sigmoid(T.dot(X, wh)) An âoldâ net (circa 2000)
- 46. A âoldâ net in Theano
- 47. A âoldâ net in Theano generalize to compute gradient descent on all model parameters
- 48. 2D moons dataset courtesy of scikit-learn Understanding SGD
- 49. A âoldâ net in Theano generalize to compute gradient descent on all model parameters 2 layers of computation input -> hidden (sigmoid) hidden -> output (softmax)
- 51. A âoldâ net in Theano generalize to compute gradient descent on all model parameters 2 layers of computation input -> hidden (sigmoid) hidden -> output (softmax) initialize both weight matrices
- 52. A âoldâ net in Theano generalize to compute gradient descent on all model parameters 2 layers of computation input -> hidden (sigmoid) hidden -> output (softmax) initialize both weight matrices updated version of updates
- 53. What an âoldâ net learns Test Accuracy: 98.4%
- 54. Zero One Two Three Four Five Six Seven Eight Nine 0.0 0. 0. 0.1 0. 0. 0. 0. 0.9 0. y = softmax(T.dot(h2, wo)) h2 = rectify(T.dot(h, wh)) h = rectify(T.dot(X, wh)) Noise Noise Noise (or augmentation) A âmodernâ net - 2012+
- 55. A âmodernâ net in Theano
- 56. rectifier A âmodernâ net in Theano
- 58. rectifier numerically stable softmax A âmodernâ net in Theano
- 59. rectifier numerically stable softmax a running average of the magnitude of the gradient A âmodernâ net in Theano
- 60. rectifier numerically stable softmax a running average of the magnitude of the gradient scale the gradient based on running average A âmodernâ net in Theano
- 61. 2D moons dataset courtesy of scikit-learn Understanding RMSprop
- 62. rectifier numerically stable softmax a running average of the magnitude of the gradient scale the gradient based on running average A âmodernâ net in Theano randomly drop values and scale rest
- 63. rectifier numerically stable softmax a running average of the magnitude of the gradient scale the gradient based on running average A âmodernâ net in Theano randomly drop values and scale rest Noise injected into model rectifiers now used 2 hidden layers
- 64. What a âmodernâ net learns Test Accuracy: 99.0%
- 66. What a âmodernâ net is doing
- 67. from deeplearning.net Convolutional Networks
- 68. A convolutional network in Theano
- 69. a âblockâ of computation conv -> activate -> pool -> noise A convolutional network in Theano
- 70. a âblockâ of computation conv -> activate -> pool -> noise convert from 4tensor to normal matrix A convolutional network in Theano
- 71. a âblockâ of computation conv -> activate -> pool -> noise convert from 4tensor to normal matrix reshape into conv 4tensor (b, c, 0, 1) format A convolutional network in Theano
- 72. a âblockâ of computation conv -> activate -> pool -> noise convert from 4tensor to normal matrix reshape into conv 4tensor (b, c, 0, 1) format now 4tensor for conv instead of matrix A convolutional network in Theano
- 73. a âblockâ of computation conv -> activate -> pool -> noise convert from 4tensor to normal matrix reshape into conv 4tensor (b, c, 0, 1) format now 4tensor for conv instead of matrix conv weights (n_kernels, n_channels, kernel_w, kerbel_h) A convolutional network in Theano
- 74. a âblockâ of computation conv -> activate -> pool -> noise convert from 4tensor to normal matrix reshape into conv 4tensor (b, c, 0, 1) format now 4tensor for conv instead of matrix conv weights (n_kernels, n_channels, kernel_w, kerbel_h) highest conv layer has 128 filters and a 3x3 grid of responses A convolutional network in Theano
- 75. a âblockâ of computation conv -> activate -> pool -> noise convert from 4tensor to normal matrix reshape into conv 4tensor (b, c, 0, 1) format now 4tensor for conv instead of matrix conv weights (n_kernels, n_channels, kernel_w, kerbel_h) highest conv layer has 128 filters and a 3x3 grid of responses A convolutional network in Theano noise during training
- 76. a âblockâ of computation conv -> activate -> pool -> noise convert from 4tensor to normal matrix reshape into conv 4tensor (b, c, 0, 1) format now 4tensor for conv instead of matrix conv weights (n_kernels, n_channels, kernel_w, kerbel_h) highest conv layer has 128 filters and a 3x3 grid of responses A convolutional network in Theano noise during training no noise for prediction
- 77. Test Accuracy: 99.5% What a convolutional network learns
- 78. Takeaways â A few tricks are needed to get good results â Noise important for regularization â Rectifiers for faster, better, learning â Donât use SGD - lots of cheap simple improvements â Models need room to compute. â If your data has structure, your model should respect it.
- 79. Resources â More in-depth theano tutorials â http://www.deeplearning.net/tutorial/ â Theano docs â http://www.deeplearning.net/software/theano/library/ â Community â http://www.reddit.com/r/machinelearning
- 80. A plug Keep up to date with indico: https://indico1.typeform.com/to/DgN5SP
- 81. Questions?