Deep Learning with CNTK
5 likes651 views
This document provides an introduction to deep learning with Microsoft's Cognitive Toolkit (CNTK). It discusses key deep learning concepts and how they are implemented in CNTK, including neural networks, backpropagation, loss functions, and common network architectures like convolutional neural networks. It also outlines several of Microsoft's products that use deep learning like Cortana, Bing, and Skype Translator. Examples of training deep learning models with CNTK on datasets like MNIST using logistic regression, multi-layer perceptrons, and CNNs are also presented.
![Microsoft’s historic
speech breakthrough
• Microsoft 2016 research system for
conversational speech recognition
• 5.9% word-error rate
• enabled by CNTK’s multi-server scalability
[W. Xiong, J. Droppo, X. Huang, F. Seide, M. Seltzer, A. Stolcke,
D. Yu, G. Zweig: “Achieving Human Parity in Conversational
Speech Recognition,” https://arxiv.org/abs/1610.05256]](https://image.slidesharecdn.com/datafestdeeplearning-171012202850/85/Deep-Learning-with-CNTK-7-320.jpg)
Deep Learning with CNTK
- 1. Introduction to Deep Learning with Microsoft Cognitive Toolkit
- 3. Deep learning at Microsoft • Microsoft Cognitive Services • Skype Translator • Cortana • Bing • Bing Ads • Augmented Reality • Microsoft Research
- 4. deep learning at Microsoft • Microsoft Cognitive Services • Skype Translator • Cortana • Bing • HoloLens • Microsoft Research
- 7. Microsoft’s historic speech breakthrough • Microsoft 2016 research system for conversational speech recognition • 5.9% word-error rate • enabled by CNTK’s multi-server scalability [W. Xiong, J. Droppo, X. Huang, F. Seide, M. Seltzer, A. Stolcke, D. Yu, G. Zweig: “Achieving Human Parity in Conversational Speech Recognition,” https://arxiv.org/abs/1610.05256]
- 8. Machine Learning in a Nutshell Machine learning algorithm Model Application Data Contains patterns Finds patterns Recognizes patterns Provides new data to see if it matches known patterns
- 9. Deploy chosen model Chosen Model Apply learning algorithm to data Candidate Model The Machine Learning Process Prepared Data Apply pre- processing to data Iterate to find the best model Data Preprocessing Modules Iterate until data is ready Preprocessing Modules Machine Learning Algorithms Applications The goal: Smarter applications Raw Data Raw Data Choose data
- 10. Terminology Training data The prepared data used to create a model Creating a model is called training a model Supervised learning The value you want to predict is in the training data The data is labeled Unsupervised learning The value you want to predict is not in the training data The data is unlabeled The most common approach
- 11. Styles of Machine Learning Algorithms Examples Decision tree Neural network Bayesian K-means P(A) P(B|A) P(B) P(A|B) = Deep learning uses this
- 12. What is deep learning --- Yoshua Bengio Machine learning is a way to try to make machines intelligent by allowing computers to learn from examples about the world around us or about some specific aspect of it. Deep learning is an approach to machine learning, particular among all the machine learning methods in that it is inspired by some of the things we know about the brain. It’s trying to make computers learn multiple levels of abstraction and representation, which is presumably what makes these systems so successful Reinforcement learning is a type of machine learning where the learner doesn’t get to know what a human would do in this context. The learner only gets to see if the actions were good or bad after a long set of actions. A lot of the recent progress in this area is in things like playing games, but reinforcement learning probably is going to be very important for things like self-driving cars.
- 13. Deep learning fundamentals • Model Generalization • Network Architecture • Activation function • Regularization • Model Training • Loss functions • Parameter gradient computation with backpropagation • Gradient descent algorithms DL is trying to make computers learn multiple levels of abstraction and representation
- 16. Logistic regression Features Weights 1.21 x 0.12 -2.2 x -3.4 -0.32 x 1.11 -1.29 x -0.94 1.4 x -1.2 Multiply features by weights dot product Sum them up, add a bias term Pass result through the logistic function 0.94
- 17. A single-layer neural network (also known as doing multiple logistic regressions at the same time) Features 0.12 0.92 0.33 0.02 0.99 0.42 0.12 0.92 0.33 0.02 0.99 0.42 Another vector! What might we do with this? Weight matrix Input vector Bias vector(pretend the weights are different)
- 18. Multi-Layer Perceptrons Network depth Result of one layer becomes features to next
- 19. These diagrams are getting hard to read Let’s make each individual input and logistic regression into a circle
- 20. How do we figure out the weights? Start with random weights Try out the model Determine the error Figure out how responsible each weight is for the error Punish each weight in proportion to its crime Truth: Forward propagate to make prediction Backward propagate errors
- 21. CNTK expresses (nearly) arbitrary neural networks by composing simple building blocks into complex computational networks, supporting relevant network types and applications Microsoft Cognitive Toolkit
- 22. • Microsoft’s open-source deep-learning toolkit • https://github.com/Microsoft/CNTK • Created by Microsoft Speech researchers (Dong Yu et al.) in 2012, “Computational Network Toolkit” • On GitHub since Jan 2016 under MIT license • Renamed from CNTK to “Cognitive Toolkit” • Community contributions e.g. from MIT, Stanford and NVidia Microsoft Cognitive Toolkit
- 23. CNTK - Other Benefits • Python and C++ API • Mostly implemented in C++ • Low level + high level Python API • Extensibility • User functions and learners in pure Python • Readers • Distributed, highly efficient built-in data readers • Details: https://docs.microsoft.com/en-us/cognitive-toolkit/reasons-to-switch-from-tensorflow- to-cntk Microsoft Cognitive Toolkit
- 24. Anatomy of a CNTK training job Script configure and executes through CNTK Python APIs… trainer • SGD (momentum, Adam, …) • minibatching reader • minibatch source • task-specific deserializer • automatic randomization • distributed reading corpus model network • model function • criterion function • CPU/GPU execution engine • packing, padding
- 25. Terms to remember: Neural Networks/Deep Networks • Backpropagation • Forward Pass • Loss Function • Backward pass • Weight Adjustment • Hidden layer – Neither an input nor and output layer • Activation Function & Activation Value • Activation Matrix (CNN) • Stochastic Gradient Descent • Convolutional Layers • Auto Feature Detection • Weights • Bias • Regularization
- 26. CNTK – Jupyter Notebook MNIST Data – Recognize Digits 1. Logistic Regression 2. Multi Layer Perceptron 3. Convolution Neural Networks
- 27. Machine Learning – Logistic Regression
- 28. Deep Learning – Multi Level Perceptron
- 29. Deep Learning – CNN
- 30. Thank You @ashishjaiman cntkhelp@microsoft.com https://github.com/Microsoft/CNTK Introduction To CNTK notebooks.azure.com/library/cntkbeta2