Convolutional Neural Networks at scale in Spark MLlib
5 likes3,446 views
The document discusses advancements in convolutional neural networks (CNNs) and recurrent neural networks (RNNs) at the Spark Technology Center, highlighting their application in various fields such as image recognition and natural language processing. It addresses the framework for deep learning, optimization techniques, and the benefits of a flexible deep learning API. Future work will include enhancements like more modern optimizers, dropout regularization, and support for advanced features such as batch normalization and tensor operations.
Convolutional Neural Networks at scale in Spark MLlib
- 1. Spark Technology Center Convolutional Neural Networks at Scale in MLlib Jeremy Nixon
- 2. Spark Technology Center 1. Machine Learning Engineer at the Spark Technology Center 2. Contributor to MLlib, dedicated to scalable deep learning. 3. Previously, studied Applied Mathematics to Computer Science and Economics at Harvard Jeremy Nixon
- 3. Future Work 1. Convolutional Neural Networks a. Convolutional Layer Type b. Max Pooling Layer Type 2. Flexible Deep Learning API 3. More Modern Optimizers a. Adam b. Adadelta + Nesterov Momentum 4. More Modern activations 5. Dropout / L2 Regularization 6. Batch Normalization 7. Tensor Support 8. Recurrent Neural Networks (LSTM)
- 4. Spark Technology Center 1. Framing Deep Learning 2. MLlib Deep Learning API 3. Optimization 4. Performance 5. Future Work Structure
- 5. Spark Technology Center 1. Structural Assumptions 2. Automated Feature Engineering 3. Learning Representations 4. Applications Framing Convolutional Neural Networks
- 6. Spark Technology Center - Network depth creates an extraordinary range of possible models. - That flexibility creates value in large datasets to reduce variance. Structural Assumptions: Combinatorial Flexibility
- 7. Spark Technology Center X = Normalized Data, W1 , W2 = Weights Forward: 1. Multiply data by first layer weights | (X*W1 ) 2. Put output through non-linear activation | max(0, X*W1 ) 3. Multiply output by second layer weights | max(0, X*W1 ) * W2 4. Return predicted output Structural Assumption: The Model
- 8. Spark Technology Center - Pixels - Edges - Shapes - Parts - Objects - Learn features that are optimized for the data - Makes transfer learning feasible Structural Assumptions: Hierarchical Abstraction
- 9. Spark Technology Center Structural Assumptions: Location Invariance - Convolution is a restriction on the features that can be combined. - Location Invariance leads to strong accuracy in vision, audio, and language. colah.github.io
- 12. Spark Technology Center 1. CNNs - State of the art a. Object Recognition b. Object Localization c. Image Segmentation d. Image Restoration e. Music Recommendation 2. RNNs (LSTM) - State of the Art a. Speech Recognition b. Question Answering c. Machine Translation d. Text Summarization e. Named Entity Recognition f. Natural Language Generation g. Word Sense Disambiguation h. Image / Video Captioning i. Sentiment Analysis Applications
- 13. Spark Technology Center Flexibility. High level enough to be efficient. Low level enough to be expressive. MLlib Flexible Deep Learning API
- 14. Spark Technology Center Flexibility. High level enough to be efficient. Low level enough to be expressive. MLlib Flexible Deep Learning API
- 15. Spark Technology Center Modularity enables Logistic Regression, Feedforward Networks. MLlib Flexible Deep Learning API
- 16. Spark Technology Center Introducing Convolutional and Max-Pooling Layer types. MLlib Convolutional Neural Network
- 19. Spark Technology Center Parallel implementation of backpropagation: 1. Each worker gets weights from master node. 2. Each worker computes a gradient on its data. 3. Each worker sends gradient to master. 4. Master averages the gradients and updates the weights. Distributed Optimization
- 20. Spark Technology Center ● Parallel MLP on Spark with 7 nodes ~= Caffe w/GPU (single node). ● Advantages to parallelism diminish with additional nodes due to communication costs. ● Additional workers are valuable up to ~20 workers. ● See https://github.com/avulanov/ann-benc hmark for more details Performance
- 21. Spark Technology Center Github: https://github.com/JeremyNixon/sparkdl Spark Package: https://spark-packages.org/package/JeremyNixon/s parkdl Access
- 22. Spark Technology Center 1. GPU Acceleration (External) 2. Keras Integration 3. Residual Layers 4. Hardening 5. Regularization 6. Batch Normalization 7. Tensor Support Future Work
- 23. Spark Technology Center Thank you for your attention! Questions?