Deep Learning and Recurrent Neural Networks in the Enterprise
Download as PPTX, PDF5 likes1,051 views
This document discusses deep learning and recurrent neural networks. It provides an overview of deep learning, including definitions, automated feature learning, and popular deep learning architectures. It also describes DL4J, a tool for building deep learning models in Java and Scala, and discusses applications of recurrent neural networks for tasks like anomaly detection using time series data and audio processing.
![Certified on Two Hadoop Distributions
• Running Spark on Hadoop via YARN gives us
– Sharing cluster resources between heterogeneous
workloads concurrently
– Access to the yarn scheduler capabilities
– Better control of executors in Spark
– Kerberos support for security
• Certified on CDH 5.4
• Certified on HDP 2.4
– [ Coming later this month ]](https://image.slidesharecdn.com/jpstampedecon2016v1-160801195614/85/Deep-Learning-and-Recurrent-Neural-Networks-in-the-Enterprise-30-320.jpg)
Deep Learning and Recurrent Neural Networks in the Enterprise
- 1. Deep Learning and Recurrent Neural Networks in the Enterprise StampedeCon St. Louis 2016 Josh Patterson, Skymind
- 2. Presenter: Josh Patterson Past Research in Swarm Algorithms: Real-time optimization techniques in mesh sensor networks TVA / NERC: Smartgrid, Sensor Collection, and Big Data Cloudera: Principal SA, Working with Fortune 500 Patterson Consulting: Working with Fortune 500 on Big Data, ML Today Skymind, Director Field Engineering josh@skymind.io / @jpatanooga DL4J Co-creator, Co-Author on Upcoming Oreilly Book “Deep Learning: A Practitioner’s Approach”
- 3. Topics • What is Deep Learning? • DL4J • Recurrent Neural Network Applications
- 4. WHAT IS DEEP LEARNING?
- 5. Defining Deep Learning • Higher neuron counts than in previous generation neural networks • Different and evolved ways to connect layers inside neural networks • More computing power to train • Automated Feature Learning
- 6. Automated Feature Learning • Deep Learning can be thought of as workflows for automated feature construction – From “feature construction” to “feature learning” • As Yann LeCun says: – “machines that learn to represent the world”
- 9. These are the features learned at each neuron in a Restricted Boltzmann Machine (RBMS) These features are passed to higher levels of RBMs to learn more complicated things. Part of the “7” digit
- 10. Unreasonable Effectiveness: Benchmark Records 1. Text-to-speech synthesis (Fan et al., Microsoft, Interspeech 2014) 2. Language identification (Gonzalez-Dominguez et al., Google, Interspeech 2014) 3. Large vocabulary speech recognition (Sak et al., Google, Interspeech 2014) 4. Prosody contour prediction (Fernandez et al., IBM, Interspeech 2014) 5. Medium vocabulary speech recognition (Geiger et al., Interspeech 2014) 6. English to French translation (Sutskever et al., Google, NIPS 2014) 7. Audio onset detection (Marchi et al., ICASSP 2014) 8. Social signal classification (Brueckner & Schulter, ICASSP 2014) 9. Arabic handwriting recognition (Bluche et al., DAS 2014) 10. TIMIT phoneme recognition (Graves et al., ICASSP 2013) 11. Optical character recognition (Breuel et al., ICDAR 2013) 12. Image caption generation (Vinyals et al., Google, 2014) 13. Video to textual description (Donahue et al., 2014) 14. Syntactic parsing for Natural Language Processing (Vinyals et al., Google, 2014) 15. Photo-real talking heads (Soong and Wang, Microsoft, 2014).
- 11. Four Major Architectures • Deep Belief Networks • Convolutional Neural Networks • Recurrent Neural Networks • Recursive Neural Networks
- 12. Quick Usage Guide • If I have Timeseries or Audio Input – I should use a Recurrent Neural Network – Examples: Fraud Detection, Anomaly Detection • If I have Image input – I should use a Convolutional Neural Network • If I have Video input – I should use a hybrid Convolutional + Recurrent Architecture!
- 14. The More Things Change… • Deep Learning is still trying to answer the same fundamental questions such as: – “is this image a face?” • The difference is Deep Learning makes hard questions easier to answer with better architectures and more computing power – We do this by matching the correct architecture w the right problem
- 15. DL4J Building Deep Neural Networks with
- 16. DL4J • “The Hadoop of Deep Learning” – Java, Scala, and Python APIs – ASF 2.0 Licensed • Java implementation – Parallelization (Yarn + Spark) – GPU support • Also Supports multi-GPU per host • Runtime Neutral – Local – Hadoop / YARN + Spark • https://github.com/deeplearning4j/deeplearning4j
- 17. DL4J Workflow Toolchain ETL (DataVec) Vectorization (DataVec) Modeling (DL4J) Evaluation (Arbiter) Execution Platforms: Spark, Single Machine ND4J - Linear Algebra Runtime: CPU, GPU
- 18. ND4J: The Need for Speed • Javacpp (cython for java) – Auto generate JNI bindings for C++ by parsing classes – Allows for easy maintenance and deployment of c++ binaries in java • CPU backends – Openmp (multithreading within native operations) – Openblas or MKL (BLAS operations) – SIMD-extensions • GPU backends – DL4J supports Cuda 7.5 at the moment, and will support 8.0 support as soon as it comes out. – Leverages cudnn as well
- 19. Prepping Data is Time Consuming http://www.forbes.com/sites/gilpress/2016/03/23/data-preparation-most-time-consuming-least-enjoyable-data-science-task-survey-says/#633ea7f67f75
- 20. Preparing Data for Modeling is Hard
- 21. DataVec • DataVec is a tool for machine learning ETL (Extract, Transform, Load) operations. – Spark-Enabled and focused on Supporting DL4J • Also performs vectorization – Image, CSV, Sequences (timeseries), more • Open Source, ASF 2.0 Licensed – https://github.com/deeplearning4j/DataVec
- 22. RECURRENT NEURAL NETWORK APPLICATIONS Using DL4J for
- 23. Source: IDC White Paper - sponsored by EMC. As the Economy Contracts, the Digital Universe Expands. May 2009. . Transactional Data Explosion • 2,500 exabytes of new information in 2012 with Internet as primary driver • Digital universe grew by 62% last year to 800K petabytes and will grow to 1.2 “zettabytes” this year Relational Transactional (Logs, Sensors) (You)
- 24. NERC Sensor Data Collection openPDC PMU Data Collection circa 2009 • 120 Sensors • 30 samples/second • 4.3B Samples/day • Housed in Hadoop
- 25. Sensor Timeseries Classification with RNNs • Recurrent Neural Networks have the ability to model change of input over time • Older techniques (mostly) do not retain time domain – Hidden Markov Models do… • but are more limited • Key Takeaway: – For working with Timeseries data, RNNs will be more accurate
- 26. RNN Architectures Standard supervised learning Image captioning Sentiment analysis Video captioning, Natural language translation Part of speech tagging Generative mode for text
- 27. Anomaly Detection • Model the normal patterns in the data • Autoencoders give us the ability to look at data that it hasn’t seen before – Find anomalous patterns in sequences – Can also use RNNs for pattern classification • Interesting Industry Applications – Telecom – Financial Services
- 28. Audio Applications • Text-to-Speech • Recognize specific songs / audio • Enables natural language interfaces
- 29. “Google is living a few years in the future and sending the rest of us messages” -- Doug Cutting in 2013 • However – Most organizations are not built like Google • (and Jeff Dean does not work at your company…) • Anyone building Next-Gen infrastructure has to consider these things
- 30. Certified on Two Hadoop Distributions • Running Spark on Hadoop via YARN gives us – Sharing cluster resources between heterogeneous workloads concurrently – Access to the yarn scheduler capabilities – Better control of executors in Spark – Kerberos support for security • Certified on CDH 5.4 • Certified on HDP 2.4 – [ Coming later this month ]
- 31. Questions? Thank you for your time and attention “Deep Learning: A Practitioner’s Approach” (Oreilly, October 2016)
- 32. Running DL4J Workflows on Spark • DataVec is built to scale out via Spark RDDs – RDD<LabeledPoint> – RDD<DataSet> • DL4J Uses same MultiLayerConfiguration as single host version – Uses SparkDl4jMultiLayer to drive the training on spark – Performs Parameter Averaging spark-submit --class io.skymind.spark.dl4j.datavec.BasicDataVecExample --master yarn -- num-executors 1 --properties-file ./spark_extra.props ./Skymind_spark-1.0-SNAPSHOT.jar