Enterprise Deep Learning with DL4J
Download as PPTX, PDF2 likes1,091 views
Josh Patterson presented on deep learning and DL4J. He began with an overview of deep learning, explaining it as automated feature engineering where machines learn representations of the world. He then discussed DL4J, describing it as the "Hadoop of deep learning" - an open source deep learning library with Java, Scala, and Python APIs that supports parallelization on Hadoop, Spark, and GPUs. He demonstrated building deep learning workflows with DL4J and Canova, using the Iris dataset as an example to show how data can be vectorized with Canova and then a model trained on it using DL4J from the command line. He concluded by describing Skymind as a distribution of DL4J with enterprise
![WHAT IS DEEP LEARNING?
“Cooper: [When Cooper tries to reconfigure TARS] Humour 75%.
TARS: 75%. Self destruct sequence in T minus 10, 9, 8...
Cooper: Let's make it 65%.
TARS: Knock, knock.”
--- Interstellar](https://image.slidesharecdn.com/deeplearninggdsc20150502v4-150506075556-conversion-gate02/85/Enterprise-Deep-Learning-with-DL4J-4-320.jpg)
![DL4J from the Command Line
bin/dl4j
Usage:
dl4j [command] [params]
Commands:
train build a deep learning model
test test a deep learning model
predict score new records against a deep learning model](https://image.slidesharecdn.com/deeplearninggdsc20150502v4-150506075556-conversion-gate02/85/Enterprise-Deep-Learning-with-DL4J-24-320.jpg)
![JSON Model Config
{"sparsity":0.0,"useAdaGrad":true,"lr":0.10000000149011612,"corruptionLevel":0.30000001192092896,"numIterations":1000,"momentu
m":0.5,"l2":0.0,"useRegularization":false,"momentumAfter":null,"resetAdaGradIterations":-
1,"numLineSearchIterations":100,"dropOut":0.0,"applySparsity":false,"weightInit":"VI","optimizationAlgo":"CONJUGATE_GRADIENT",
"lossFunction":"RECONSTRUCTION_CROSSENTROPY","concatBiases":false,"constrainGradientToUnitNorm":false,"seed":123,"nIn":0,"nOut
":0,"activationFunction":"sigmoid","visibleUnit":"BINARY","hiddenUnit":"BINARY","k":1,"weightShape":[0,0],"filterSize":[2,2,2,
2],"numFeatureMaps":2,"featureMapSize":[2,2],"stride":[2,2],"kernel":5,"batchSize":0,"minimize":false,"rng":"org.apache.common
s.math3.random.MersenneTwister","layerFactory":"org.deeplearning4j.nn.layers.factory.PretrainLayerFactory,org.deeplearning4j.m
odels.featuredetectors.rbm.RBM","stepFunction":"org.deeplearning4j.optimize.stepfunctions.DefaultStepFunction","renderWeightIt
erations":-1,"dist":"org.apache.commons.math3.distribution.NormalDistributiont{SQRT2=1.4142135623730951, mean=0.0,
solverAbsoluteAccuracy=1.0E-9, DEFAULT_INVERSE_ABSOLUTE_ACCURACY=1.0E-9, standardDeviation=0.1,
logStandardDeviationPlusHalfLog2Pi=-1.3836465597893728,
serialVersionUID=8589540077390120676}","listeners":["org.deeplearning4j.optimize.listeners.ScoreIterationListenert{printItera
tions=10, log=Logger[org.deeplearning4j.optimize.listeners.ScoreIterationListener]},"]}](https://image.slidesharecdn.com/deeplearninggdsc20150502v4-150506075556-conversion-gate02/85/Enterprise-Deep-Learning-with-DL4J-26-320.jpg)
Enterprise Deep Learning with DL4J
- 1. 3 DAYS ON JANUARY 16th, 17th & 18th 2015 www.globalbigdataconference.com Twitter : @bigdataconf
- 2. Presenter: Josh Patterson Past Research in Swarm Algorithms TVA / NERC Cloudera Today Patterson Consulting Skymind (Advisor) josh@pattersonconsultingtn.com / @jpatanooga
- 3. • What is Deep Learning? • What is DL4J? • Enterprise Grade Deep Learning Workflows
- 4. WHAT IS DEEP LEARNING? “Cooper: [When Cooper tries to reconfigure TARS] Humour 75%. TARS: 75%. Self destruct sequence in T minus 10, 9, 8... Cooper: Let's make it 65%. TARS: Knock, knock.” --- Interstellar
- 5. We Want to be able to recognize Handwriting This is a Hard Problem
- 6. Automated Feature Engineering • Deep Learning can be thought of as workflows for automated feature construction – Where previously we’d consider each stage in the workflow as unique technique • Many of the techniques have been around for years – But now are being chained together in a way that automates exotic feature engineering • As LeCunn says: – “machines that learn to represent the world”
- 9. These are the features learned at each neuron in a Restricted Boltzmann Machine (RBMS) These features are passed to higher levels of RBMs to learn more complicated things. Part of the “7” digit
- 11. Deep Learning Architectures • Deep Belief Networks – Most common architecture • Convolutional Neural Networks – State of the art in image classification • Recurrent Networks – Timeseries • Recursive Networks – Text / image – Can break down scenes
- 12. DL4J Next Generation Deep Learning with
- 13. What is DL4J? • “The Hadoop of Deep Learning” – Command line driven – Java, Scala, and Python APIs • ASF 2.0 Licensed • Java implementation – Parallelization / GPU support • Runtime Neutral – Local – Hadoop / YARN – Spark – AWS • https://github.com/deeplearning4j/deeplearning4j
- 14. DL4J and Parallelization 14 Model Training Data Worker 1 Master Partial Model Global Model Worker 2 Partial Model Worker N Partial Model Split 1 Split 2 Split 3 … Traditional Serial Training Modern Parallel Engine (Hadoop / Spark)
- 15. DL4J Suite of Tools • DL4J – Main library for deep learning • Canova – Vectorization library • ND4J – Linear Algebra framework – Swappable backends (JBLAS, GPUs) • Arbiter – Model evaluation and testing platform
- 16. Canova for Command Line Vectorization • Library of tools to take – Audio – Video – Image – Text – CSV data • And convert the input data into vectors in a standardized format – Adaptable with custom input/output formats • Open Source, ASF 2.0 Licensed – https://github.com/deeplearning4j/Canova – Part of DL4J suite
- 17. DEEP LEARNING WORKFLOWS Enterprise Grade
- 18. Building Deep Learning Workflows • Our terminology in data science has gotten more exotic – But its still about gather, cleaning, visualizing, and feature construction of data • We need to get data from a raw format into a baseline raw vector – Which is why Canova exists • Deep Learning is not just classification – But an automated feature construction pipeline • Together, DL4J and Canova give us the base workflow
- 19. Modeling UCI Data: Iris • We need to vectorize the data – Possibly with some per column transformations – Which is why Canova exists • To feed raw data into a form DL4J can consume • We then need to build a deep learning model over the data – We’ll use the DL4J lib to do this
- 20. Vectorization with Canova • Setup the configuration file • Setup the schema transforms for the input CSV data • Generate the SVMLight vector data as the output – with the command line interface
- 22. Canova Vector Schema @RELATION UCIIrisDataset @DELIMITER , @ATTRIBUTE sepallength NUMERIC !NORMALIZE @ATTRIBUTE sepalwidth NUMERIC !NORMALIZE @ATTRIBUTE petallength NUMERIC !NORMALIZE @ATTRIBUTE petalwidth NUMERIC !NORMALIZE @ATTRIBUTE class STRING !LABEL
- 23. Canova CLI ./bin/canova vectorize -conf /tmp/iris_conf.txt File path already exists, deleting the old file before proceeding... Output vectors written to: /tmp/iris_svmlight.txt
- 24. DL4J from the Command Line bin/dl4j Usage: dl4j [command] [params] Commands: train build a deep learning model test test a deep learning model predict score new records against a deep learning model
- 26. JSON Model Config {"sparsity":0.0,"useAdaGrad":true,"lr":0.10000000149011612,"corruptionLevel":0.30000001192092896,"numIterations":1000,"momentu m":0.5,"l2":0.0,"useRegularization":false,"momentumAfter":null,"resetAdaGradIterations":- 1,"numLineSearchIterations":100,"dropOut":0.0,"applySparsity":false,"weightInit":"VI","optimizationAlgo":"CONJUGATE_GRADIENT", "lossFunction":"RECONSTRUCTION_CROSSENTROPY","concatBiases":false,"constrainGradientToUnitNorm":false,"seed":123,"nIn":0,"nOut ":0,"activationFunction":"sigmoid","visibleUnit":"BINARY","hiddenUnit":"BINARY","k":1,"weightShape":[0,0],"filterSize":[2,2,2, 2],"numFeatureMaps":2,"featureMapSize":[2,2],"stride":[2,2],"kernel":5,"batchSize":0,"minimize":false,"rng":"org.apache.common s.math3.random.MersenneTwister","layerFactory":"org.deeplearning4j.nn.layers.factory.PretrainLayerFactory,org.deeplearning4j.m odels.featuredetectors.rbm.RBM","stepFunction":"org.deeplearning4j.optimize.stepfunctions.DefaultStepFunction","renderWeightIt erations":-1,"dist":"org.apache.commons.math3.distribution.NormalDistributiont{SQRT2=1.4142135623730951, mean=0.0, solverAbsoluteAccuracy=1.0E-9, DEFAULT_INVERSE_ABSOLUTE_ACCURACY=1.0E-9, standardDeviation=0.1, logStandardDeviationPlusHalfLog2Pi=-1.3836465597893728, serialVersionUID=8589540077390120676}","listeners":["org.deeplearning4j.optimize.listeners.ScoreIterationListenert{printItera tions=10, log=Logger[org.deeplearning4j.optimize.listeners.ScoreIterationListener]},"]}
- 27. DL4J CLI Training ./bin/dl4j train -conf /tmp/iris_conf.txt
- 28. Skymind as DL4J Distribution • Just as Redhat was to Linux – A distribution of Linux with enterprise grade packaging • Just as Cloudera was to Hadoop – A distribution of Apache Hadoop with enterprise grade packaging • Skymind is to DL4J – A distribution of DL4J (+tool suite) with enterprise grade packaging
- 29. Thank you for your time and attention “Deep Learning: A Practitioner’s Approach” (Oreilly, October 2015)
Editor's Notes
- #8: we plot the learned filter for each hidden neuron, one per column of W. Each filter is of the same dimension as the input data, and it is most useful to visualize the filters in the same way as the input data is visualized. In the cases of image patches, we show each filter as an image patch
- #9: we plot the learned filter for each hidden neuron, one per column of W. Each filter is of the same dimension as the input data, and it is most useful to visualize the filters in the same way as the input data is visualized. In the cases of image patches, we show each filter as an image patch
- #10: we plot the learned filter for each hidden neuron, one per column of W. Each filter is of the same dimension as the input data, and it is most useful to visualize the filters in the same way as the input data is visualized. In the cases of image patches, we show each filter as an image patch
- #15: POLR: Parallel Online Logistic Regression Talking points: wanted to start with a known tool to the hadoop community, with expected characteristics Mahout’s SGD is well known, and so we used that as a base point