Deep Learning in the Wild with Arno Candel
0 likes1,595 views
The document discusses a presentation by Arno Candel from H2O.ai on 'deep learning in the wild', covering H2O's open-source predictive analytics platform designed for effective machine learning. Key topics include deep learning methods, live demos of applications like MNIST digit classification, Higgs boson classification, and eBay text classification, showcasing H2O's speed, scalability, and advanced features. The presentation emphasizes community engagement and the continuous improvement of the H2O platform for real-world data applications.
![H2O Deep Learning, @ArnoCandel 6
H2O Architecture - Designed for speed,
scale, accuracy & ease of use
Key technical points:
• distributed JVMs + REST API
• no Java GC issues
(data in byte[], Double)
• loss-less number compression
• Hadoop integration (v1,YARN)
• R package (CRAN)
Pre-built fully featured algos:
K-Means, NB, PCA, CoxPH,
GLM, RF, GBM, DeepLearning](https://image.slidesharecdn.com/h2odeeplearningarnocandel031115-150312000848-conversion-gate01/85/Deep-Learning-in-the-Wild-with-Arno-Candel-6-320.jpg)
Deep Learning in the Wild with Arno Candel
- 1. Deep Learning in the Wild H2o Meetup Mountain View, 3/11/2015 Arno Candel, H2O.ai
- 2. Who am I? PhD in Computational Physics, 2005 from ETH Zurich Switzerland 6 years at SLAC - Accelerator Physics Modeling 2 years at Skytree - Machine Learning 15 months at H2O.ai - Machine Learning 15 years in Supercomputing & Modeling • Named “2014 Big Data All-Star” by Fortune Magazine • http://www.kdnuggets.com/tag/arno-candel @ArnoCandel
- 3. H2O Deep Learning, @ArnoCandel Outline Introduction (5 mins) Methods & Implementation (5 mins) Results and Live Demos (20 mins) MNIST handwritten digits Higgs boson classification Ebay text classification h2o-dev Outlook: Flow, Python 3
- 4. H2O Deep Learning, @ArnoCandel Teamwork at H2O.ai Java, Apache v2 Open-Source #1 Java Machine Learning in Github Join the community! 4
- 5. H2O Deep Learning, @ArnoCandel H2O: Open-Source (Apache v2) Predictive Analytics Platform 5
- 6. H2O Deep Learning, @ArnoCandel 6 H2O Architecture - Designed for speed, scale, accuracy & ease of use Key technical points: • distributed JVMs + REST API • no Java GC issues (data in byte[], Double) • loss-less number compression • Hadoop integration (v1,YARN) • R package (CRAN) Pre-built fully featured algos: K-Means, NB, PCA, CoxPH, GLM, RF, GBM, DeepLearning
- 7. H2O Deep Learning, @ArnoCandel Wikipedia: Deep learning is a set of algorithms in machine learning that attempt to model high-level abstractions in data by using architectures composed of multiple non-linear transformations. What is Deep Learning? Input: Image Output: User ID 7 Example: Facebook DeepFace
- 8. H2O Deep Learning, @ArnoCandel What is NOT Deep Linear models are not deep (by definition) Neural nets with 1 hidden layer are not deep (only 1 layer - no feature hierarchy) SVMs and Kernel methods are not deep (2 layers: kernel + linear) Classification trees are not deep (operate on original input space, no new features generated) 8
- 9. H2O Deep Learning, @ArnoCandel 1970s multi-layer feed-forward Neural Network (stochastic gradient descent with back-propagation) + distributed processing for big data (fine-grain in-memory MapReduce on distributed data) + multi-threaded speedup (async fork/join worker threads operate at FORTRAN speeds) + smart algorithms for fast & accurate results (automatic standardization, one-hot encoding of categoricals, missing value imputation, weight & bias initialization, adaptive learning rate, momentum, dropout/l1/L2 regularization, grid search, N-fold cross-validation, checkpointing, load balancing, auto-tuning, model averaging, etc.) = powerful tool for (un)supervised machine learning on real-world data H2O Deep Learning 9 all 320 cores maxed out
- 10. H2O Deep Learning, @ArnoCandel Adaptive learning rate - ADADELTA (Google) Automatically set learning rate for each neuron based on its training history Grid Search and Checkpointing Run a grid search to scan many hyper- parameters, then continue training the most promising model(s) Regularization L1: penalizes non-zero weights L2: penalizes large weights Dropout: randomly ignore certain inputs Hogwild!: intentional race conditions Distributed mode: weight averaging 10 “Secret” Sauce to Higher Accuracy
- 11. H2O Deep Learning, @ArnoCandel MNIST: digits classification Standing world record: Without distortions or convolutions, the best-ever published error rate on test set: 0.83% (Microsoft) 11 Train: 60,000 rows 784 integer columns 10 classes Test: 10,000 rows 784 integer columns 10 classes MNIST = Digitized handwritten digits database (Yann LeCun) Data: 28x28=784 pixels with (gray-scale) values in 0…255 Yann LeCun: “Yet another advice: don't get fooled by people who claim to have a solution to Artificial General Intelligence. Ask them what error rate they get on MNIST or ImageNet.”
- 12. H2O Deep Learning, @ArnoCandel 12 H2O Deep Learning beats MNIST Standard 60k/10k data No distortions No convolutions No unsupervised training No ensemble 10 hours on 10 16-core nodes World-record! 0.83% test set error http://learn.h2o.ai/content/hands-on_training/deep_learning.html
- 13. H2O Deep Learning, @ArnoCandel POJO Model Export for Production Scoring 13 Plain old Java code is auto-generated to take your H2O Deep Learning models into production!
- 14. H2O Deep Learning, @ArnoCandel Parallel Scalability (for 64 epochs on MNIST, with “0.83%” parameters) 14 Speedup 0.00 10.00 20.00 30.00 40.00 1 2 4 8 16 32 63 H2O Nodes (4 cores per node, 1 epoch per node per MapReduce) 2.7 mins Training Time 0 25 50 75 100 1 2 4 8 16 32 63 H2O Nodes in minutes
- 15. H2O Deep Learning, @ArnoCandel MNIST: Unsupervised Anomaly Detection with Deep Learning (Autoencoder) 15 The good The bad The ugly Download the script and run it yourself!
- 16. H2O Deep Learning, @ArnoCandel 16 Application: Higgs Boson Classification Higgs vs Background Large Hadron Collider: Largest experiment of mankind! $13+ billion, 16.8 miles long, 120 MegaWatts, -456F, 1PB/day, etc. Higgs boson discovery (July ’12) led to 2013 Nobel prize! http://arxiv.org/pdf/1402.4735v2.pdf Images courtesy CERN / LHC HIGGS UCI Dataset: 21 low-level features AND 7 high-level derived features (physics formulae) Train: 10M rows, Valid: 500k, Test: 500k rows
- 17. H2O Deep Learning, @ArnoCandel 17 Algorithm Paper’s* l-l AUC low-level H2O AUC all features H2O AUC Parameters (not heavily tuned), H2O running on 10 nodes Generalized Linear Model - 0.596 0.684 default, binomial Random Forest - 0.764 0.840 50 trees, max depth 50 Gradient Boosted Trees 0.73 0.753 0.839 50 trees, max depth 15 Neural Net 1 layer 0.733 0.760 0.830 1x300 Rectifier, 100 epochs Deep Learning 3 hidden layers 0.836 0.850 - 3x1000 Rectifier, L2=1e-5, 40 epochs Deep Learning 4 hidden layers 0.868 0.869 - 4x500 Rectifier, L1=L2=1e-5, 300 epochs Deep Learning 5 hidden layers 0.880 0.871 - 5x500 Rectifier, L1=L2=1e-5 Deep Learning on low-level features alone beats everything else! Prelim. H2O results compare well with paper’s results* (TMVA & Theano) Higgs Particle Detection with H2O *Nature paper: http://arxiv.org/pdf/1402.4735v2.pdf HIGGS UCI Dataset: 21 low-level features AND 7 high-level derived features Train: 10M rows, Test: 500k rows
- 18. H2O Deep Learning, @ArnoCandel Goal: Predict the item from seller’s text description 18 Train: 578,361 rows 8,647 cols 467 classes Test: 64,263 rows 8,647 cols 143 classes “Vintage 18KT gold Rolex 2 Tone in great condition” Data: Bag of words vector 0,0,1,0,0,0,0,0,1,0,0,0,1,…,0 vintagegold condition Text Classification
- 19. H2O Deep Learning, @ArnoCandel Out-Of-The-Box: 11.6% test set error after 10 epochs! Predicts the correct class (out of 143) 88.4% of the time! 19 Note 2: No tuning was done (results are for illustration only) Train: 578,361 rows 8,647 cols 467 classes Test: 64,263 rows 8,647 cols 143 classes Note 1: H2O columnar-compressed in-memory store only needs 60 MB to store 5 billion values (dense CSV needs 18 GB) Text Classification
- 20. H2O Deep Learning, @ArnoCandel 20 H2O GitBooks https://leanpub.com/u/h2oai
- 21. H2O Deep Learning, @ArnoCandel Re-Live H2O World! 21 http://h2o.ai/h2o-world/ http://learn.h2o.ai Watch the Videos Day 2 • Speakers from Academia & Industry • Trevor Hastie (ML) • John Chambers (S, R) • Josh Bloch (Java API) • Many use cases from customers • 3 Top Kaggle Contestants (Top 10) • 3 Panel discussions Day 1 • Hands-On Training • Supervised • Unsupervised • Advanced Topics • Markting Usecase • Product Demos • Hacker-Fest with Cliff Click (CTO, Hotspot)
- 22. H2O Deep Learning, @ArnoCandel H2O Kaggle Starter R Scripts 22 Final ranking: #26 out of 1604
- 23. H2O Deep Learning, @ArnoCandel Currently Ongoing Challenge 23
- 24. H2O Deep Learning, @ArnoCandel New h2o-dev: Flow-based GUI 24
- 25. H2O Deep Learning, @ArnoCandel h2o-dev: iPython Notebooks 25
- 26. H2O Deep Learning, @ArnoCandel Sparkling Water: Spark+H2O 26
- 27. H2O Deep Learning, @ArnoCandel Key Take-Aways H2O is an open source predictive analytics platform for data scientists and business analysts who need scalable and fast machine learning. H2O Deep Learning is ready to take your advanced analytics to the next level - Try it on your data! Join our Community and Meetups! https://github.com/h2oai h2ostream community forum www.h2o.ai @h2oai 27 Thank you!