Machine Learning Streams with Spark 1.0
6 likes2,298 views
This document provides an agenda and overview for a meetup on machine learning streams with Spark 1.0. The agenda includes sections on machine learning and business analytics, streams and real-time analytics, and a deep dive into MLlib. The MLlib section summarizes descriptive, predictive, and prescriptive algorithms in MLlib, including linear algebra techniques, summary statistics, SVD, PCA, Bayesian classification, logistic regression, SVM, regression, K-means, and matrix factorization. Gradient descent, L-BFGS, and KNN algorithms are also discussed.
Machine Learning Streams with Spark 1.0
- 1. Seattle Spark Meetup Machine Learning Streams with Spark 1.0 Drew Minkin Principal Program Manager, Ubix Labs
- 2. Machine Learning and Business Analytics Streams and Real Time Analytics Deep Dive into MLlib AGENDA
- 3. Machine Learning and Business Analytics
- 4. Machine Learning is Not A Spectator Sport
- 5. Machine Learning and Data Science http://drewconway.com/zia/2013/3/26/the-data-science-venn-diagram
- 6. Reactive Proactive ProductionResearch The Analytics Spectrum http://halobi.com/wp-content/uploads/Blog-1-1024x600.png Graph Data Management Simulation Process ImprovementContent Delivery Knowledge Management Data Modeling Visualization Data Quality Monitoring Analysis Optimization Algorithms Trialing Statistics Domain Expertise Integration Big Data Collaboration Descriptive Predictive Prescriptive
- 7. Five Families of Algorithms http://en.wikipedia.org/wiki/Wu_Xing Association Classification Estimation Forecasting Clustering
- 8. Classification http://akorra.com/2012/06/06/top-10-creatures-that-influenced-martial-arts/ Target a Discrete Answer –Yes/No § Find All Columns Driving its Value § Use model to score new records § Many Different Measures of Accuracy § Quick and Improving Iterations § Most Actionable Types of Models § Hospital Readmission § Equipment Failure § Likelihood to purchase Examples Credit Scoring Banding
- 9. Association and Sequencing http://38.media.tumblr.com/tumblr_m81wcfIO3V1qmzwx0o1_1280.jpg Examples § Collaborative Filtering § Identify cross-sell § Identify sequential, next-sale § Make purchase recommendations § Complex event associations § Transactions and items in § Rules, Sequences and Itemsets out Recommender Systems
- 10. Forecasting and Time Series http://akorra.com/2012/06/06/top-10-creatures-that-influenced-martial-arts/ • Input of measure over time and related series • Predictions generated for short term trends • Based on cycles and events Examples § Workforce Optimization § Timing Purchasing Decisions § Optimizing Maintenance Windows § Material Cost Planning § Equipment Usage Planning Demand Sensing
- 11. Estimation and Regression http://akorra.com/2012/06/06/top-10-creatures-that-influenced-martial-arts/ Predicting a Continuous Distribution § Many Different Measures of Accuracy § Quick and Improving Iterations § Most Actionable Types of Models § Length Of Stay Estimation § Customer Lifetime Value Examples Pricing Optimization
- 12. Clustering http://akorra.com/2012/06/06/top-10-creatures-that-influenced-martial-arts/ § Hard and Soft Groupings § Profiles of Subgroups § Likenesses and Differences Examples • Marketing Campaigns • Reward Programs • Equipment Utilization • Process Improvement Analysis Market Segmentation
- 13. Combining Algorithms in Harmony http://en.wikipedia.org/wiki/Wu_Xing
- 14. Streams and Real Time Analytics
- 15. The Challenges of Scaling Analytics Classes of Analytics Complexity Spark vs. Storm, etc. Stream Paradigms and Spark AGENDA Streams and Real Time Analytics
- 16. Will Business Run out of Modeling Opportunities?
- 17. The Approaching Crisis for Machine Learning
- 18. Hype vs. Reality in Scaling Data Science http://www.kdnuggets.com/2013/04/poll-results-largest-dataset-analyzed-data-mined.html
- 19. 2009 vs. 2014 Scaling Data Science http://www.kdnuggets.com
- 20. Spectrum of Stream Based Analytics Latency Events/Sec Months Days Hours Minutes Seconds 100 ms < 1 ms 0 10 102 103 104 105 106 Big Data NoSQL RDBMS Business Monitoring Machine Monitoring Real Time Monitoring Web Analytics EDW Analytics Operational Analytics http://www.cs.ucr.edu/~mueen/ppt/StreamInsigh%205%20SLIDE%20DEMO.pptx
- 21. Challenges of Stream Based Applications http://www.cs.ucr.edu/~mueen/ppt/StreamInsigh%205%20SLIDE%20DEMO.pptx Devices Sensors Web servers Feeds Complex Analytics & Mining
- 22. Challenges of Stream Based Applications http://www.cs.ucr.edu/~mueen/ppt/StreamInsigh%205%20SLIDE%20DEMO.pptx Hopping Windows Tumbling Windows Event Synchronization Latency Time Window Management
- 23. Deep Dive into MLlib
- 24. Architecture Descriptive Analytics Predictive Analytics Prescriptive Analytics AGENDA Deep Dive into MLlib
- 25. MLlib Descriptive Analytics http://halobi.com/wp-content/uploads/Blog-1-1024x600.png Graph Data Management Simulation Process Improvement Reactive Proactive ProductionResearch Content Delivery Knowledge Management Data Modeling Visualization Data Quality Monitoring Analysis Optimization Algorithms Trialing Statistics Domain Expertise Integration Big Data Collaboration
- 26. MLlib Descriptive Analytics - Data Types http://halobi.com/wp-content/uploads/Blog-1-1024x600.png Vectors • Dense
- 27. MLlib Descriptive Analytics - Data Types http://halobi.com/wp-content/uploads/Blog-1-1024x600.png Vectors • Sparse
- 28. MLlib Descriptive Analytics - Data Types http://halobi.com/wp-content/uploads/Blog-1-1024x600.png Linear Algebra • CoordinateMatrix • DistributedMatrix • IndexedRow • IndexedRowMatrix • MatrixEntry • RowMatrix
- 29. MLlib Descriptive Analytics – Summary Statistics http://halobi.com/wp-content/uploads/Blog-1-1024x600.png Sample size Maximum value of each column Sample mean vector Minimum value of each column Number of nonzero elements Sample variance vector
- 30. MLlib Descriptive Analytics - SVD http://public.lanl.gov/mewall/kluwer2002.html Singular Value Decomposition Can Collapse Sparse Matrices to Denser Forms
- 31. MLlib Descriptive Analytics – PCA http://halobi.com/wp-content/uploads/Blog-1-1024x600.png Primary Component Analysis Reduces Dimensionality with Feature Selection
- 32. MLLib Predictive Analytics http://halobi.com/wp-content/uploads/Blog-1-1024x600.png Graph Data Management Simulation Process Improvement Reactive Proactive ProductionResearch Content Delivery Knowledge Management Data Modeling Visualization Data Quality Monitoring Analysis Optimization Algorithms Trialing Statistics Domain Expertise Integration Big Data Collaboration
- 33. MLlib Predictive Analytics – Bayesian Classifier http://xkcd.com/1132/
- 34. MLlib Predictive Analytics – Logistic Regression http://halobi.com/wp-content/uploads/Blog-1-1024x600.png Grandaddy of Algorithms Coefficients from states or exact values Small scores can make big changes
- 35. MLlib Predictive Analytics - SVM http://www.youtube.com/watch?v=3liCbRZPrZA http://www.projectrho.com/public_html/rocket/fasterlight.php Linear Support Vector Machine for classifiers Behold the “kernel trick”
- 36. MLlib Predictive Analytics – Regression http://halobi.com/wp-content/uploads/Blog-1-1024x600.png Linear Ridge Least Absolute Shrinkage & Selection Operator
- 37. MLlib Predictive Analytics – Kmeans http://halobi.com/wp-content/uploads/Blog-1-1024x600.png
- 38. MLlib Predictive Analytics – Matrix Factorization http://halobi.com/wp-content/uploads/Blog-1-1024x600.png Collaborative Filtering Alternating Least Squares (ALS)
- 39. Reactive Proactive ProductionResearch Prescriptive Analytics http://halobi.com/wp-content/uploads/Blog-1-1024x600.png Graph Data Management Simulation Process ImprovementContent Delivery Knowledge Management Data Modeling Visualization Data Quality Monitoring Analysis Optimization Algorithms Trialing Statistics Domain Expertise Integration Big Data Collaboration
- 40. MLlib Prescriptive Analytics – Gradient Descent http://bleedingedgemachine.blogspot.com/2012/12/gradient-descent.html http://kungfupanda.wikia.com/wiki/Monkey Linear and Nonlinear Optimization minimize smooth functions without constraints,
- 41. MLlib Prescriptive Analytics – L-BFGS http://graphics.utdallas.edu/sites/default/files/gpucvt.png Limited-Memory BFGS Nonlinear Minimize Smoothing Constraint is Memory
- 42. Notes from the MLlib Streams Field
- 43. MLlib Predictive Analytics – K Nearest Neighbor http://www.youtube.com/watch?v=3liCbRZPrZA http://www.projectrho.com/public_html/rocket/fasterlight.php Variation for classifiers
- 44. MLlib – A Call to Action http://www.fanpop.com/clubs/voltron/images/2172709/title/original-fanart http://adventuretime.wikia.com/wiki/Princess_Monster_Wife Coming Soon • Decision Trees • Model Performance Tools It Takes A Village • Time Series • Ensemble MLI