MLSD18. End-to-End Machine Learning
1 like333 views
The document summarizes a presentation about machine learning. It discusses what machine learning is and is not, provides examples of machine learning tasks like predictive maintenance and personalized music, and addresses realities of applying machine learning like the fact that all models are wrong and the concepts of overfitting and underfitting. It also discusses choosing machine learning algorithms and automating the process.
MLSD18. End-to-End Machine Learning
- 1. 1st edition November 4-5, 2018 Machine Learning School in Doha
- 2. BigML, Inc X End-to-end Machine Learning How to get from here to there Poul Petersen CIO, BigML
- 3. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Sampling the Audience Expert: Published papers at KDD, ICML, NIPS, etc or developed own ML algorithms used at large scale Aficionado: Understands pros/cons of different techniques and/or can tweak algorithms as needed Practitioner: Very familiar with ML packages (Weka, Scikit, BigML, etc.) Newbie: Just taking Coursera ML class or reading an introductory book to ML Absolute beginner: ML sounds like science fiction
- 4. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · A Present for You
- 5. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Free 1-Month PRO Subscription https://bigml.com/accounts/register/ MLSD18
- 6. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · What is Machine Learning?
- 7. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · What is Machine Learning? Let’s start with what is NOT Machine Learning… • Sentience • Killer robots • Generalized Artificial Intelligence • Anything to do with the word “singularity”
- 8. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Oh the Hype! AlphaGo Zero beats a human at Go… killer robots far off? • First of all, AlphaGo Zero is impressive! • But, no need to fear killer robots power by AlphaGo Zero: • Learning is not transferrable: retrain for chess, etc. • Works only for rule based systems / perfect simulator • Relies on games/systems with clear objectives (win/lose) • Cost $25 million1 “While AlphaGo Zero is a step towards a general-purpose AI, it can only work on problems that can be perfectly simulated in a computer, making tasks such as driving a car out of the question. AIs that match humans at a huge range of tasks are still a long way off” - Demis Hassabis, CEO of DeepMind2 2. https://www.theguardian.com/science/2017/oct/18/its-able-to-create-knowledge-itself-google-unveils-ai-learns-all-on-its-own 1. https://www.inc.com/lisa-calhoun/google-artificial-intelligence-alpha-go-zero-just-pressed-reset-on-how-we-learn.html
- 9. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Three Domains Artificial Cool/Scary things… that mostly don’t exist Machine AI Concepts applied to very specific problems Deep Learning Specific techniques of Machine Learning
- 10. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · What is Machine Learning? Let’s start with what is NOT Machine Learning… • Sentience • Killer robots • Generalized Artificial Intelligence • Anything to do with the word “singularity” • Something “new” • First International Conference on ML held in 1980 • Top-performing algorithms have been around for decades
- 11. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · ML Adoption "The gap for most companies isn’t that machine learning doesn’t work, but that they struggle to actually use it” • Why? • Too much focus on algorithms • Not enough focus on applying Machine
- 12. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Machine Learning Evolution Genesis Custom built Product Service Utility Academics & Researchers Scientists Developers Analysts Everyone 1950s 2000s 2011 2030 Commodity 2020 Ubiquity CertaintyUnknown Defined NovelCommon Weka, Scikit BigML, Azure ML, Amazon ML, Google Cloud ML1st Workshop on Machine Learning 1980 1980 ML is a largely a commodity already, but applying it is still an art
- 13. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · AIRLINE ORIGIN DESTINATION DEPARTURE DELAY DISTANCE ARRIVAL DELAY AS ANC SEA -11 1448,0 -22 AA LAX PBI -8 2330,0 -9 US SFO CLT -2 2296,0 5 AA LAX MIA -5 2342,0 -9 AS SEA ANC -1 1448,0 -21 DL SFO MSP -5 1589 8 NK LAS MSP -6 1299 -17 US LAX CLT 14 2125,0 -10 AA SFO DFW -11 1464,0 -13 DL LAS ATL 3 1747,0 -15 What is Machine Learning? Finding patterns in data that can be used to make inferences Predictive Models A practical definition…
- 14. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Machine Learning TerminologyInstances Features New Instance Predictive model Prediction Confidence ML algorithm Label Training / Learning Predicting / Scoring Data
- 15. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Why Machine Learning?
- 16. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Why Machine LearningCOMPLEXITYOFTASKS TIME20th century 21st century - +
- 17. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Traditional Programming Lost Baggage Policy • Explicit rules defined by requirements and experience • How do we program when the rules are unknown or very difficult to determine?
- 18. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Programming with ML AIRLINE ORIGIN DESTINATION DEPARTURE DELAY DISTANCE ARRIVAL DELAY AS ANC SEA -11 1448,0 -22 AA LAX PBI -8 2330,0 -9 US SFO CLT -2 2296,0 5 AA LAX MIA -5 2342,0 -9 AS SEA ANC -1 1448,0 -21 DL SFO MSP -5 1589 8 NK LAS MSP -6 1299 -17 US LAX CLT 14 2125,0 -10 AA SFO DFW -11 1464,0 -13 DL LAS ATL 3 1747,0 -15 Have: Flight Delay Data Want: Flight Delay Prediction Flight Delay Model???? What else can ML do?
- 19. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Machine Learning Tasks
- 20. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Machine Learning Tasks CLUSTER ANALYSIS ANOMALY DETECTION ASSOCIATION DISCOVERY TOPIC MODELING TIME SERIES UNSUPERVISED CLASSIFICATION AND REGRESSION SUPERVISED
- 21. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Predictive Maintenance CLASSIFICATION Will this component fail? REGRESSION How many days until this component fails? TIME SERIES FORECASTING How many components will fail in a week from now? CLUSTER ANALYSIS Which machines behave similarly? ANOMALY DETECTION Is this behavior normal? ASSOCIATION DISCOVERY What alerts are triggered together before a failure?
- 22. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Personalized Music CLASSIFICATION Will this song be a hit? REGRESSION How many users will play this song next month? TIME SERIES FORECASTING How many downloads this song will have in 3 months? CLUSTER ANALYSIS Which songs are similar? ANOMALY DETECTION Is this song being played more than normal? ASSOCIATION DISCOVERY What songs people like to play together?
- 23. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Airline Revenue Management CLASSIFICATION Will this flight be booked at 80% 14 days out? REGRESSION How many passengers will book this flight 7 days out? TIME SERIES FORECASTING How many tickets will be cancelled this week? CLUSTER ANALYSIS Which flight booking patterns are similar? ANOMALY DETECTION Are these flights booking patterns normal? ASSOCIATION DISCOVERY What price changes help overbook sooner?
- 24. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Network Security CLASSIFICATION Is this email part of a phishing attack? REGRESSION How many logins after work per week? TIME SERIES FORECASTING What will be the number of false alarms next week? CLUSTER ANALYSIS Are these users behaving similarly? ANOMALY DETECTION Is this user behavior worth to inspect? ASSOCIATION DISCOVERY What alerts were triggered before this attack?
- 25. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · ML Reality Check
- 26. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Reality Check • All Machine Learned models are wrong • Real-world Machine Learning is iterative • End-to-end Machine Learning is compositional Three Important Concepts in Applying ML…
- 27. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · All ML Models are Wrong
- 28. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Model Complexity TRUE FALSE DEEPNET ENSEMBLELOGISTIC REGRESION DECISION TREE Some model(s) is wrong… which one? Same patient… different models
- 29. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Evaluating Models TEST TRAINING CONFIDENCEPREDICTION % EVALUATION % ENSEMBLE PATIENT DATA
- 30. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Measuring ML Mistakes TRUE FALSE TRUE TRUE POSITIVE FALSE POSITIVE FALSE FALSE NEGATIVE TRUE NEGATIVE MODEL ACTUAL We can bend the rules a bit…
- 31. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Operating Point TRUE FALSE 100% 0% 0% 100% Operating Point More False Positives More False Negatives Why would you do this?
- 32. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Comparing Models %TRUEPOSITIVES % FALSE POSITIVES WORST(?) MODEL IDEAL MODEL GOOD BETTER R AN D O M TRIVIAL MODEL TRIVIAL MODEL
- 33. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Mistakes can be Costly + = FUN! DANGER!
- 34. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Cost Functions GOOD BETTER?%TRUEPOSITIVES % FALSE POSITIVES • What is the cost of predicting cancer incorrectly? • What is the cost of labeling a fraudulent transaction as valid? • What is the cost of incorrectly predicting an aircraft part is safe? • Why can’t I just have a perfect model? FALSE NEGATIVE COST FALSE POSITIVE COST One possibility
- 35. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · How it Goes All Wrong • Over-fitting • Under-fitting
- 36. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Hunting Dog Image Classifier TRU E FAL SE Which images are pictures of dogs that are bred to be hunters?
- 37. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Over-fitting… “Hunting dogs are short- haired spotted puppies that lay out on the grass”
- 38. A perfect model! How about some new images… TRU E FAL SE
- 39. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Over-fitting Model: true Reality: false Model: false Reality: true • This is an example or poor generalization • The model “fit” the training data perfectly • But it does not generalize to new instances well
- 40. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Under-fitting “Dogs with drop or pendant ears are hunters” Only use ear shape:
- 41. An imperfect model… now we are making some mistakes on the training data. TRU E FAL SE
- 42. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Under-fitting • This is an example of good generalization • The model “under-fit” the training data • But it is generalizing to new instances better Model: true Reality: true Model: false Reality: false
- 43. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Under-fitting Model: false Reality: true Model: false Reality: true
- 44. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Learning Problems / Complexity Under-fitting Over-fitting • High Complexity Model • Fitting the data too well • Low Complexity Model • Not fitting the data very well One way to mitigate this is with different types of models…
- 45. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Choosing the ML Algorithm Decreasing Interpretability / Better Representation / Longer Training IncreasingDataSize/Complexity Early Stage Rapid Prototyping Mid Stage Proven Application Late Stage Critical Performance DeepnetsSingle Tree Model Logistic Regression Boosted Trees Random Decision Forest Decision Forest Hard?
- 46. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Automating Machine Learning
- 47. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Deepnet Structure x1 x2 x3 x4 y1 y2 y3Outputs Inputs h1 h2 h3 h4 h5 Hidden layer 3 Classes 4 Features h1 h2 h3 h4 h5 Hidden layer h1 h2 h3 h4 h9 Hidden layer…. h1 = activation?(wx, x) ?
- 48. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · BigML Deepnet • The success of a Deepnet is dependent on getting the right network structure for the dataset • But, there are too many parameters: • Nodes, layers, activation function, learning rate, etc… • And setting them takes significant expert knowledge • Solution: Metalearning (a good initial guess) • Solution: Network search (try a bunch)
- 49. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Automating Machine Learning http://www.clparker.org/ml_benchmark/
- 50. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Automating Machine Learning • Each resource has several parameters that impact quality • Number of trees, missing splits, nodes, weight • Rather than trial and error, we can use ML to find ideal parameters • Why not make the model type, Decision Tree, Boosted Tree, etc, a parameter as well? • Similar to Deepnet network search, but finds the optimum machine learning algorithm and parameters for your data automatically Key Insight: We can solve any parameter selection problem in a similar way.
- 51. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · BigML OptiML
- 52. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Fusions Key Insight: ML algorithms each have unique strengths and weaknesses Single Tree: output changes abruptly with inputs near decision boundary Tree + Deepnet: output changes smoothly with inputs near decision boundary
- 53. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Fusions Model Skills: Some ML algorithms “generally” do better on some feature types: • RDF for sparse text vectors • LR/Deepnets for numeric features • Trees for categorical features Full Numeric Text
- 54. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Reality Check • All Machine Learned models are wrong • Real-world Machine Learning is iterative • End-to-end Machine Learning is compositional Three Important Concepts in Applying ML…
- 55. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Real-world ML is Iterative
- 56. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Real-world ML Applications • Should you sign that NDA? • Upload the NDA to the website • The service uses Machine Learning to decide if the terms are fair https://ndalynn.com/
- 57. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Real-world ML Applications • Gathers over 500 features about companies: • Crunchbase / Tweets / Patents / LinkedIn / etc. • Creates a label for success/failure: • IPO or acquisition = success • Bankruptcy or irrelevance = failure • Uses Machine Learning to build a model that predicts the success or failure of startups • And puts all of the information together into an investor dashboard https://preseries.com
- 58. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Real-world ML Applications https://thepointsguy.com/news/this-is-the-reason-you-arent-feeling-as-much-turbulence-on-delta-flights/ …collecting and analyzing “hundreds of thousands of data points,” with a plan to boost that to “millions,” creating a model that forecasts turbulence with a level of confidence heretofore unseen.
- 59. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Reality of a ML Application Data Transformations Feature Engineering Data Collection Evaluation & Retraining Seen Unseen ML Application
- 60. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Effort of a ML Application State the problem as an ML task Data wrangling Feature engineering Modeling and Evaluations Predictions Measure Results Data transformations ~80% effort ~5% effort ~5% effort This is only such low effort because of platforms like This is an area where is currently innovating Task ~10% effort Effort
- 61. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Reality Check • All Machine Learned models are wrong • Real-world Machine Learning is iterative • End-to-end Machine Learning is compositional Three Important Concepts in Applying ML…
- 62. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · End-to-end ML Compositions
- 63. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · End-to-end ML is Compositional • Real-world problems • Solved by applying a combination of algorithms • Very rarely is it one-and-done
- 64. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Basic Workflow SOURCE DATASET MODEL PREDICTION
- 65. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Feature Engineering MODEL FILTERSOLD HOMES BATCH PREDICTION NEW FEATURES DATASET DEALS DATASET FILTERFORSALE HOMES NEW FEATURES
- 66. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · End-to-end ML is Compositional • Real-world problems • Solved by applying a combination of algorithms • Very rarely is it one-and-done • Each “step” is often multi-stage as well • Filtering/Cleaning data
- 67. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Anomaly Filter and Evaluate DIABETES SOURCE DIABETES DATASET TRAIN SET TEST SET ALL MODEL CLEAN DATASET FILTER ALL MODEL ALL EVALUATION CLEAN EVALUATION COMPARE EVALUATIONS ANAOMALY DETECTOR
- 68. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Fixing Missing Values Fix Missing Values in a “Meaningful” Way Filter Zeros Model insulin Predict insulin Select insulin Fixed Dataset Amended Dataset Original Dataset Clean Dataset
- 69. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · End-to-end ML is Compositional • Real-world problems • Solved by applying a combination of algorithms • Very rarely is it one-and-done • Each “step” is often multi-stage as well • Filtering/Cleaning data • Tuning a model for optimum performance
- 70. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Ensemble Tuning ENSEMBLE N=20 EVALUATION SOURCE DATASET TRAINING TEST EVALUATIONEVALUATION ENSEMBLE N=10 ENSEMBLE N=1000 CHOOSE
- 71. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · End-to-end ML is Compositional • Real-world problems • Solved by applying a combination of algorithms • Very rarely is it one-and-done • Each “step” is often multi-stage as well • Filtering/Cleaning data • Tuning a model for optimum performance • Finding the best features
- 72. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Best-first Feature Selection {F1} CHOOSE BEST S = {Fa} {F2} {F3} {F4} Fn S+{F1} S+{F2} S+{F3} S+{F4} S+{Fn-1} CHOOSE BEST S = {Fa, Fb} S+{F1} S+{F2} S+{F3} S+{F4} S+{Fn-1} CHOOSE BEST S = {Fa, Fb, Fc}
- 73. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · End-to-end ML is Compositional • Real-world problems • Solved by applying a combination of algorithms • Very rarely is it one-and-done • Each “step” is often multi-stage as well • Filtering/Cleaning data • Tuning a model for optimum performance • Finding the best features • May require models for several domains of knowledge • Multiple Training / Scoring
- 74. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · AGGREGATED BY CARD AGGREGATED BY USER AGGREGATED BY PROFILE Multiple Domains TRANSACTIONS ANOMALY BY CARD ANOMALY BY USER ANOMALY BY CARD ANOMALY SCORE ANOMALY SCORE ANOMALY SCORE NEW TRANSACTION APPROVED?
- 75. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · End-to-end ML is Compositional • Real-world problems • Solved by applying a combination of algorithms • Very rarely is it one-and-done • Each “step” is often multi-stage as well • Filtering/Cleaning data • Tuning a model for optimum performance • Finding the best features • May require models for several domains of knowledge • Multiple Training / Scoring • Even after deploying a model • Workflow to monitor performance, know when to retrain
- 76. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Model Retraining TRAINING INPUT DATA PREDICTIONS ANOMALY SCORES OUTCOMES RETRAIN DATA
- 77. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Reality Check • All Machine Learned models are wrong Three Important Concepts in Applying ML… • Real-world Machine Learning is iterative • End-to-end Machine Learning is compositional
- 78. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · • Better features always beat better algorithms • Good algorithms already exist and are good enough • Tools like OptiML exist which can help optimize performance • The data is never good enough Tenets of Machine Learning • All Machine Learned models are wrong • Real-world Machine Learning is iterative • End-to-end Machine Learning is compositional • Automation is better than hand tuning - you need an API! • When data changes quickly, training speed is more important than accuracy • Repeatability is superior to a single strong result • Problems are solved with workflows of algorithms • A ML solution is not real until it is in production • ML is here: Now we need 100,000x people applying ML , but some are useful