All Models Are Wrong, But Some Are Useful: 6 Lessons for Making Predictive Analytics Work
2 likes1,141 views
The document outlines six essential lessons for effective predictive analytics, emphasizing that all models are imperfect yet some can be useful. Key lessons include the importance of using better data over larger models, the need for careful evaluation measures, and sticking to simpler solutions before introducing complexity. It also highlights various predictive modeling techniques and stresses the significance of tuning models while avoiding overfitting.
![Deep-learning methods are representaUon-learning
methods with mulple levels of representaon,
obtained by composing simple but non-linear modules
that each transform the representaon at one level
(starng with the raw input) into a representaon at a
higher, slightly more abstract level.
[LeCun et al, 2014]
Deep Learning
Yann LeCun, Yoshua Bengio & Geoffrey Hinton
http://www.nature.com/nature/journal/v521/n7553/full/nature14539.html](https://image.slidesharecdn.com/predictiveanalyticsruletalk-ver0-170219150002/85/All-Models-Are-Wrong-But-Some-Are-Useful-6-Lessons-for-Making-Predictive-Analytics-Work-73-320.jpg)
All Models Are Wrong, But Some Are Useful: 6 Lessons for Making Predictive Analytics Work
- 6. if LOAN-SALARY RATIO < 1.5 then OUTCOME=’repay’ else if LOAN-SALARY RATIO > 4 then OUTCOME=’default’ else if AGE < 40 and OCCUPATION =’industrial’then OUTCOME=’default’ else OUTCOME=’repay’ end if Fundamentals of Machine Learning for Predictive Data Analytics John Kelleher, Brian Mac Namee, and Aoife D'Arcy www.machinelearningbook.com
- 10. Better data usually beats bigger models Prediction is a lot of things1 2 There is no such thing as a free lunch 3 Look for Goldilocks 4 Choose your evaluation carefully5 6 Remember Occam’s Razor
- 14. Forecast
- 15. 0 27.5 55 82.5 110 July September November January March May
- 16. 0 27.5 55 82.5 110 July September November January March May
- 18. Label
- 19. Image Set
- 22. Rank
- 23. Population
- 30. "We have dubbed the associated results No Free Lunch theorems because they demonstrate that if an algorithm performs well on a certain class of problems then it necessarily pays for that with degraded performance on the set of all remaining problems." Wolpert & Macready "No Free Lunch Theorems for Optimization", David H. Wolpert and William G. Macready, IEEE Transactions On Evolutionary Computation, vol. 1, no. 1, 1997 http://ti.arc.nasa.gov/m/profile/dhw/papers/78.pdf
- 39. There are a huge number of different predictive modelling algorithms. You need to experiment with lots of different ones. Lesson random forest decision tree istonic regression neural network nearest neighbour naive Bayes support vector machine logistic regression Bayesian network ensemble gradient boosting linear model winnow
- 41. ● ● ● ● ● 0 20 40 60 80 100 20000400006000080000 Age Income Fundamentals of Machine Learning for Predictive Data Analytics John Kelleher, Brian Mac Namee, and Aoife D'Arcy www.machinelearningbook.com
- 42. ● ● ● ● ● 0 20 40 60 80 100 20000400006000080000 Age Income Fundamentals of Machine Learning for Predictive Data Analytics John Kelleher, Brian Mac Namee, and Aoife D'Arcy www.machinelearningbook.com
- 43. ● ● ● ● ● 0 20 40 60 80 100 20000400006000080000 Age Income Fundamentals of Machine Learning for Predictive Data Analytics John Kelleher, Brian Mac Namee, and Aoife D'Arcy www.machinelearningbook.com
- 44. ● ● ● ● ● 0 20 40 60 80 100 20000400006000080000 Age Income Fundamentals of Machine Learning for Predictive Data Analytics John Kelleher, Brian Mac Namee, and Aoife D'Arcy www.machinelearningbook.com
- 45. 0 50 100 150 200 0.10.20.30.40.5 Training Iteration MisclassificationRate Performance on Training Set Performance on Validation Set
- 46. 0 50 100 150 200 0.10.20.30.40.5 Training Iteration MisclassificationRate Performance on Training Set Performance on Validation Set Fundamentals of Machine Learning for Predictive Data Analytics John Kelleher, Brian Mac Namee, and Aoife D'Arcy www.machinelearningbook.com
- 47. 0 50 100 150 200 0.10.20.30.40.5 Training Iteration MisclassificationRate Performance on Training Set Performance on Validation Set Fundamentals of Machine Learning for Predictive Data Analytics John Kelleher, Brian Mac Namee, and Aoife D'Arcy www.machinelearningbook.com
- 48. 0 50 100 150 200 0.10.20.30.40.5 Training Iteration MisclassificationRate Performance on Training Set Performance on Validation Set Fundamentals of Machine Learning for Predictive Data Analytics John Kelleher, Brian Mac Namee, and Aoife D'Arcy www.machinelearningbook.com
- 49. 0 50 100 150 200 0.10.20.30.40.5 Training Iteration MisclassificationRate Performance on Training Set Performance on Validation Set Fundamentals of Machine Learning for Predictive Data Analytics John Kelleher, Brian Mac Namee, and Aoife D'Arcy www.machinelearningbook.com
- 50. 0 50 100 150 200 0.10.20.30.40.5 Training Iteration MisclassificationRate Performance on Training Set Performance on Validation Set Fundamentals of Machine Learning for Predictive Data Analytics John Kelleher, Brian Mac Namee, and Aoife D'Arcy www.machinelearningbook.com
- 51. 0 50 100 150 200 0.10.20.30.40.5 Training Iteration MisclassificationRate Performance on Training Set Performance on Validation Set Fundamentals of Machine Learning for Predictive Data Analytics John Kelleher, Brian Mac Namee, and Aoife D'Arcy www.machinelearningbook.com
- 52. 0 50 100 150 200 0.10.20.30.40.5 Training Iteration MisclassificationRate Performance on Training Set Performance on Validation Set Fundamentals of Machine Learning for Predictive Data Analytics John Kelleher, Brian Mac Namee, and Aoife D'Arcy www.machinelearningbook.com
- 53. 0 50 100 150 200 0.10.20.30.40.5 Training Iteration MisclassificationRate Performance on Training Set Performance on Validation Set
- 54. 0 50 100 150 200 0.10.20.30.40.5 Training Iteration MisclassificationRate Performance on Training Set Performance on Validation Set
- 58. Digital Image Processing, Gonzalez & Woods, 2002 Denoised image 100 200 300 400 500 600 50 100 150 200 250 300 350 400 450
- 62. Raw Activity
- 66. Activity
- 72. Deep Learning Google Trends: http://www.google.com/trends/ 2005 2007 2009 2011 2013 2015
- 73. Deep-learning methods are representaUon-learning methods with mulple levels of representaon, obtained by composing simple but non-linear modules that each transform the representaon at one level (starng with the raw input) into a representaon at a higher, slightly more abstract level. [LeCun et al, 2014] Deep Learning Yann LeCun, Yoshua Bengio & Geoffrey Hinton http://www.nature.com/nature/journal/v521/n7553/full/nature14539.html
- 77. On Welsh Corgis, Computer Vision, and the Power of Deep Learning, Microsoft Research, 2014 http://research.microsoft.com/en-us/news/features/dnnvision-071414.aspx Rise of the machines, The Economist, 2015 http://www.economist.com/news/briefing/21650526-artificial-intelligence-scares-peopleexcessively-so-rise-machines
- 81. A marketing company working for a charity has developed two different models that predict the likelihood that donors will respond to a mail- shot asking them to make a special extra donation. Two models have been built and an evaluation experiment had been performed. Now we must decide which model to use.
- 82. Prediction TRUE FALSE Target TRUE 2355 337 FALSE 329 1714 Classification Accuracy: 85.93% Model 1
- 83. Prediction TRUE FALSE Target TRUE 2198 494 FALSE 471 1572 Classification Accuracy: 79.62% Model 2