SlideShare a Scribd company logo

Barga Data Science lecture 6

Roger Barga, profile picture
Roger Barga

The document discusses feature extraction and selection as important steps in machine learning. It notes that better features often lead to better algorithms. It then describes five clusters identified through clustering analysis. Each cluster contains individuals (male or female) with certain average demographic characteristics like age, location, income, and whether they have accounts or loans. The document emphasizes that feature extraction and selection are underrated but important for machine learning.

Data & Analytics
Related topics:
Data Mining Insights • Data Science Insights • Predictive Analytics
1 of 142
Download to read offline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale Feature extraction and selection are the most important but underrated step of machine learning. Better features are better than better algorithms…
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale Lecture Objectives homework There is an order or workflow that takes place here, don’t lose the forest in the trees…
Deriving Knowledge from Data at Scale Review…
Deriving Knowledge from Data at Scale • Cluster 0 – It contains a cluster of Females with an average age of 37 who live in inner city and possess saving account number and current account number. They are unmarried and do not have any mortgage or pep. The average monthly income is 23,300. • Cluster 1 - It contains a cluster of Females with an average age of 44 who live in rural area and possess saving account number and current account number. They are married and do not have any mortgage or pep. The average monthly income is 27,772. • Cluster 2 - It contains a cluster of Females with an average age of 48 who live in inner city and possess current account number but no saving account number. They are unmarried and do not have mortgage but do have pep. The average monthly income is 27,668. • Cluster 3 - It contains a cluster of Females with an average age of 39 who live in town and possess saving account number and current account number. They are married and do not have any mortgage or pep. The average monthly income is 24,047. • Cluster 4 - It contains a cluster of Males with an average age of 39 who live in inner city and possess current account number but no saving account number. They are married and have mortgage and pep. The average monthly income is 26,359. • Cluster 5 - It contains a cluster of Males with an average age of 47 who live in inner city and possess saving account number and current account number. They are unmarried and do not have mortgage but do have pep. The average monthly income is 35,419.
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale Classifiers  Lazy –> IBk
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale15
Deriving Knowledge from Data at Scale No Prob Target CustID Age 1 0.97 Y 1746 … 2 0.95 N 1024 … 3 0.94 Y 2478 … 4 0.93 Y 3820 … 5 0.92 N 4897 … … … … … 99 0.11 N 2734 … 100 0.06 N 2422 Use a model to assign score (probability) to each instance Sort instances by decreasing score Expect more targets (hits) near the top of the list 3 hits in top 5% of the list If there 15 targets overall, then top 5 has 3/15=20% of targets
Deriving Knowledge from Data at Scale 40% of responses for 10% of cost Lift factor = 4 80% of responses for 40% of cost Lift factor = 2 Model Random
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale to impact… 1. Build our predictive model in WEKA Explorer; 2. Use our model to score (predict) which new customers to target in our upcoming advertising campaign; • ARFF file manipulation (hacking), all too common pita… • Excel manipulation to join model output with our customers list 3. Compute the lift chart to assess business impact of our predictive model on the advertising campaign • How are Lift charts built, of all the charts and/or performance measures from a model this one is ‘on you’ to construct; • Where is the business ‘bang for the buck’?
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale You can’t turn data lead into modeling gold – we’re data scientists, not data alchemists…
Deriving Knowledge from Data at Scale Motivation: Real world examples Example (1) Lesson: Correct data transformation is important!
Deriving Knowledge from Data at Scale Motivation: Real world examples Example (2): KDD Cup 2001 Lesson: A model that uses lots of features can turn out to be very sub-optimal, however well it is designed!
Deriving Knowledge from Data at Scale Motivation: Real world examples Example (3) Lesson: Feature selection can be crucial even when the number of features is small!
Deriving Knowledge from Data at Scale Motivation: Real world examples Example (4) Lesson: Variations of the same ML method can give vastly different performances!
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale Predictive modeling competitions
Deriving Knowledge from Data at Scale Global competitions 1½ weeks 70.8% Competition closes 77% State of the art 70% Predicting HIV viral load Improved by 10%
Deriving Knowledge from Data at Scale Mismatch between those with data and those with the skills to analyse it Crowdsourcing
Deriving Knowledge from Data at Scale Forecast Error (MASE) Existing model Tourism Forecasting Competition Aug 9 2 weeks later 1 month later Competition End
Deriving Knowledge from Data at Scale • neural networks • logistic regression • support vector machine • decision trees • ensemble methods • adaBoost • Bayesian networks • genetic algorithms • random forest • Monte Carlo methods • principal component analysis • Kalman filter • evolutionary fuzzy modeling Users apply different techniques
Deriving Knowledge from Data at Scale VicRoads has an algorithm they use to forecast travel time on Melbourne freeways (taking into account time, weather, accidents, etc). Their current model is inaccurate and somewhat useless. They want to do better (or at least find out about whether it’s possible to do better).
Deriving Knowledge from Data at Scale 1 2 3 Upload Submit Evaluate & Exchange
Deriving Knowledge from Data at Scale Use the wizard to post a competition
Deriving Knowledge from Data at Scale Participants make their entries
Deriving Knowledge from Data at Scale Competitions are judged based on predictive accuracy
Deriving Knowledge from Data at Scale Competition Mechanics Competitions are judged on objective criteria
Deriving Knowledge from Data at Scale Kaggle How They Won It…
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale Three Files ford_train • 510 Trials, ~1,200 observations each spaced by 0.1 sec -> 604,330 rows ford_test • 100 Trials,~1,200 observations/trial, 120,841 rows example_submission.csv
Deriving Knowledge from Data at Scale Junpei Komiyama (#4)
Deriving Knowledge from Data at Scale Junpei Komiyama (#4)
Deriving Knowledge from Data at Scale Mick Wagner (#2)
Deriving Knowledge from Data at Scale Mick Wagner (#2)
Deriving Knowledge from Data at Scale Inference (#1)
Deriving Knowledge from Data at Scale VicRoads has an algorithm they use to forecast travel time on Melbourne freeways (taking into account time, weather, accidents etc). Their current model is inaccurate and somewhat useless. They want to do better (or at least find out about whether it’s possible to do better).
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale François GUILLEM (#14)
Deriving Knowledge from Data at Scale #1 used Random Forests
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale Homework Week 6 Monday Sept. 21st Upload to site… http://blog.kaggle.com/category/dojo/ Content is 10 pages of interview on how the team(s) built their models, some have multiple interviews; You will review at least 10 interviews, bounce around do not go sequentially. 1) What model(s) did they use, 2) insights they had that influenced modeling, 3) what feature creation and selection, 4) other observations. I will cons all these together and upload as shared document on our site.
Deriving Knowledge from Data at Scale 5 Minute Break…
Deriving Knowledge from Data at Scale Course Project
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale https://www.kaggle.com/c/springleaf-marketing-response not Determine whether to send a direct mail piece to a customer
Deriving Knowledge from Data at Scale The Data
Deriving Knowledge from Data at Scale The Rules
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale what is the data telling you
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale Data Wrangling
Deriving Knowledge from Data at Scale Data Acquisition Data Exploration Pre- processing Feature and Target construction Train/ Test split Feature selection Model training Model scoring Model scoring Evaluation Evaluation Compare metrics
Deriving Knowledge from Data at Scale • Data preparation step is by far the most time consuming step 0 10 20 30 40 50 60 70 Understanding of Domain Understanding of Data Preparation of Data Data Mining Evaluation of Results Deployment of Results KDDM steps relative effort [%] Cabena et al. estimates Shearer estimates Cios and Kurgan estimates
Deriving Knowledge from Data at Scale Out of Class Reading, highly recommended
Deriving Knowledge from Data at Scale Out of Class Reading, highly recommended
Deriving Knowledge from Data at Scale 1. Do you have domain knowledge? 2. Are your features commensurate? 3. Do you suspect interdependence of features? 4. Do you need to prune the input variables 5. Do you need to assess features individually 6. Do you need a predictor? 7. Do you suspect your data is “dirty” 8. Do you know what to try first? 9. Do you have new ideas, time, computational resources, and enough examples? 10. Do you want a stable solution
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale 15 15 𝑃 = 0.5 𝑃 = 0.5
Deriving Knowledge from Data at Scale 15 157 13 𝑃 = 0.5 𝑃 = 0.5 𝑃 = 0.35 𝑃 = 0.65
Deriving Knowledge from Data at Scale 15 1515 15 𝑃 = 0.5 𝑃 = 0.510 10
Deriving Knowledge from Data at Scale 15 1515 15 𝑃 = 0.5 𝑃 = 0.5 Time T r a i n T e s t Horizontal Vertical
Deriving Knowledge from Data at Scale Data Characterization…
Deriving Knowledge from Data at Scale 1. Unique values 2. Most frequent values 3. Highest and lowest values 4. Location and dispersion – gini, statistical test for dispersion 5. Quartiles
Deriving Knowledge from Data at Scale 1. Missing values 2. Outliers 3. Coding 4. Constraints
Deriving Knowledge from Data at Scale Missing values – UCI machine learning repository, 31 of 68 data sets reported to have missing values. “Missing” can mean many things… MAR: "Missing at Random": – usually best case – usually not true Non-randomly missing Presumed normal, so not measured Causally missing – attribute value is missing because of other attribute values (or because of the outcome value!)
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale Outliers – may indicate ‘bad data’ or it may represent something scientifically interesting in the data… Simple working definition: an outlier is an element of a data sequence S that is inconsistent with expectations, based on the majority of other elements of S. Sources of outliers • Measurement errors • Other uninteresting anomalous data • Surprising observations that may be important
Deriving Knowledge from Data at Scale Outliers – may indicate ‘bad data’ or it may represent something scientifically interesting in the data… Simple working definition: an outlier is an element of a data sequence S that is inconsistent with expectations, based on the majority of other elements of S. Sources of outliers • Insurance company sees niche of sports car enthusiasts, married boomers with kids and second family car. Low risk, lower rate to attract. Simple case where outlier carries meaning for modeling…
Deriving Knowledge from Data at Scale Outliers can distort the regression results. When an outlier is included in the analysis, it pulls the regression line towards itself. This can result in a solution that is more accurate for the outlier, but less accurate for all the other cases in the data set. Outliers – may indicate ‘bad data’ or it may represent something scientifically interesting in the data…
Deriving Knowledge from Data at Scale Identify outliers • Question origin, domain knowledge invaluable • Dispersion – "spread" of a data set, departure from central tendency, use a box plot… Deal with outliers • Winsorize – Set all outliers to a specified percentile of the data. Not equivalent to trimming, which simply excludes data. In a Winsorized estimator, extreme values are instead replaced by certain percentiles (the trimmed minimum and maximum). Same as clipping in signal processing. Outliers – may indicate ‘bad data’ or it may represent something scientifically interesting in the data…
Deriving Knowledge from Data at Scale Identify outliers • Question origin, domain knowledge invaluable • Dispersion – "spread" of a data set, departure from central tendency, use a box plot… Deal with outliers • Include – Robust statistics, a convenient way to summarize results when they include a small proportion of outliers. A hot topic for research, see NIPS 2010 Workshop, Robust Statistical learning (robustml). Outliers – may indicate ‘bad data’ or it may represent something scientifically interesting in the data…
Deriving Knowledge from Data at Scale • Entity integrity • Referential integrity • Type checking • Format • Bounds checking Constraints
Deriving Knowledge from Data at Scale • weka.filters.unsupervised.instance.RemoveMisclassified • weka.filters.unsupervised.instance.RemovePercentage • weka.filters.unsupervised.instance.RemoveRange • weka.filters.unsupervised.instance.RemoveWithValues • weka.filters.unsupervised.instance.Resample
Deriving Knowledge from Data at Scale 5 Minute Break…
Deriving Knowledge from Data at Scale Simple Definition feature selection problem Feature extraction 11 . { ,..., ,..., } { ,..., ,..., }j mi n i i if selection f f f f f f F F‘ F F‘ 1 1 1 1 1. { ,..., ,..., } { ( ,..., ),..., ( ,..., ),..., ( ,..., )}i n n j n m nf extraction f f f g f f g f f g f f
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale 3 types of methods Filter Methods Wrapper Methods Embedded Methods decision trees, random forests
Deriving Knowledge from Data at Scale Most learning methods implicitly do feature selection: • Decision Trees: use info gain or gain ratio to decide what attributes to use as tests. Many features don’t get used. • neural nets: backprop learns strong connections to some inputs, and near- zero connections to other inputs. • kNN, MBL (any similarity based learning): weights in Weighted Euclidean Distance determine how important each feature is. Weights near zero mean feature is not used. • SVMs: maximum margin hyperplane may focus on important features, ignore irrelevant features. So why do we need feature selection? Data Integration
Deriving Knowledge from Data at Scale Curse of Dimensionality exponentially In many cases the information lost by discarding variables is made up for by a more accurate mapping/sampling in the lower-dimensional space !
Deriving Knowledge from Data at Scale Feature Selection and Engineering Optimality? This deserves a deeper treatment, which we will cover next week with hands-on exercises in class…
Deriving Knowledge from Data at Scale Numerical data • Binning – a mapping to discrete categories; • Recenter – shift by c where max, min, avg and median shift, the range and standard deviation will not shift; • Rescale – multiply everything by d, all measures change; • Standard ND – recenter, make mean 0, divide all previous values by SD Character data • Lower case • Spellcheck • Data extraction (e.g. regular expressions) Coding – shape and enrich…
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale feature red blue green red red green blue red blue green 1 0 0 0 1 0 0 0 1 1 0 0 1 0 0 0 0 1 0 1 0
Deriving Knowledge from Data at Scale Outlook T emperature Humidity Windy Play sunny 85 85 false no sunny 80 90 true no overcast 83 78 false yes rain 70 96 false yes rain 68 80 false yes rain 65 70 true no overcast 64 65 true yes sunny 72 95 false no sunny 69 70 false yes rain 75 80 false yes sunny 75 70 true yes overcast 72 90 true yes overcast 81 75 false yes rain 71 80 true no Attributes: Outlook (overcast, rain, sunny) Temperature real Humidity real Windy (true, false) Play (yes, no) OutLook OutLook OutLook Temp Humidity Windy Windy Play Play overcast rain sunny TRUE FALSE yes no 0 0 1 85 85 0 1 1 0 0 0 1 80 90 1 0 0 1 1 0 0 83 78 0 1 1 0 0 1 0 70 96 0 1 1 0 0 1 0 68 80 0 1 1 0 0 1 0 65 70 1 0 0 1 1 0 0 64 65 1 0 1 0 . . . . . . . . . . . . . . . . . . Standard Spreadsheet Format
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale Household income $10.000 $200.000 very low low average high very high
Deriving Knowledge from Data at Scale Less features, more discrimination ability concept hierarchies
Deriving Knowledge from Data at Scale • Equal-width (distance) partitioning uniform grid • Equal-depth (frequency) partitioning • Class label based partitioning
Deriving Knowledge from Data at Scale into the user- specified
Deriving Knowledge from Data at Scale [64,67) [67,70) [70,73) [73,76) [76,79) [79,82) [82,85] Temperature values: 64 65 68 69 70 71 72 72 75 75 80 81 83 85 2 2 Count 4 2 2 20
Deriving Knowledge from Data at Scale [0 – 200,000) … …. 1 Count Salary in a corporation [1,800,000 – 2,000,000]
Deriving Knowledge from Data at Scale user-specified nFi number of intervals
Deriving Knowledge from Data at Scale [64 .. .. .. .. 69] [70 .. 72] [73 .. .. .. .. .. .. .. .. 81] [83 .. 85] Temperature values: 64 65 68 69 70 71 72 72 75 75 80 81 83 85 4 Count 4 4 2
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 119
Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 120
Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 121
Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 122
Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 123
Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 124
Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 125
Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 126
Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 127
Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 128
Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 129
Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 130
Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 131
Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 132
Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 133
Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 134
Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 135
Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 136
Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 137
Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 138
Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 139
Deriving Knowledge from Data at Scale
Deriving Knowledge from Data at Scale Domain expertise, play a hunch in terms of feature discrimination
Deriving Knowledge from Data at Scale That’s all for tonight….

More Related Content

PDF
Barga Data Science lecture 5
Roger Barga
 
PDF
Barga Data Science lecture 3
Roger Barga
 
PDF
Barga Data Science lecture 2
Roger Barga
 
PDF
Barga Galvanize Sept 2015
Roger Barga
 
PDF
Barga Data Science lecture 8
Roger Barga
 
PDF
Barga Data Science lecture 9
Roger Barga
 
PDF
Barga Data Science lecture 4
Roger Barga
 
PDF
Barga Data Science lecture 10
Roger Barga
 
Barga Data Science lecture 5
Roger Barga
 
Barga Data Science lecture 3
Roger Barga
 
Barga Data Science lecture 2
Roger Barga
 
Barga Galvanize Sept 2015
Roger Barga
 
Barga Data Science lecture 8
Roger Barga
 
Barga Data Science lecture 9
Roger Barga
 
Barga Data Science lecture 4
Roger Barga
 
Barga Data Science lecture 10
Roger Barga
 

What's hot (20)

PDF
Barga Data Science lecture 1
Roger Barga
 
PDF
Data Driven Engineering 2014
Roger Barga
 
PDF
Barga ACM DEBS 2013 Keynote
Roger Barga
 
PDF
Barga DIDC'14 Invited Talk
Roger Barga
 
PDF
Barga Data Science lecture 7
Roger Barga
 
PDF
Managing machine learning
David Murgatroyd
 
PDF
Introduction to machine learning and deep learning
Shishir Choudhary
 
PDF
Heuristic design of experiments w meta gradient search
Greg Makowski
 
PPTX
H2O World - Top 10 Data Science Pitfalls - Mark Landry
Sri Ambati
 
PDF
Machine learning for_finance
Stefan Duprey
 
PDF
Introduction to Data Science
Niko Vuokko
 
PPTX
Data Science Training | Data Science For Beginners | Data Science With Python...
Simplilearn
 
PPTX
Machine Learning and Real-World Applications
MachinePulse
 
PDF
From Raw Data to Deployed Product. Fast & Agile with CRISP-DM
Michał Łopuszyński
 
PPTX
End-to-End Machine Learning Project
Eng Teong Cheah
 
PDF
CRISP-DM - Agile Approach To Data Mining Projects
Michał Łopuszyński
 
PPTX
Random Forest Tutorial | Random Forest in R | Machine Learning | Data Science...
Edureka!
 
PDF
Exploring the Data science Process
Vishal Patel
 
PPTX
A Practical-ish Introduction to Data Science
Mark West
 
PDF
The path to be a data scientist
Poo Kuan Hoong
 
Barga Data Science lecture 1
Roger Barga
 
Data Driven Engineering 2014
Roger Barga
 
Barga ACM DEBS 2013 Keynote
Roger Barga
 
Barga DIDC'14 Invited Talk
Roger Barga
 
Barga Data Science lecture 7
Roger Barga
 
Managing machine learning
David Murgatroyd
 
Introduction to machine learning and deep learning
Shishir Choudhary
 
Heuristic design of experiments w meta gradient search
Greg Makowski
 
H2O World - Top 10 Data Science Pitfalls - Mark Landry
Sri Ambati
 
Machine learning for_finance
Stefan Duprey
 
Introduction to Data Science
Niko Vuokko
 
Data Science Training | Data Science For Beginners | Data Science With Python...
Simplilearn
 
Machine Learning and Real-World Applications
MachinePulse
 
From Raw Data to Deployed Product. Fast & Agile with CRISP-DM
Michał Łopuszyński
 
End-to-End Machine Learning Project
Eng Teong Cheah
 
CRISP-DM - Agile Approach To Data Mining Projects
Michał Łopuszyński
 
Random Forest Tutorial | Random Forest in R | Machine Learning | Data Science...
Edureka!
 
Exploring the Data science Process
Vishal Patel
 
A Practical-ish Introduction to Data Science
Mark West
 
The path to be a data scientist
Poo Kuan Hoong
 
Ad

Viewers also liked (19)

DOC
Kajul verma-Product Implementation Engineer_4 years
KAJUL VERMA
 
PPT
Toxoplasmosis
degatitos
 
PDF
Giacomo
Giacomo Fiumara
 
PPT
Rfs Overhead Presentation[1]
guest6e5f8
 
PDF
21. Kecamatan kerajaan
kabupaten_pakpakbharat
 
PPT
Participantes Concurso Miradas DeGatitos 2013 Homenaje al Día del Animal
degatitos
 
PDF
Slides jose falck zepeda nas study economics december 2016 original submtted
Jose Falck Zepeda
 
PPTX
Rescate y adopcion canina
tatycagiraldo
 
PDF
Francesco Micali : Big data per le smart cities - Mediabeta srl / Lo Stretto ...
f.micali
 
PPTX
ENBE infographic Pop-out Poster
Darryl Harvey
 
PPTX
Neev QA Offering
Neev Technologies
 
PPTX
Presentación sustento teórico
Ada Luz Olivares
 
PPTX
Отчет о работе Школы кадрового резерва, 2000-2015
nizhgma.ru
 
PDF
Francesco Micali : Formazione aziendale social media - Mediabeta srl
f.micali
 
PPT
Single instruction multiple data
Syed Zaid Irshad
 
PPTX
Big Data Platform adopting Spark and Use Cases with Open Data
Jongwook Woo
 
PPT
Distributed system
Syed Zaid Irshad
 
PPT
V slide budidaya sayuran
hjfgjghjghj
 
PPTX
Hasil Observasi Tanaman Kelapa (Bhs. Indonesia)
Andri_Ferdians
 
Kajul verma-Product Implementation Engineer_4 years
KAJUL VERMA
 
Toxoplasmosis
degatitos
 
Giacomo
Giacomo Fiumara
 
Rfs Overhead Presentation[1]
guest6e5f8
 
21. Kecamatan kerajaan
kabupaten_pakpakbharat
 
Participantes Concurso Miradas DeGatitos 2013 Homenaje al Día del Animal
degatitos
 
Slides jose falck zepeda nas study economics december 2016 original submtted
Jose Falck Zepeda
 
Rescate y adopcion canina
tatycagiraldo
 
Francesco Micali : Big data per le smart cities - Mediabeta srl / Lo Stretto ...
f.micali
 
ENBE infographic Pop-out Poster
Darryl Harvey
 
Neev QA Offering
Neev Technologies
 
Presentación sustento teórico
Ada Luz Olivares
 
Отчет о работе Школы кадрового резерва, 2000-2015
nizhgma.ru
 
Francesco Micali : Formazione aziendale social media - Mediabeta srl
f.micali
 
Single instruction multiple data
Syed Zaid Irshad
 
Big Data Platform adopting Spark and Use Cases with Open Data
Jongwook Woo
 
Distributed system
Syed Zaid Irshad
 
V slide budidaya sayuran
hjfgjghjghj
 
Hasil Observasi Tanaman Kelapa (Bhs. Indonesia)
Andri_Ferdians
 
Ad

Similar to Barga Data Science lecture 6 (20)

PPTX
Classification & Clustering.pptx
ImXaib
 
PPTX
Data preparation and processing chapter 2
Mahmoud Alfarra
 
PDF
Anomaly detection Workshop slides
QuantUniversity
 
PPTX
lec01-IntroductionToDataMining.pptx
AmjadAlDgour
 
PPTX
Machine Learning 2 deep Learning: An Intro
Si Krishan
 
PPT
Data mining applications
Dr. C.V. Suresh Babu
 
PPTX
What Is Data Science? Data Science Course - Data Science Tutorial For Beginne...
Edureka!
 
PDF
Module-1-IntroductionToDataMining (Data Mining)
kalpanagangwar3
 
PDF
Data Science for Fundraising: Build Data-Driven Solutions Using R - Rodger De...
Rodger Devine
 
PDF
Post Graduate Admission Prediction System
IRJET Journal
 
PPTX
Graphs in the Real World
Neo4j
 
PPTX
Decoding Loan Approval: Predictive Modeling in Action
Boston Institute of Analytics
 
PPT
MIS637_Final_Project_Rahul_Bhatia
Rahul Bhatia
 
PPTX
Machine Learning Algorithms | Machine Learning Tutorial | Data Science Algori...
Simplilearn
 
PDF
Classification and Clustering Analysis using Weka
Ishan Awadhesh
 
PPTX
Capstone project- Airbnb (using Python)
Sophie (C.F.) Tsai
 
PPTX
Best Selenium certification course
KumarNaik21
 
PPTX
Predictive Model and Record Description with Segmented Sensitivity Analysis (...
Greg Makowski
 
PDF
Deep-Dive-into.cloud and data presentation
MohanArumugam24
 
PDF
LOAN APPROVAL PRDICTION SYSTEM USING MACHINE LEARNING.
Souma Maiti
 
Classification & Clustering.pptx
ImXaib
 
Data preparation and processing chapter 2
Mahmoud Alfarra
 
Anomaly detection Workshop slides
QuantUniversity
 
lec01-IntroductionToDataMining.pptx
AmjadAlDgour
 
Machine Learning 2 deep Learning: An Intro
Si Krishan
 
Data mining applications
Dr. C.V. Suresh Babu
 
What Is Data Science? Data Science Course - Data Science Tutorial For Beginne...
Edureka!
 
Module-1-IntroductionToDataMining (Data Mining)
kalpanagangwar3
 
Data Science for Fundraising: Build Data-Driven Solutions Using R - Rodger De...
Rodger Devine
 
Post Graduate Admission Prediction System
IRJET Journal
 
Graphs in the Real World
Neo4j
 
Decoding Loan Approval: Predictive Modeling in Action
Boston Institute of Analytics
 
MIS637_Final_Project_Rahul_Bhatia
Rahul Bhatia
 
Machine Learning Algorithms | Machine Learning Tutorial | Data Science Algori...
Simplilearn
 
Classification and Clustering Analysis using Weka
Ishan Awadhesh
 
Capstone project- Airbnb (using Python)
Sophie (C.F.) Tsai
 
Best Selenium certification course
KumarNaik21
 
Predictive Model and Record Description with Segmented Sensitivity Analysis (...
Greg Makowski
 
Deep-Dive-into.cloud and data presentation
MohanArumugam24
 
LOAN APPROVAL PRDICTION SYSTEM USING MACHINE LEARNING.
Souma Maiti
 

Recently uploaded (20)

PPT
Reliability_Chapter_ presentation 1221.5784
abobaker13
 
PPTX
Supervised vs unsupervised machine learning algorithms
agarwal18harsh08
 
PDF
Lecture1 pattern recognition............
ZafriFarid
 
PPTX
Microsoft-Fabric-Unifying-Analytics-for-the-Modern-Enterprise Solution.pptx
Mahesh Reddy
 
PDF
BF and FI - Blockchain, fintech and Financial Innovation Lesson 2.pdf
JamesWilliams752225
 
PDF
TRAFFIC-MANAGEMENT-AND-ACCIDENT-INVESTIGATION-WITH-DRIVING-PDF-FILE.pdf
chouziyu14
 
PPTX
Data_Analytics_and_PowerBI_Presentation.pptx
ZubyrAhmed
 
PPTX
Acceptance and paychological effects of mandatory extra coach I classes.pptx
ZaheerAhmad228692
 
PDF
Galatica Smart Energy Infrastructure Startup Pitch Deck
Shahzaib Ajmal
 
PDF
"Python Programming for Geospatial Data Science." ...
institute of Geoinformatics and Earth Observation at PMAS ARID Agriculture University of Rawalpindi
 
PPTX
1_Introduction to advance data techniques.pptx
AshutoshDeshmukh33
 
PDF
Fluorescence-microscope_Botany_detailed content
dollydoll12
 
PPTX
STUDY DESIGN details- Lt Col Maksud (21).pptx
AyeshaAsha11
 
PDF
“Getting Started with Data Analytics Using R – Concepts, Tools & Case Studies”
Nusrat Gulbarga
 
PPTX
Business Ppt On Nestle.pptx huunnnhhgfvu
ashaurya327
 
PDF
.pdf is not working space design for the following data for the following dat...
jerinjoy242
 
PPTX
Introduction to machine learning and Linear Models
ShivanshMehta17
 
PPTX
mbdjdhjjodule 5-1 rhfhhfjtjjhafbrhfnfbbfnb
CHINTU5000
 
PDF
Recruitment and Placement PPT.pdfbjfibjdfbjfobj
Alka392097
 
PPTX
Introduction-to-Cloud-ComputingFinal.pptx
testnet29393883
 
Reliability_Chapter_ presentation 1221.5784
abobaker13
 
Supervised vs unsupervised machine learning algorithms
agarwal18harsh08
 
Lecture1 pattern recognition............
ZafriFarid
 
Microsoft-Fabric-Unifying-Analytics-for-the-Modern-Enterprise Solution.pptx
Mahesh Reddy
 
BF and FI - Blockchain, fintech and Financial Innovation Lesson 2.pdf
JamesWilliams752225
 
TRAFFIC-MANAGEMENT-AND-ACCIDENT-INVESTIGATION-WITH-DRIVING-PDF-FILE.pdf
chouziyu14
 
Data_Analytics_and_PowerBI_Presentation.pptx
ZubyrAhmed
 
Acceptance and paychological effects of mandatory extra coach I classes.pptx
ZaheerAhmad228692
 
Galatica Smart Energy Infrastructure Startup Pitch Deck
Shahzaib Ajmal
 
"Python Programming for Geospatial Data Science." ...
institute of Geoinformatics and Earth Observation at PMAS ARID Agriculture University of Rawalpindi
 
1_Introduction to advance data techniques.pptx
AshutoshDeshmukh33
 
Fluorescence-microscope_Botany_detailed content
dollydoll12
 
STUDY DESIGN details- Lt Col Maksud (21).pptx
AyeshaAsha11
 
“Getting Started with Data Analytics Using R – Concepts, Tools & Case Studies”
Nusrat Gulbarga
 
Business Ppt On Nestle.pptx huunnnhhgfvu
ashaurya327
 
.pdf is not working space design for the following data for the following dat...
jerinjoy242
 
Introduction to machine learning and Linear Models
ShivanshMehta17
 
mbdjdhjjodule 5-1 rhfhhfjtjjhafbrhfnfbbfnb
CHINTU5000
 
Recruitment and Placement PPT.pdfbjfibjdfbjfobj
Alka392097
 
Introduction-to-Cloud-ComputingFinal.pptx
testnet29393883
 

Barga Data Science lecture 6

  • 1. Deriving Knowledge from Data at Scale
  • 2. Deriving Knowledge from Data at Scale Feature extraction and selection are the most important but underrated step of machine learning. Better features are better than better algorithms…
  • 3. Deriving Knowledge from Data at Scale
  • 4. Deriving Knowledge from Data at Scale
  • 5. Deriving Knowledge from Data at Scale Lecture Objectives homework There is an order or workflow that takes place here, don’t lose the forest in the trees…
  • 6. Deriving Knowledge from Data at Scale Review…
  • 7. Deriving Knowledge from Data at Scale • Cluster 0 – It contains a cluster of Females with an average age of 37 who live in inner city and possess saving account number and current account number. They are unmarried and do not have any mortgage or pep. The average monthly income is 23,300. • Cluster 1 - It contains a cluster of Females with an average age of 44 who live in rural area and possess saving account number and current account number. They are married and do not have any mortgage or pep. The average monthly income is 27,772. • Cluster 2 - It contains a cluster of Females with an average age of 48 who live in inner city and possess current account number but no saving account number. They are unmarried and do not have mortgage but do have pep. The average monthly income is 27,668. • Cluster 3 - It contains a cluster of Females with an average age of 39 who live in town and possess saving account number and current account number. They are married and do not have any mortgage or pep. The average monthly income is 24,047. • Cluster 4 - It contains a cluster of Males with an average age of 39 who live in inner city and possess current account number but no saving account number. They are married and have mortgage and pep. The average monthly income is 26,359. • Cluster 5 - It contains a cluster of Males with an average age of 47 who live in inner city and possess saving account number and current account number. They are unmarried and do not have mortgage but do have pep. The average monthly income is 35,419.
  • 8. Deriving Knowledge from Data at Scale
  • 9. Deriving Knowledge from Data at Scale Classifiers  Lazy –> IBk
  • 10. Deriving Knowledge from Data at Scale
  • 11. Deriving Knowledge from Data at Scale
  • 12. Deriving Knowledge from Data at Scale
  • 13. Deriving Knowledge from Data at Scale
  • 14. Deriving Knowledge from Data at Scale
  • 15. Deriving Knowledge from Data at Scale15
  • 16. Deriving Knowledge from Data at Scale No Prob Target CustID Age 1 0.97 Y 1746 … 2 0.95 N 1024 … 3 0.94 Y 2478 … 4 0.93 Y 3820 … 5 0.92 N 4897 … … … … … 99 0.11 N 2734 … 100 0.06 N 2422 Use a model to assign score (probability) to each instance Sort instances by decreasing score Expect more targets (hits) near the top of the list 3 hits in top 5% of the list If there 15 targets overall, then top 5 has 3/15=20% of targets
  • 17. Deriving Knowledge from Data at Scale 40% of responses for 10% of cost Lift factor = 4 80% of responses for 40% of cost Lift factor = 2 Model Random
  • 18. Deriving Knowledge from Data at Scale
  • 19. Deriving Knowledge from Data at Scale
  • 20. Deriving Knowledge from Data at Scale
  • 21. Deriving Knowledge from Data at Scale
  • 22. Deriving Knowledge from Data at Scale to impact… 1. Build our predictive model in WEKA Explorer; 2. Use our model to score (predict) which new customers to target in our upcoming advertising campaign; • ARFF file manipulation (hacking), all too common pita… • Excel manipulation to join model output with our customers list 3. Compute the lift chart to assess business impact of our predictive model on the advertising campaign • How are Lift charts built, of all the charts and/or performance measures from a model this one is ‘on you’ to construct; • Where is the business ‘bang for the buck’?
  • 23. Deriving Knowledge from Data at Scale
  • 24. Deriving Knowledge from Data at Scale
  • 25. Deriving Knowledge from Data at Scale
  • 26. Deriving Knowledge from Data at Scale You can’t turn data lead into modeling gold – we’re data scientists, not data alchemists…
  • 27. Deriving Knowledge from Data at Scale Motivation: Real world examples Example (1) Lesson: Correct data transformation is important!
  • 28. Deriving Knowledge from Data at Scale Motivation: Real world examples Example (2): KDD Cup 2001 Lesson: A model that uses lots of features can turn out to be very sub-optimal, however well it is designed!
  • 29. Deriving Knowledge from Data at Scale Motivation: Real world examples Example (3) Lesson: Feature selection can be crucial even when the number of features is small!
  • 30. Deriving Knowledge from Data at Scale Motivation: Real world examples Example (4) Lesson: Variations of the same ML method can give vastly different performances!
  • 31. Deriving Knowledge from Data at Scale
  • 32. Deriving Knowledge from Data at Scale Predictive modeling competitions
  • 33. Deriving Knowledge from Data at Scale Global competitions 1½ weeks 70.8% Competition closes 77% State of the art 70% Predicting HIV viral load Improved by 10%
  • 34. Deriving Knowledge from Data at Scale Mismatch between those with data and those with the skills to analyse it Crowdsourcing
  • 35. Deriving Knowledge from Data at Scale Forecast Error (MASE) Existing model Tourism Forecasting Competition Aug 9 2 weeks later 1 month later Competition End
  • 36. Deriving Knowledge from Data at Scale • neural networks • logistic regression • support vector machine • decision trees • ensemble methods • adaBoost • Bayesian networks • genetic algorithms • random forest • Monte Carlo methods • principal component analysis • Kalman filter • evolutionary fuzzy modeling Users apply different techniques
  • 37. Deriving Knowledge from Data at Scale VicRoads has an algorithm they use to forecast travel time on Melbourne freeways (taking into account time, weather, accidents, etc). Their current model is inaccurate and somewhat useless. They want to do better (or at least find out about whether it’s possible to do better).
  • 38. Deriving Knowledge from Data at Scale 1 2 3 Upload Submit Evaluate & Exchange
  • 39. Deriving Knowledge from Data at Scale Use the wizard to post a competition
  • 40. Deriving Knowledge from Data at Scale Participants make their entries
  • 41. Deriving Knowledge from Data at Scale Competitions are judged based on predictive accuracy
  • 42. Deriving Knowledge from Data at Scale Competition Mechanics Competitions are judged on objective criteria
  • 43. Deriving Knowledge from Data at Scale Kaggle How They Won It…
  • 44. Deriving Knowledge from Data at Scale
  • 45. Deriving Knowledge from Data at Scale
  • 46. Deriving Knowledge from Data at Scale Three Files ford_train • 510 Trials, ~1,200 observations each spaced by 0.1 sec -> 604,330 rows ford_test • 100 Trials,~1,200 observations/trial, 120,841 rows example_submission.csv
  • 47. Deriving Knowledge from Data at Scale Junpei Komiyama (#4)
  • 48. Deriving Knowledge from Data at Scale Junpei Komiyama (#4)
  • 49. Deriving Knowledge from Data at Scale Mick Wagner (#2)
  • 50. Deriving Knowledge from Data at Scale Mick Wagner (#2)
  • 51. Deriving Knowledge from Data at Scale Inference (#1)
  • 52. Deriving Knowledge from Data at Scale VicRoads has an algorithm they use to forecast travel time on Melbourne freeways (taking into account time, weather, accidents etc). Their current model is inaccurate and somewhat useless. They want to do better (or at least find out about whether it’s possible to do better).
  • 53. Deriving Knowledge from Data at Scale
  • 54. Deriving Knowledge from Data at Scale François GUILLEM (#14)
  • 55. Deriving Knowledge from Data at Scale #1 used Random Forests
  • 56. Deriving Knowledge from Data at Scale
  • 57. Deriving Knowledge from Data at Scale Homework Week 6 Monday Sept. 21st Upload to site… http://blog.kaggle.com/category/dojo/ Content is 10 pages of interview on how the team(s) built their models, some have multiple interviews; You will review at least 10 interviews, bounce around do not go sequentially. 1) What model(s) did they use, 2) insights they had that influenced modeling, 3) what feature creation and selection, 4) other observations. I will cons all these together and upload as shared document on our site.
  • 58. Deriving Knowledge from Data at Scale 5 Minute Break…
  • 59. Deriving Knowledge from Data at Scale Course Project
  • 60. Deriving Knowledge from Data at Scale
  • 61. Deriving Knowledge from Data at Scale https://www.kaggle.com/c/springleaf-marketing-response not Determine whether to send a direct mail piece to a customer
  • 62. Deriving Knowledge from Data at Scale The Data
  • 63. Deriving Knowledge from Data at Scale The Rules
  • 64. Deriving Knowledge from Data at Scale
  • 65. Deriving Knowledge from Data at Scale
  • 66. Deriving Knowledge from Data at Scale
  • 67. Deriving Knowledge from Data at Scale what is the data telling you
  • 68. Deriving Knowledge from Data at Scale
  • 69. Deriving Knowledge from Data at Scale
  • 70. Deriving Knowledge from Data at Scale Data Wrangling
  • 71. Deriving Knowledge from Data at Scale Data Acquisition Data Exploration Pre- processing Feature and Target construction Train/ Test split Feature selection Model training Model scoring Model scoring Evaluation Evaluation Compare metrics
  • 72. Deriving Knowledge from Data at Scale • Data preparation step is by far the most time consuming step 0 10 20 30 40 50 60 70 Understanding of Domain Understanding of Data Preparation of Data Data Mining Evaluation of Results Deployment of Results KDDM steps relative effort [%] Cabena et al. estimates Shearer estimates Cios and Kurgan estimates
  • 73. Deriving Knowledge from Data at Scale Out of Class Reading, highly recommended
  • 74. Deriving Knowledge from Data at Scale Out of Class Reading, highly recommended
  • 75. Deriving Knowledge from Data at Scale 1. Do you have domain knowledge? 2. Are your features commensurate? 3. Do you suspect interdependence of features? 4. Do you need to prune the input variables 5. Do you need to assess features individually 6. Do you need a predictor? 7. Do you suspect your data is “dirty” 8. Do you know what to try first? 9. Do you have new ideas, time, computational resources, and enough examples? 10. Do you want a stable solution
  • 76. Deriving Knowledge from Data at Scale
  • 77. Deriving Knowledge from Data at Scale
  • 78. Deriving Knowledge from Data at Scale
  • 79. Deriving Knowledge from Data at Scale 15 15 𝑃 = 0.5 𝑃 = 0.5
  • 80. Deriving Knowledge from Data at Scale 15 157 13 𝑃 = 0.5 𝑃 = 0.5 𝑃 = 0.35 𝑃 = 0.65
  • 81. Deriving Knowledge from Data at Scale 15 1515 15 𝑃 = 0.5 𝑃 = 0.510 10
  • 82. Deriving Knowledge from Data at Scale 15 1515 15 𝑃 = 0.5 𝑃 = 0.5 Time T r a i n T e s t Horizontal Vertical
  • 83. Deriving Knowledge from Data at Scale Data Characterization…
  • 84. Deriving Knowledge from Data at Scale 1. Unique values 2. Most frequent values 3. Highest and lowest values 4. Location and dispersion – gini, statistical test for dispersion 5. Quartiles
  • 85. Deriving Knowledge from Data at Scale 1. Missing values 2. Outliers 3. Coding 4. Constraints
  • 86. Deriving Knowledge from Data at Scale Missing values – UCI machine learning repository, 31 of 68 data sets reported to have missing values. “Missing” can mean many things… MAR: "Missing at Random": – usually best case – usually not true Non-randomly missing Presumed normal, so not measured Causally missing – attribute value is missing because of other attribute values (or because of the outcome value!)
  • 87. Deriving Knowledge from Data at Scale
  • 88. Deriving Knowledge from Data at Scale
  • 89. Deriving Knowledge from Data at Scale
  • 90. Deriving Knowledge from Data at Scale Outliers – may indicate ‘bad data’ or it may represent something scientifically interesting in the data… Simple working definition: an outlier is an element of a data sequence S that is inconsistent with expectations, based on the majority of other elements of S. Sources of outliers • Measurement errors • Other uninteresting anomalous data • Surprising observations that may be important
  • 91. Deriving Knowledge from Data at Scale Outliers – may indicate ‘bad data’ or it may represent something scientifically interesting in the data… Simple working definition: an outlier is an element of a data sequence S that is inconsistent with expectations, based on the majority of other elements of S. Sources of outliers • Insurance company sees niche of sports car enthusiasts, married boomers with kids and second family car. Low risk, lower rate to attract. Simple case where outlier carries meaning for modeling…
  • 92. Deriving Knowledge from Data at Scale Outliers can distort the regression results. When an outlier is included in the analysis, it pulls the regression line towards itself. This can result in a solution that is more accurate for the outlier, but less accurate for all the other cases in the data set. Outliers – may indicate ‘bad data’ or it may represent something scientifically interesting in the data…
  • 93. Deriving Knowledge from Data at Scale Identify outliers • Question origin, domain knowledge invaluable • Dispersion – "spread" of a data set, departure from central tendency, use a box plot… Deal with outliers • Winsorize – Set all outliers to a specified percentile of the data. Not equivalent to trimming, which simply excludes data. In a Winsorized estimator, extreme values are instead replaced by certain percentiles (the trimmed minimum and maximum). Same as clipping in signal processing. Outliers – may indicate ‘bad data’ or it may represent something scientifically interesting in the data…
  • 94. Deriving Knowledge from Data at Scale Identify outliers • Question origin, domain knowledge invaluable • Dispersion – "spread" of a data set, departure from central tendency, use a box plot… Deal with outliers • Include – Robust statistics, a convenient way to summarize results when they include a small proportion of outliers. A hot topic for research, see NIPS 2010 Workshop, Robust Statistical learning (robustml). Outliers – may indicate ‘bad data’ or it may represent something scientifically interesting in the data…
  • 95. Deriving Knowledge from Data at Scale • Entity integrity • Referential integrity • Type checking • Format • Bounds checking Constraints
  • 96. Deriving Knowledge from Data at Scale • weka.filters.unsupervised.instance.RemoveMisclassified • weka.filters.unsupervised.instance.RemovePercentage • weka.filters.unsupervised.instance.RemoveRange • weka.filters.unsupervised.instance.RemoveWithValues • weka.filters.unsupervised.instance.Resample
  • 97. Deriving Knowledge from Data at Scale 5 Minute Break…
  • 98. Deriving Knowledge from Data at Scale Simple Definition feature selection problem Feature extraction 11 . { ,..., ,..., } { ,..., ,..., }j mi n i i if selection f f f f f f F F‘ F F‘ 1 1 1 1 1. { ,..., ,..., } { ( ,..., ),..., ( ,..., ),..., ( ,..., )}i n n j n m nf extraction f f f g f f g f f g f f
  • 99. Deriving Knowledge from Data at Scale
  • 100. Deriving Knowledge from Data at Scale 3 types of methods Filter Methods Wrapper Methods Embedded Methods decision trees, random forests
  • 101. Deriving Knowledge from Data at Scale Most learning methods implicitly do feature selection: • Decision Trees: use info gain or gain ratio to decide what attributes to use as tests. Many features don’t get used. • neural nets: backprop learns strong connections to some inputs, and near- zero connections to other inputs. • kNN, MBL (any similarity based learning): weights in Weighted Euclidean Distance determine how important each feature is. Weights near zero mean feature is not used. • SVMs: maximum margin hyperplane may focus on important features, ignore irrelevant features. So why do we need feature selection? Data Integration
  • 102. Deriving Knowledge from Data at Scale Curse of Dimensionality exponentially In many cases the information lost by discarding variables is made up for by a more accurate mapping/sampling in the lower-dimensional space !
  • 103. Deriving Knowledge from Data at Scale Feature Selection and Engineering Optimality? This deserves a deeper treatment, which we will cover next week with hands-on exercises in class…
  • 104. Deriving Knowledge from Data at Scale Numerical data • Binning – a mapping to discrete categories; • Recenter – shift by c where max, min, avg and median shift, the range and standard deviation will not shift; • Rescale – multiply everything by d, all measures change; • Standard ND – recenter, make mean 0, divide all previous values by SD Character data • Lower case • Spellcheck • Data extraction (e.g. regular expressions) Coding – shape and enrich…
  • 105. Deriving Knowledge from Data at Scale
  • 106. Deriving Knowledge from Data at Scale feature red blue green red red green blue red blue green 1 0 0 0 1 0 0 0 1 1 0 0 1 0 0 0 0 1 0 1 0
  • 107. Deriving Knowledge from Data at Scale Outlook T emperature Humidity Windy Play sunny 85 85 false no sunny 80 90 true no overcast 83 78 false yes rain 70 96 false yes rain 68 80 false yes rain 65 70 true no overcast 64 65 true yes sunny 72 95 false no sunny 69 70 false yes rain 75 80 false yes sunny 75 70 true yes overcast 72 90 true yes overcast 81 75 false yes rain 71 80 true no Attributes: Outlook (overcast, rain, sunny) Temperature real Humidity real Windy (true, false) Play (yes, no) OutLook OutLook OutLook Temp Humidity Windy Windy Play Play overcast rain sunny TRUE FALSE yes no 0 0 1 85 85 0 1 1 0 0 0 1 80 90 1 0 0 1 1 0 0 83 78 0 1 1 0 0 1 0 70 96 0 1 1 0 0 1 0 68 80 0 1 1 0 0 1 0 65 70 1 0 0 1 1 0 0 64 65 1 0 1 0 . . . . . . . . . . . . . . . . . . Standard Spreadsheet Format
  • 108. Deriving Knowledge from Data at Scale
  • 109. Deriving Knowledge from Data at Scale
  • 110. Deriving Knowledge from Data at Scale Household income $10.000 $200.000 very low low average high very high
  • 111. Deriving Knowledge from Data at Scale Less features, more discrimination ability concept hierarchies
  • 112. Deriving Knowledge from Data at Scale • Equal-width (distance) partitioning uniform grid • Equal-depth (frequency) partitioning • Class label based partitioning
  • 113. Deriving Knowledge from Data at Scale into the user- specified
  • 114. Deriving Knowledge from Data at Scale [64,67) [67,70) [70,73) [73,76) [76,79) [79,82) [82,85] Temperature values: 64 65 68 69 70 71 72 72 75 75 80 81 83 85 2 2 Count 4 2 2 20
  • 115. Deriving Knowledge from Data at Scale [0 – 200,000) … …. 1 Count Salary in a corporation [1,800,000 – 2,000,000]
  • 116. Deriving Knowledge from Data at Scale user-specified nFi number of intervals
  • 117. Deriving Knowledge from Data at Scale [64 .. .. .. .. 69] [70 .. 72] [73 .. .. .. .. .. .. .. .. 81] [83 .. 85] Temperature values: 64 65 68 69 70 71 72 72 75 75 80 81 83 85 4 Count 4 4 2
  • 118. Deriving Knowledge from Data at Scale
  • 119. Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 119
  • 120. Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 120
  • 121. Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 121
  • 122. Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 122
  • 123. Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 123
  • 124. Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 124
  • 125. Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 125
  • 126. Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 126
  • 127. Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 127
  • 128. Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 128
  • 129. Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 129
  • 130. Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 130
  • 131. Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 131
  • 132. Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 132
  • 133. Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 133
  • 134. Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 134
  • 135. Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 135
  • 136. Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 136
  • 137. Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 137
  • 138. Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 138
  • 139. Deriving Knowledge from Data at Scale 4/12/2016 University of Waikato 139
  • 140. Deriving Knowledge from Data at Scale
  • 141. Deriving Knowledge from Data at Scale Domain expertise, play a hunch in terms of feature discrimination
  • 142. Deriving Knowledge from Data at Scale That’s all for tonight….