SlideShare a Scribd company logo

An Introduction to Supervised Machine Learning and Pattern Classification: The Big Picture

Sebastian Raschka, profile picture
Sebastian Raschka

The document provides an introduction to supervised machine learning and pattern classification. It begins with an overview of the speaker's background and research interests. Key concepts covered include definitions of machine learning, examples of machine learning applications, and the differences between supervised, unsupervised, and reinforcement learning. The rest of the document outlines the typical workflow for a supervised learning problem, including data collection and preprocessing, model training and evaluation, and model selection. Common classification algorithms like decision trees, naive Bayes, and support vector machines are briefly explained. The presentation concludes with discussions around choosing the right algorithm and avoiding overfitting.

Data & Analytics
Related topics:
Data Mining Insights • Data Science Insights • Supervised Learning • Computer Science Basics
1 of 55
Downloaded 2,796 times
1
2
3
4
5
Most read
6
7
8
Most read
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
Most read
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
Practical Data Science An Introduction to Supervised Machine Learning and Pattern Classification: The Big Picture Michigan State University NextGen Bioinformatics Seminars - 2015 Sebastian Raschka Feb. 11, 2015
A Little Bit About Myself ... Developing software & methods for - Protein ligand docking - Large scale drug/inhibitor discovery PhD candidate in Dr. L. Kuhn’s Lab: and some other machine learning side-projects …
What is Machine Learning? http://drewconway.com/zia/2013/3/26/the-data-science-venn-diagram "Field of study that gives computers the ability to learn without being explicitly programmed.” (Arthur Samuel, 1959) By Phillip Taylor [CC BY 2.0]
http://commons.wikimedia.org/wiki/ File:American_book_company_1916._letter_envelope-2.JPG#filelinks [public domain] https://flic.kr/p/5BLW6G [CC BY 2.0] Text Recognition Spam Filtering Biology Examples of Machine Learning
Examples of Machine Learning http://googleresearch.blogspot.com/2014/11/a-picture-is-worth-thousand-coherent.html By Steve Jurvetson [CC BY 2.0] Self-driving cars Photo search and many, many more ... Recommendation systems http://commons.wikimedia.org/wiki/File:Netflix_logo.svg [public domain]
How many of you have used machine learning before?
Our Agenda • Concepts and the big picture • Workflow • Practical tips & good habits
Learning • Labeled data • Direct feedback • Predict outcome/future • Decision process • Reward system • Learn series of actions • No labels • No feedback • “Find hidden structure” Unsupervised Supervised Reinforcement
Unsupervised learning Supervised learning Clustering: [DBSCAN on a toy dataset] Classification: [SVM on 2 classes of the Wine dataset] Regression: [Soccer Fantasy Score prediction] Today’s topic Supervised LearningUnsupervised Learning
Instances (samples, observations) Features (attributes, dimensions) Classes (targets) Nomenclature sepal_length sepal_width petal_length petal_width class 1 5.1 3.5 1.4 0.2 setosa 2 4.9 3.0 1.4 0.2 setosa … … … … … … 50 6.4 3.2 4.5 1.5 veriscolor … … … … … … 150 5.9 3.0 5.1 1.8 virginica https://archive.ics.uci.edu/ml/datasets/Iris IRIS
Classification x1 x2 class1 class2 1) Learn from training data 2) Map unseen (new) data ?
Feature Extraction Feature Selection Dimensionality Reduction Feature Scaling Raw Data Collection Pre-Processing Sampling Test Dataset Training Dataset Learning Algorithm Training Post-Processing Cross Validation Final Classification/ Regression Model New DataPre-Processing Refinement Prediction Split Supervised Learning Sebastian Raschka 2014 Missing Data Performance Metrics Model Selection Hyperparameter Optimization This work is licensed under a Creative Commons Attribution 4.0 International License. Final Model Evaluation
Feature Extraction Feature Selection Dimensionality Reduction Feature Scaling Raw Data Collection Pre-Processing Sampling Test Dataset Training Dataset Learning Algorithm Training Post-Processing Cross Validation Final Classification/ Regression Model New DataPre-Processing Refinement Prediction Split Supervised Learning Sebastian Raschka 2014 Missing Data Performance Metrics Model Selection Hyperparameter Optimization This work is licensed under a Creative Commons Attribution 4.0 International License. Final Model Evaluation
A Few Common Classifiers Decision Tree Perceptron Naive Bayes Ensemble Methods: Random Forest, Bagging, AdaBoost Support Vector Machine K-Nearest Neighbor Logistic Regression Artificial Neural Network / Deep Learning
Discriminative Algorithms Generative Algorithms • Models a more general problem: how the data was generated. • I.e., the distribution of the class; joint probability distribution p(x,y). • Naive Bayes, Bayesian Belief Network classifier, Restricted Boltzmann Machine … • Map x → y directly. • E.g., distinguish between people speaking different languages without learning the languages. • Logistic Regression, SVM, Neural Networks …
Examples of Discriminative Classifiers: Perceptron xi1 xi2 w1 w2 Σ yi y = wTx = w0 + w1x1 + w2x2 1 F. Rosenblatt. The perceptron, a perceiving and recognizing automaton Project Para. Cornell Aeronautical Laboratory, 1957. x1 x2 y ∈ {-1,1} w0 wj = weight xi = training sample yi = desired output yi = actual output t = iteration step η = learning rate θ = threshold (here 0) update rule: wj(t+1) = wj(t) + η(yi - yi)xi 1 if wTxi ≥ θ -1 otherwise ^ ^ ^ ^ yi ^ until t+1 = max iter or error = 0
Discriminative Classifiers: Perceptron F. Rosenblatt. The perceptron, a perceiving and recognizing automaton Project Para. Cornell Aeronautical Laboratory, 1957. - Binary classifier (one vs all, OVA) - Convergence problems (set n iterations) - Modification: stochastic gradient descent - “Modern” perceptron: Support Vector Machine (maximize margin) - Multilayer perceptron (MLP) xi1 xi2 w1 w2 Σ yi 1 y ∈ {-1,1} w0 ^ x1 x2
Generative Classifiers: Naive Bayes Bayes Theorem: P(ωj | xi) = P(xi | ωj) P(ωj) P(xi) Posterior probability = Likelihood x Prior probability Evidence Iris example: P(“Setosa"| xi), xi = [4.5 cm, 7.4 cm]
Generative Classifiers: Naive Bayes Decision Rule: Bayes Theorem: P(ωj | xi) = P(xi | ωj) P(ωj) P(xi) pred. class label ωj argmax P(ωj | xi) i = 1, …, m e.g., j ∈ {Setosa, Versicolor, Virginica}
Class-conditional probability (here Gaussian kernel): Generative Classifiers: Naive Bayes Prior probability: Evidence: (cancels out) (class frequency) P(ωj | xi) = P(xi | ωj) P(ωj) P(xi) P(ωj) = Nωj Nc P(xik |ωj) = 1 √ (2 π σωj 2) exp ( )- (xik - μωj)2 2σωj 2 P(xi |ωj) P(xik |ωj)
Generative Classifiers: Naive Bayes - Naive conditional independence assumption typically violated - Works well for small datasets - Multinomial model still quite popular for text classification (e.g., spam filter)
Non-Parametric Classifiers: K-Nearest Neighbor - Simple! - Lazy learner - Very susceptible to curse of dimensionality k=3 e.g., k=1
Iris Example Setosa Virginica Versicolor k = 3 mahalanobis dist. uniform weights C = 3 depth = 2
Decision Tree Entropy = depth = 4 petal length <= 2.45? petal length <= 4.75? Setosa Virginica Versicolor Yes No e.g., 2 (- 0.5 log2(0.5)) = 1 ∑−pi logk pi i Information Gain = entropy(parent) – [avg entropy(children)] NoYes depth = 2
"No Free Lunch" :( Roughly speaking: “No one model works best for all possible situations.” Our model is a simplification of reality Simplification is based on assumptions (model bias) Assumptions fail in certain situations D. H. Wolpert. The supervised learning no-free-lunch theorems. In Soft Computing and Industry, pages 25–42. Springer, 2002.
Which Algorithm? • What is the size and dimensionality of my training set? • Is the data linearly separable? • How much do I care about computational efficiency? - Model building vs. real-time prediction time - Eager vs. lazy learning / on-line vs. batch learning - prediction performance vs. speed • Do I care about interpretability or should it "just work well?" • ...
Feature Extraction Feature Selection Dimensionality Reduction Feature Scaling Raw Data Collection Pre-Processing Sampling Test Dataset Training Dataset Learning Algorithm Training Post-Processing Cross Validation Final Classification/ Regression Model New DataPre-Processing Refinement Prediction Split Supervised Learning Sebastian Raschka 2014 Missing Data Performance Metrics Model Selection Hyperparameter Optimization This work is licensed under a Creative Commons Attribution 4.0 International License. Final Model Evaluation
Missing Values: - Remove features (columns) - Remove samples (rows) - Imputation (mean, nearest neighbor, …) Sampling: - Random split into training and validation sets - Typically 60/40, 70/30, 80/20 - Don’t use validation set until the very end! (overfitting) Feature Scaling: e.g., standardization: - Faster convergence (gradient descent) - Distances on same scale (k-NN with Euclidean distance) - Mean centering for free - Normal distributed data - Numerical stability by avoiding small weights z = xik - μk σk (use same parameters for the test/new data!)
Categorical Variables color size prize class label 0 green M 10.1 class1 1 red L 13.5 class2 2 blue XL 15.3 class1 ordinalnominal green → (1,0,0) red → (0,1,0) blue → (0,0,1) class label color=blue color=green color=red prize size 0 0 0 1 0 10.1 1 1 1 0 0 1 13.5 2 2 0 1 0 0 15.3 3 M → 1 L → 2 XL → 3
Feature Extraction Feature Selection Dimensionality Reduction Feature Scaling Raw Data Collection Pre-Processing Sampling Test Dataset Training Dataset Learning Algorithm Training Post-Processing Cross Validation Final Classification/ Regression Model New DataPre-Processing Refinement Prediction Split Supervised Learning Sebastian Raschka 2014 Missing Data Performance Metrics Model Selection Hyperparameter Optimization This work is licensed under a Creative Commons Attribution 4.0 International License. Final Model Evaluation
Generalization Error and Overfitting How well does the model perform on unseen data?
Generalization Error and Overfitting
Error Metrics: Confusion Matrix TP [Linear SVM on sepal/petal lengths] TN FN FP here: “setosa” = “positive”
Error Metrics TP [Linear SVM on sepal/petal lengths] TN FN FP here: “setosa” = “positive” TP + TN FP +FN +TP +TN Accuracy = = 1 - Error FP N TP P False Positive Rate = TP TP + FP Precision = True Positive Rate = (Recall) “micro” and “macro” averaging for multi-class
Receiver Operating Characteristic (ROC) Curves
Test set Training dataset Test dataset Complete dataset Test set Test set Test set 1st iteration calc. error calc. error calc. error calc. error calculate avg. error k-fold cross-validation (k=4): 2nd iteration 3rd iteration 4th iteration fold 1 fold 2 fold 3 fold 4 Model Selection
k-fold CV and ROC
Feature Selection - Domain knowledge - Variance threshold - Exhaustive search - Decision trees - … IMPORTANT! (Noise, overfitting, curse of dimensionality, efficiency) X = [x1, x2, x3, x4]start: stop: (if d = k) X = [x1, x3, x4] X = [x1, x3] Simplest example: Greedy Backward Selection
Dimensionality Reduction • Transformation onto a new feature subspace • e.g., Principal Component Analysis (PCA) • Find directions of maximum variance • Retain most of the information
0. Standardize data 1. Compute covariance matrix z = xik - μk σik = ∑ (xij - µj) (xik - µk) σk 1 in -1 σ2 1 σ12 σ13 σ14 σ21 σ2 2 σ23 σ24 σ31 σ32 σ2 3 σ34 σ41 σ42 σ43 σ2 4 ∑ = PCA in 3 Steps
2. Eigendecomposition and sorting eigenvalues PCA in 3 Steps X v = λ v Eigenvectors [[ 0.52237162 -0.37231836 -0.72101681 0.26199559] [-0.26335492 -0.92555649 0.24203288 -0.12413481] [ 0.58125401 -0.02109478 0.14089226 -0.80115427] [ 0.56561105 -0.06541577 0.6338014 0.52354627]] Eigenvalues [ 2.93035378 0.92740362 0.14834223 0.02074601] (from high to low)
3. Select top k eigenvectors and transform data PCA in 3 Steps Eigenvectors [[ 0.52237162 -0.37231836 -0.72101681 0.26199559] [-0.26335492 -0.92555649 0.24203288 -0.12413481] [ 0.58125401 -0.02109478 0.14089226 -0.80115427] [ 0.56561105 -0.06541577 0.6338014 0.52354627]] Eigenvalues [ 2.93035378 0.92740362 0.14834223 0.02074601] [First 2 PCs of Iris]
Hyperparameter Optimization: GridSearch in scikit-learn
Non-Linear Problems - XOR gate k=11 uniform weights C=1 C=1000, gamma=0.1 depth=4
Kernel Trick Kernel function Kernel Map onto high-dimensional space (non-linear combinations)
Kernel Trick Trick: No explicit dot product! Radius Basis Function (RBF) Kernel:
Kernel PCA PC1, linear PCA PC1, kernel PCA
Feature Extraction Feature Selection Dimensionality Reduction Feature Scaling Raw Data Collection Pre-Processing Sampling Test Dataset Training Dataset Learning Algorithm Training Post-Processing Cross Validation Final Classification/ Regression Model New DataPre-Processing Refinement Prediction Split Supervised Learning Sebastian Raschka 2014 Missing Data Performance Metrics Model Selection Hyperparameter Optimization This work is licensed under a Creative Commons Attribution 4.0 International License. Final Model Evaluation
Questions? mail@sebastianraschka.com https://github.com/rasbt @rasbt Thanks!
Additional Slides
Inspiring Literature P. N. Klein. Coding the Matrix: Linear Algebra Through Computer Science Applications. Newtonian Press, 2013. R. Schutt and C. O’Neil. Doing Data Science: Straight Talk from the Frontline. O’Reilly Media, Inc., 2013. S. Gutierrez. Data Scientists at Work. Apress, 2014. R. O. Duda, P. E. Hart, and D. G. Stork. Pattern classification. 2nd. Edition. New York, 2001.
Useful Online Resources https://www.coursera.org/course/ml http://stats.stackexchange.com http://www.kaggle.com
My Favorite Tools http://stanford.edu/~mwaskom/software/seaborn/ Seaborn http://www.numpy.org http://pandas.pydata.org http://scikit-learn.org/stable/ http://ipython.org/notebook.html
class1 class2 Which one to pick?
class1 class2 The problem of overfitting Generalization error!

More Related Content

PPTX
Decision Tree Classification Algorithm.pptx
PriyadharshiniG41
 
PPTX
Support Vector Machines
CloudxLab
 
PPTX
Feature Selection in Machine Learning
Upekha Vandebona
 
PPTX
Support Vector Machines Simply
Emad Nabil
 
PDF
Naive Bayes Classifier
Arunabha Saha
 
PDF
L2. Evaluating Machine Learning Algorithms I
Machine Learning Valencia
 
PPTX
Lect9 Decision tree
hktripathy
 
PDF
Module 4: Model Selection and Evaluation
Sara Hooker
 
Decision Tree Classification Algorithm.pptx
PriyadharshiniG41
 
Support Vector Machines
CloudxLab
 
Feature Selection in Machine Learning
Upekha Vandebona
 
Support Vector Machines Simply
Emad Nabil
 
Naive Bayes Classifier
Arunabha Saha
 
L2. Evaluating Machine Learning Algorithms I
Machine Learning Valencia
 
Lect9 Decision tree
hktripathy
 
Module 4: Model Selection and Evaluation
Sara Hooker
 

What's hot (20)

PPTX
Cnn
Nirthika Rajendran
 
PPT
Back propagation
Nagarajan
 
PPTX
Machine learning with ADA Boost
Aman Patel
 
PPTX
Breast cancer diagnosis machine learning ppt
AnkitGupta1476
 
PPTX
What Is Deep Learning? | Introduction to Deep Learning | Deep Learning Tutori...
Simplilearn
 
PDF
Classification Based Machine Learning Algorithms
Md. Main Uddin Rony
 
PPTX
Convolutional Neural Networks
Ashray Bhandare
 
PDF
Dimensionality Reduction
mrizwan969
 
PPTX
boosting algorithm
Prithvi Paneru
 
PPTX
Recurrent Neural Network (RNN) | RNN LSTM Tutorial | Deep Learning Course | S...
Simplilearn
 
PDF
Bayesian Networks - A Brief Introduction
Adnan Masood
 
PPT
Reinforcement learning
Chandra Meena
 
PPT
3.2 partitioning methods
Krish_ver2
 
PPTX
Activation function
Astha Jain
 
PPTX
Feed forward ,back propagation,gradient descent
Muhammad Rasel
 
PPTX
Decision tree
Venkata Reddy Konasani
 
PPTX
Presentation on supervised learning
Tonmoy Bhagawati
 
PPT
Support Vector Machines
nextlib
 
PPSX
Perceptron (neural network)
EdutechLearners
 
PPTX
Inductive bias
swapnac12
 
Cnn
Nirthika Rajendran
 
Back propagation
Nagarajan
 
Machine learning with ADA Boost
Aman Patel
 
Breast cancer diagnosis machine learning ppt
AnkitGupta1476
 
What Is Deep Learning? | Introduction to Deep Learning | Deep Learning Tutori...
Simplilearn
 
Classification Based Machine Learning Algorithms
Md. Main Uddin Rony
 
Convolutional Neural Networks
Ashray Bhandare
 
Dimensionality Reduction
mrizwan969
 
boosting algorithm
Prithvi Paneru
 
Recurrent Neural Network (RNN) | RNN LSTM Tutorial | Deep Learning Course | S...
Simplilearn
 
Bayesian Networks - A Brief Introduction
Adnan Masood
 
Reinforcement learning
Chandra Meena
 
3.2 partitioning methods
Krish_ver2
 
Activation function
Astha Jain
 
Feed forward ,back propagation,gradient descent
Muhammad Rasel
 
Decision tree
Venkata Reddy Konasani
 
Presentation on supervised learning
Tonmoy Bhagawati
 
Support Vector Machines
nextlib
 
Perceptron (neural network)
EdutechLearners
 
Inductive bias
swapnac12
 
Ad

Viewers also liked (20)

PPTX
Tutorial on Deep learning and Applications
NhatHai Phan
 
PPTX
Introduction to Big Data/Machine Learning
Lars Marius Garshol
 
PPTX
Artificial neural network
DEEPASHRI HK
 
PPTX
Hadoop and Machine Learning
joshwills
 
PDF
How to Interview a Data Scientist
Daniel Tunkelang
 
PDF
Data By The People, For The People
Daniel Tunkelang
 
PDF
A Statistician's View on Big Data and Data Science (Version 1)
Prof. Dr. Diego Kuonen
 
PDF
Hands-on Deep Learning in Python
Imry Kissos
 
PDF
10 Lessons Learned from Building Machine Learning Systems
Xavier Amatriain
 
PDF
Big Data [sorry] & Data Science: What Does a Data Scientist Do?
Data Science London
 
PDF
How to Become a Data Scientist
ryanorban
 
PDF
A tutorial on deep learning at icml 2013
Philip Zheng
 
PDF
Introduction to Mahout and Machine Learning
Varad Meru
 
PPTX
Deep Learning for Natural Language Processing
Devashish Shanker
 
PDF
Machine Learning and Data Mining: 12 Classification Rules
Pier Luca Lanzi
 
PDF
Myths and Mathemagical Superpowers of Data Scientists
David Pittman
 
PDF
Tips for data science competitions
Owen Zhang
 
PPTX
Deep neural networks
Si Haem
 
PPTX
10 R Packages to Win Kaggle Competitions
DataRobot
 
PPTX
Artificial Intelligence Presentation
lpaviglianiti
 
Tutorial on Deep learning and Applications
NhatHai Phan
 
Introduction to Big Data/Machine Learning
Lars Marius Garshol
 
Artificial neural network
DEEPASHRI HK
 
Hadoop and Machine Learning
joshwills
 
How to Interview a Data Scientist
Daniel Tunkelang
 
Data By The People, For The People
Daniel Tunkelang
 
A Statistician's View on Big Data and Data Science (Version 1)
Prof. Dr. Diego Kuonen
 
Hands-on Deep Learning in Python
Imry Kissos
 
10 Lessons Learned from Building Machine Learning Systems
Xavier Amatriain
 
Big Data [sorry] & Data Science: What Does a Data Scientist Do?
Data Science London
 
How to Become a Data Scientist
ryanorban
 
A tutorial on deep learning at icml 2013
Philip Zheng
 
Introduction to Mahout and Machine Learning
Varad Meru
 
Deep Learning for Natural Language Processing
Devashish Shanker
 
Machine Learning and Data Mining: 12 Classification Rules
Pier Luca Lanzi
 
Myths and Mathemagical Superpowers of Data Scientists
David Pittman
 
Tips for data science competitions
Owen Zhang
 
Deep neural networks
Si Haem
 
10 R Packages to Win Kaggle Competitions
DataRobot
 
Artificial Intelligence Presentation
lpaviglianiti
 
Ad

Similar to An Introduction to Supervised Machine Learning and Pattern Classification: The Big Picture (20)

PDF
Introduction to conventional machine learning techniques
Xavier Rafael Palou
 
PDF
machine_learning.pptx
Panchami V U
 
PPT
[ppt]
butest
 
PPT
[ppt]
butest
 
PPT
Download It
butest
 
PPTX
Machine learning and decision trees
Padma Metta
 
PPT
ai4.ppt
akshatsharma823122
 
PPT
ai4.ppt
ssuser448ad3
 
PPT
594503964-Introduction-to-Classification-PPT-Slides-1.ppt
snehajuly2004
 
PDF
Presentation-19.08.2024hvug7gugyvuvugugugugugug
amanna7980
 
PDF
MS CS - Selecting Machine Learning Algorithm
Kaniska Mandal
 
PPT
ai4.ppt
atul404633
 
PPT
Introduction to Machine Learning Aristotelis Tsirigos
butest
 
PPT
Computational Biology, Part 4 Protein Coding Regions
butest
 
PPTX
machine leraning : main principles and techniques
johngeorgakis99
 
PPTX
AI -learning and machine learning.pptx
GaytriDhingra1
 
PPTX
Machine Learning
GaytriDhingra1
 
PDF
Machine Learning Comparative Analysis - Part 1
Kaniska Mandal
 
PDF
IRJET- Performance Evaluation of Various Classification Algorithms
IRJET Journal
 
PDF
IRJET- Performance Evaluation of Various Classification Algorithms
IRJET Journal
 
Introduction to conventional machine learning techniques
Xavier Rafael Palou
 
machine_learning.pptx
Panchami V U
 
[ppt]
butest
 
[ppt]
butest
 
Download It
butest
 
Machine learning and decision trees
Padma Metta
 
ai4.ppt
akshatsharma823122
 
ai4.ppt
ssuser448ad3
 
594503964-Introduction-to-Classification-PPT-Slides-1.ppt
snehajuly2004
 
Presentation-19.08.2024hvug7gugyvuvugugugugugug
amanna7980
 
MS CS - Selecting Machine Learning Algorithm
Kaniska Mandal
 
ai4.ppt
atul404633
 
Introduction to Machine Learning Aristotelis Tsirigos
butest
 
Computational Biology, Part 4 Protein Coding Regions
butest
 
machine leraning : main principles and techniques
johngeorgakis99
 
AI -learning and machine learning.pptx
GaytriDhingra1
 
Machine Learning
GaytriDhingra1
 
Machine Learning Comparative Analysis - Part 1
Kaniska Mandal
 
IRJET- Performance Evaluation of Various Classification Algorithms
IRJET Journal
 
IRJET- Performance Evaluation of Various Classification Algorithms
IRJET Journal
 

Recently uploaded (20)

PPT
Chapter 3 METAL JOINING.pptnnnnnnnnnnnnn
JanakiRaman206018
 
PPTX
Introduction to machine learning and Linear Models
ShivanshMehta17
 
PPTX
IB Computer Science - Internal Assessment.pptx
agarwal18harsh08
 
PPTX
Major-Components-ofNKJNNKNKNKNKronment.pptx
dushyantsharma1221
 
PDF
"Python Programming for Geospatial Data Science." ...
institute of Geoinformatics and Earth Observation at PMAS ARID Agriculture University of Rawalpindi
 
PDF
Mega Projects Data Mega Projects Data
saidsalah8181
 
PPTX
Business Acumen Training GuidePresentation.pptx
stephvshelton19
 
PPTX
Data_Analytics_and_PowerBI_Presentation.pptx
ZubyrAhmed
 
PDF
Clinical guidelines as a resource for EBP(1).pdf
MashalKhan626345
 
PDF
Taxes Foundatisdcsdcsdon Certificate.pdf
PratyushPrem2
 
PPTX
Introduction to Knowledge Engineering Part 1
busyprogrammersguide
 
PPTX
The THESIS FINAL-DEFENSE-PRESENTATION.pptx
hshakobe3
 
PPT
Reliability_Chapter_ presentation 1221.5784
abobaker13
 
PPTX
CEE 2 REPORT G7.pptxbdbshjdgsgjgsjfiuhsd
hildogavino28
 
PPTX
DISORDERS OF THE LIVER, GALLBLADDER AND PANCREASE (1).pptx
chepkoitcheruiyot
 
PPTX
ALIMENTARY AND BILIARY CONDITIONS 3-1.pptx
chepkoitcheruiyot
 
PPTX
A Quantitative-WPS Office.pptx research study
ssuser7b04d8
 
PDF
Lecture1 pattern recognition............
ZafriFarid
 
PPTX
Introduction to Firewall Analytics - Interfirewall and Transfirewall.pptx
kuthubussaman1
 
PPTX
Bharatiya Antariksh Hackathon 2025 Idea Submission PPT.pptx
abhinavmemories2026
 
Chapter 3 METAL JOINING.pptnnnnnnnnnnnnn
JanakiRaman206018
 
Introduction to machine learning and Linear Models
ShivanshMehta17
 
IB Computer Science - Internal Assessment.pptx
agarwal18harsh08
 
Major-Components-ofNKJNNKNKNKNKronment.pptx
dushyantsharma1221
 
"Python Programming for Geospatial Data Science." ...
institute of Geoinformatics and Earth Observation at PMAS ARID Agriculture University of Rawalpindi
 
Mega Projects Data Mega Projects Data
saidsalah8181
 
Business Acumen Training GuidePresentation.pptx
stephvshelton19
 
Data_Analytics_and_PowerBI_Presentation.pptx
ZubyrAhmed
 
Clinical guidelines as a resource for EBP(1).pdf
MashalKhan626345
 
Taxes Foundatisdcsdcsdon Certificate.pdf
PratyushPrem2
 
Introduction to Knowledge Engineering Part 1
busyprogrammersguide
 
The THESIS FINAL-DEFENSE-PRESENTATION.pptx
hshakobe3
 
Reliability_Chapter_ presentation 1221.5784
abobaker13
 
CEE 2 REPORT G7.pptxbdbshjdgsgjgsjfiuhsd
hildogavino28
 
DISORDERS OF THE LIVER, GALLBLADDER AND PANCREASE (1).pptx
chepkoitcheruiyot
 
ALIMENTARY AND BILIARY CONDITIONS 3-1.pptx
chepkoitcheruiyot
 
A Quantitative-WPS Office.pptx research study
ssuser7b04d8
 
Lecture1 pattern recognition............
ZafriFarid
 
Introduction to Firewall Analytics - Interfirewall and Transfirewall.pptx
kuthubussaman1
 
Bharatiya Antariksh Hackathon 2025 Idea Submission PPT.pptx
abhinavmemories2026
 

An Introduction to Supervised Machine Learning and Pattern Classification: The Big Picture

  • 1. Practical Data Science An Introduction to Supervised Machine Learning and Pattern Classification: The Big Picture Michigan State University NextGen Bioinformatics Seminars - 2015 Sebastian Raschka Feb. 11, 2015
  • 2. A Little Bit About Myself ... Developing software & methods for - Protein ligand docking - Large scale drug/inhibitor discovery PhD candidate in Dr. L. Kuhn’s Lab: and some other machine learning side-projects …
  • 3. What is Machine Learning? http://drewconway.com/zia/2013/3/26/the-data-science-venn-diagram "Field of study that gives computers the ability to learn without being explicitly programmed.” (Arthur Samuel, 1959) By Phillip Taylor [CC BY 2.0]
  • 5. Examples of Machine Learning http://googleresearch.blogspot.com/2014/11/a-picture-is-worth-thousand-coherent.html By Steve Jurvetson [CC BY 2.0] Self-driving cars Photo search and many, many more ... Recommendation systems http://commons.wikimedia.org/wiki/File:Netflix_logo.svg [public domain]
  • 6. How many of you have used machine learning before?
  • 7. Our Agenda • Concepts and the big picture • Workflow • Practical tips & good habits
  • 8. Learning • Labeled data • Direct feedback • Predict outcome/future • Decision process • Reward system • Learn series of actions • No labels • No feedback • “Find hidden structure” Unsupervised Supervised Reinforcement
  • 9. Unsupervised learning Supervised learning Clustering: [DBSCAN on a toy dataset] Classification: [SVM on 2 classes of the Wine dataset] Regression: [Soccer Fantasy Score prediction] Today’s topic Supervised LearningUnsupervised Learning
  • 10. Instances (samples, observations) Features (attributes, dimensions) Classes (targets) Nomenclature sepal_length sepal_width petal_length petal_width class 1 5.1 3.5 1.4 0.2 setosa 2 4.9 3.0 1.4 0.2 setosa … … … … … … 50 6.4 3.2 4.5 1.5 veriscolor … … … … … … 150 5.9 3.0 5.1 1.8 virginica https://archive.ics.uci.edu/ml/datasets/Iris IRIS
  • 11. Classification x1 x2 class1 class2 1) Learn from training data 2) Map unseen (new) data ?
  • 12. Feature Extraction Feature Selection Dimensionality Reduction Feature Scaling Raw Data Collection Pre-Processing Sampling Test Dataset Training Dataset Learning Algorithm Training Post-Processing Cross Validation Final Classification/ Regression Model New DataPre-Processing Refinement Prediction Split Supervised Learning Sebastian Raschka 2014 Missing Data Performance Metrics Model Selection Hyperparameter Optimization This work is licensed under a Creative Commons Attribution 4.0 International License. Final Model Evaluation
  • 13. Feature Extraction Feature Selection Dimensionality Reduction Feature Scaling Raw Data Collection Pre-Processing Sampling Test Dataset Training Dataset Learning Algorithm Training Post-Processing Cross Validation Final Classification/ Regression Model New DataPre-Processing Refinement Prediction Split Supervised Learning Sebastian Raschka 2014 Missing Data Performance Metrics Model Selection Hyperparameter Optimization This work is licensed under a Creative Commons Attribution 4.0 International License. Final Model Evaluation
  • 14. A Few Common Classifiers Decision Tree Perceptron Naive Bayes Ensemble Methods: Random Forest, Bagging, AdaBoost Support Vector Machine K-Nearest Neighbor Logistic Regression Artificial Neural Network / Deep Learning
  • 15. Discriminative Algorithms Generative Algorithms • Models a more general problem: how the data was generated. • I.e., the distribution of the class; joint probability distribution p(x,y). • Naive Bayes, Bayesian Belief Network classifier, Restricted Boltzmann Machine … • Map x → y directly. • E.g., distinguish between people speaking different languages without learning the languages. • Logistic Regression, SVM, Neural Networks …
  • 16. Examples of Discriminative Classifiers: Perceptron xi1 xi2 w1 w2 Σ yi y = wTx = w0 + w1x1 + w2x2 1 F. Rosenblatt. The perceptron, a perceiving and recognizing automaton Project Para. Cornell Aeronautical Laboratory, 1957. x1 x2 y ∈ {-1,1} w0 wj = weight xi = training sample yi = desired output yi = actual output t = iteration step η = learning rate θ = threshold (here 0) update rule: wj(t+1) = wj(t) + η(yi - yi)xi 1 if wTxi ≥ θ -1 otherwise ^ ^ ^ ^ yi ^ until t+1 = max iter or error = 0
  • 17. Discriminative Classifiers: Perceptron F. Rosenblatt. The perceptron, a perceiving and recognizing automaton Project Para. Cornell Aeronautical Laboratory, 1957. - Binary classifier (one vs all, OVA) - Convergence problems (set n iterations) - Modification: stochastic gradient descent - “Modern” perceptron: Support Vector Machine (maximize margin) - Multilayer perceptron (MLP) xi1 xi2 w1 w2 Σ yi 1 y ∈ {-1,1} w0 ^ x1 x2
  • 18. Generative Classifiers: Naive Bayes Bayes Theorem: P(ωj | xi) = P(xi | ωj) P(ωj) P(xi) Posterior probability = Likelihood x Prior probability Evidence Iris example: P(“Setosa"| xi), xi = [4.5 cm, 7.4 cm]
  • 19. Generative Classifiers: Naive Bayes Decision Rule: Bayes Theorem: P(ωj | xi) = P(xi | ωj) P(ωj) P(xi) pred. class label ωj argmax P(ωj | xi) i = 1, …, m e.g., j ∈ {Setosa, Versicolor, Virginica}
  • 20. Class-conditional probability (here Gaussian kernel): Generative Classifiers: Naive Bayes Prior probability: Evidence: (cancels out) (class frequency) P(ωj | xi) = P(xi | ωj) P(ωj) P(xi) P(ωj) = Nωj Nc P(xik |ωj) = 1 √ (2 π σωj 2) exp ( )- (xik - μωj)2 2σωj 2 P(xi |ωj) P(xik |ωj)
  • 21. Generative Classifiers: Naive Bayes - Naive conditional independence assumption typically violated - Works well for small datasets - Multinomial model still quite popular for text classification (e.g., spam filter)
  • 22. Non-Parametric Classifiers: K-Nearest Neighbor - Simple! - Lazy learner - Very susceptible to curse of dimensionality k=3 e.g., k=1
  • 23. Iris Example Setosa Virginica Versicolor k = 3 mahalanobis dist. uniform weights C = 3 depth = 2
  • 24. Decision Tree Entropy = depth = 4 petal length <= 2.45? petal length <= 4.75? Setosa Virginica Versicolor Yes No e.g., 2 (- 0.5 log2(0.5)) = 1 ∑−pi logk pi i Information Gain = entropy(parent) – [avg entropy(children)] NoYes depth = 2
  • 25. "No Free Lunch" :( Roughly speaking: “No one model works best for all possible situations.” Our model is a simplification of reality Simplification is based on assumptions (model bias) Assumptions fail in certain situations D. H. Wolpert. The supervised learning no-free-lunch theorems. In Soft Computing and Industry, pages 25–42. Springer, 2002.
  • 26. Which Algorithm? • What is the size and dimensionality of my training set? • Is the data linearly separable? • How much do I care about computational efficiency? - Model building vs. real-time prediction time - Eager vs. lazy learning / on-line vs. batch learning - prediction performance vs. speed • Do I care about interpretability or should it "just work well?" • ...
  • 27. Feature Extraction Feature Selection Dimensionality Reduction Feature Scaling Raw Data Collection Pre-Processing Sampling Test Dataset Training Dataset Learning Algorithm Training Post-Processing Cross Validation Final Classification/ Regression Model New DataPre-Processing Refinement Prediction Split Supervised Learning Sebastian Raschka 2014 Missing Data Performance Metrics Model Selection Hyperparameter Optimization This work is licensed under a Creative Commons Attribution 4.0 International License. Final Model Evaluation
  • 28. Missing Values: - Remove features (columns) - Remove samples (rows) - Imputation (mean, nearest neighbor, …) Sampling: - Random split into training and validation sets - Typically 60/40, 70/30, 80/20 - Don’t use validation set until the very end! (overfitting) Feature Scaling: e.g., standardization: - Faster convergence (gradient descent) - Distances on same scale (k-NN with Euclidean distance) - Mean centering for free - Normal distributed data - Numerical stability by avoiding small weights z = xik - μk σk (use same parameters for the test/new data!)
  • 29. Categorical Variables color size prize class label 0 green M 10.1 class1 1 red L 13.5 class2 2 blue XL 15.3 class1 ordinalnominal green → (1,0,0) red → (0,1,0) blue → (0,0,1) class label color=blue color=green color=red prize size 0 0 0 1 0 10.1 1 1 1 0 0 1 13.5 2 2 0 1 0 0 15.3 3 M → 1 L → 2 XL → 3
  • 30. Feature Extraction Feature Selection Dimensionality Reduction Feature Scaling Raw Data Collection Pre-Processing Sampling Test Dataset Training Dataset Learning Algorithm Training Post-Processing Cross Validation Final Classification/ Regression Model New DataPre-Processing Refinement Prediction Split Supervised Learning Sebastian Raschka 2014 Missing Data Performance Metrics Model Selection Hyperparameter Optimization This work is licensed under a Creative Commons Attribution 4.0 International License. Final Model Evaluation
  • 31. Generalization Error and Overfitting How well does the model perform on unseen data?
  • 32. Generalization Error and Overfitting
  • 33. Error Metrics: Confusion Matrix TP [Linear SVM on sepal/petal lengths] TN FN FP here: “setosa” = “positive”
  • 34. Error Metrics TP [Linear SVM on sepal/petal lengths] TN FN FP here: “setosa” = “positive” TP + TN FP +FN +TP +TN Accuracy = = 1 - Error FP N TP P False Positive Rate = TP TP + FP Precision = True Positive Rate = (Recall) “micro” and “macro” averaging for multi-class
  • 36. Test set Training dataset Test dataset Complete dataset Test set Test set Test set 1st iteration calc. error calc. error calc. error calc. error calculate avg. error k-fold cross-validation (k=4): 2nd iteration 3rd iteration 4th iteration fold 1 fold 2 fold 3 fold 4 Model Selection
  • 38. Feature Selection - Domain knowledge - Variance threshold - Exhaustive search - Decision trees - … IMPORTANT! (Noise, overfitting, curse of dimensionality, efficiency) X = [x1, x2, x3, x4]start: stop: (if d = k) X = [x1, x3, x4] X = [x1, x3] Simplest example: Greedy Backward Selection
  • 39. Dimensionality Reduction • Transformation onto a new feature subspace • e.g., Principal Component Analysis (PCA) • Find directions of maximum variance • Retain most of the information
  • 40. 0. Standardize data 1. Compute covariance matrix z = xik - μk σik = ∑ (xij - µj) (xik - µk) σk 1 in -1 σ2 1 σ12 σ13 σ14 σ21 σ2 2 σ23 σ24 σ31 σ32 σ2 3 σ34 σ41 σ42 σ43 σ2 4 ∑ = PCA in 3 Steps
  • 41. 2. Eigendecomposition and sorting eigenvalues PCA in 3 Steps X v = λ v Eigenvectors [[ 0.52237162 -0.37231836 -0.72101681 0.26199559] [-0.26335492 -0.92555649 0.24203288 -0.12413481] [ 0.58125401 -0.02109478 0.14089226 -0.80115427] [ 0.56561105 -0.06541577 0.6338014 0.52354627]] Eigenvalues [ 2.93035378 0.92740362 0.14834223 0.02074601] (from high to low)
  • 42. 3. Select top k eigenvectors and transform data PCA in 3 Steps Eigenvectors [[ 0.52237162 -0.37231836 -0.72101681 0.26199559] [-0.26335492 -0.92555649 0.24203288 -0.12413481] [ 0.58125401 -0.02109478 0.14089226 -0.80115427] [ 0.56561105 -0.06541577 0.6338014 0.52354627]] Eigenvalues [ 2.93035378 0.92740362 0.14834223 0.02074601] [First 2 PCs of Iris]
  • 44. Non-Linear Problems - XOR gate k=11 uniform weights C=1 C=1000, gamma=0.1 depth=4
  • 45. Kernel Trick Kernel function Kernel Map onto high-dimensional space (non-linear combinations)
  • 46. Kernel Trick Trick: No explicit dot product! Radius Basis Function (RBF) Kernel:
  • 47. Kernel PCA PC1, linear PCA PC1, kernel PCA
  • 48. Feature Extraction Feature Selection Dimensionality Reduction Feature Scaling Raw Data Collection Pre-Processing Sampling Test Dataset Training Dataset Learning Algorithm Training Post-Processing Cross Validation Final Classification/ Regression Model New DataPre-Processing Refinement Prediction Split Supervised Learning Sebastian Raschka 2014 Missing Data Performance Metrics Model Selection Hyperparameter Optimization This work is licensed under a Creative Commons Attribution 4.0 International License. Final Model Evaluation
  • 51. Inspiring Literature P. N. Klein. Coding the Matrix: Linear Algebra Through Computer Science Applications. Newtonian Press, 2013. R. Schutt and C. O’Neil. Doing Data Science: Straight Talk from the Frontline. O’Reilly Media, Inc., 2013. S. Gutierrez. Data Scientists at Work. Apress, 2014. R. O. Duda, P. E. Hart, and D. G. Stork. Pattern classification. 2nd. Edition. New York, 2001.
  • 55. class1 class2 The problem of overfitting Generalization error!