![6
ML Pipeline
Evaluation & Tuning Methods [1]
Models have various parameters that are tuned in a process
called hyper parameter optimization to gate models with the best
and optimal results.
3-fold cross validation
ROC curve for binary and multi-class model evaluation
Classification models can be evaluated and tested on validation
datasets (k-fold cross) and based on metrics like:
• Accuracy
• Confusion matrix, ROC
Regression models can be evaluated by:
• Coefficient of Determination, R2
• Mean Squared Error
Clustering Models can be validated by:
• Homogeneity
• Completeness
• V-measures (combination)
• Silhouette Coefficient
• Calinski-Harabaz Index
Purpose
Methods
1 2 3 4 5
Source: Practical Machine Learning with Python](https://image.slidesharecdn.com/20180305deteconmlpipeline-180305164431/85/Guiding-through-a-typical-Machine-Learning-Pipeline-6-320.jpg)
![7
ML Pipeline
Evaluation & Tuning Methods [2]
Bias Variance Trade-Off
• Finding the best balance between Bias and Variance
Errors.
• Bias Error is the difference between expected and
predicted value of the model estimator. It is caused
by the underlying data and patterns.
• Variance errors arises due to model sensitivity of
outliers and random noise.
Bias Variance Trade Off
Underfitting
• Underfitting is seen as a parameter setup resulting in
a low variance and high bias.
Overfitting
• Overfitting is seen as a parameter setup resulting in
a high variance and low bias.
Grid Search
Simplest hyper-parameter
optimization method. Tries out a
predefined grid of hyper parameter
set to find the best.
Randomized Search
This is a modification of Grid
Search and uses a randomized
grid of hyper-parameter settings
to find the best one.
1 2 3 4 5
Source: Practical Machine Learning with Python](https://image.slidesharecdn.com/20180305deteconmlpipeline-180305164431/85/Guiding-through-a-typical-Machine-Learning-Pipeline-7-320.jpg)
Guiding through a typical Machine Learning Pipeline
- 1. Guiding through a typical Machine Learning Pipeline
- 2. 2 ML Pipeline The Standard Machine Learning Pipeline is derived from the CRISP-DM Model Datasets Data Retrieval Data Preparation & Feature Engineering Modeling Model Evaluation & Tuning Deployment & Monitoring ML Algorithm Satisfactory Perfor- mance? Data Processing & Wrangling Feature Extraction & Engineering Feature Scaling & Selection No Yes 1 2 3 4 5 Source: Practical Machine Learning with Python
- 3. 3 ML Pipeline Data Retrieval Raw Data Set Data Retrieval is mainly data collection, extraction and acquisition from various data sources and data stores. Data Sources or Formats, e.g.: • CSV • JSON • XML • SQL • SQLite • Web Scraping (DOM, HTML) Data Descriptions: • Numeric • Text • Categorical (Nominal, Ordinal) More data beats clever algorithms, but better data beats more data. Peter Norvig “ “ 1 2 3 4 5 Source: Practical Machine Learning with Python
- 4. 4 ML Pipeline Data Preparation & Feature Engineering Data outcome labels Dataset Features Feature set with categorical variables • In this step the data is pre-processed by cleaning, wrangling (munging) and manipulation as needed. • Initial exploratory data analysis is also carried out. • Data Wrangling • Data Understanding • Filtering • Typecasting • Data Transformation • Imputing Missing Values • Handling Duplicates • Handling Categorical Data • Normalizing Values • String Manipulations • Data Summarization • Data Visualization • Feature Engineering, Scaling, Selection • Dimensionality Reduction Data Visualization Purpose Methods 1 2 3 4 5 Source: Practical Machine Learning with Python
- 5. 5 Modelling Procedure ML Pipeline Modeling In the process of modeling, data features are usually fed to a ML method or algorithm and train the model, typically to optimize a specific cost function in most cases with the objective of reducing errors and generalizing the representations learned from the data. Model Types • Linear models • Logistic Regression • Naïve Bayes • Support Vector Machines • Non parametric models • K-Nearest Neighbors • Tree based models • Decision tree • Ensemble methods • Random forests • Gradient Boosted Machines • Neural Networks • Densely Neural networks (DNN) • Convolutional Neural networks (CNN) • Recurrent Neural networks (RNN) Regression models • Simple linear regression • Multiple linear regression • Non linear regression Clustering models • Partition based clustering • Hierarchical clustering • Density based clustering Classification models • Binary Classification • Multi-Class Classification • Multi Label Classification Activation Function Initializing Parameters Cost function, Metric definition Train with # of epochs Evaluate model with test data 1 2 3 4 5 Source: Practical Machine Learning with Python
- 6. 6 ML Pipeline Evaluation & Tuning Methods [1] Models have various parameters that are tuned in a process called hyper parameter optimization to gate models with the best and optimal results. 3-fold cross validation ROC curve for binary and multi-class model evaluation Classification models can be evaluated and tested on validation datasets (k-fold cross) and based on metrics like: • Accuracy • Confusion matrix, ROC Regression models can be evaluated by: • Coefficient of Determination, R2 • Mean Squared Error Clustering Models can be validated by: • Homogeneity • Completeness • V-measures (combination) • Silhouette Coefficient • Calinski-Harabaz Index Purpose Methods 1 2 3 4 5 Source: Practical Machine Learning with Python
- 7. 7 ML Pipeline Evaluation & Tuning Methods [2] Bias Variance Trade-Off • Finding the best balance between Bias and Variance Errors. • Bias Error is the difference between expected and predicted value of the model estimator. It is caused by the underlying data and patterns. • Variance errors arises due to model sensitivity of outliers and random noise. Bias Variance Trade Off Underfitting • Underfitting is seen as a parameter setup resulting in a low variance and high bias. Overfitting • Overfitting is seen as a parameter setup resulting in a high variance and low bias. Grid Search Simplest hyper-parameter optimization method. Tries out a predefined grid of hyper parameter set to find the best. Randomized Search This is a modification of Grid Search and uses a randomized grid of hyper-parameter settings to find the best one. 1 2 3 4 5 Source: Practical Machine Learning with Python
- 8. 8 ML Pipeline Deployment & Monitoring Selected models are deployed in production and are constantly monitored based on their predictions and results. Deployment Persistence Model Persistence is the simplest was of deploying A model. The final model will persist on permanent media Like hard drive. A new program must route real-life data to the persistent model which creates the predicted output. Custom Development Another option to deploy a model is by developing the implementation of model prediction method separately. The output is just the values of parameters that were learned. Method for the software development domain. In-House Model Deployment Due to data protection reasons a lot of enterprises do not want to expose their data on which models need to be built and deployed. Models can be easily integrated internally with web dev frameworks, APIs or micro-services on top of the prediction models. Model Deployment as a Service Model is open accessible and can be integrated via a cloud based API request. 1 2 3 4 5 Source: Practical Machine Learning with Python
- 9. 9 Michael Gerke Detecon International GmbH Sternengasse 14-16 50676 Cologne (Germany) Phone: +49 221 91611138 Mobile: +49 160 6907433 Email: Michael.Gerke@detecon.com ML Pipeline Contact Special Thanks to the author team: • Dipanjan Sarkar • Raghav Bali • Tushar Sharma