Polong Lin（林伯龍）/how to approach data science problems from start to end
8 likes2,364 views
The document outlines a comprehensive approach to solving data science problems, focusing on a methodology that starts with business understanding and follows through to model evaluation and feedback. It illustrates the application of this process using examples such as predicting flight delays and classifying cuisines based on ingredients. Key elements include defining project objectives, data preparation, model selection, and iterative evaluation for deployment.
![Using departing & arrival airport, date, carrier, etc.,
we could predict flight [DELAY] or [NO-DELAY] using
logistic regression.
• Identify suitable statistical/machine learning technique(s)
10
2. Analytic
Approach
• Linear regression
• Logistic regression
• Clustering
• Decision Trees
• Principal component
analysis
• Text analysis
• SVM/SVR
• Neural networks
• Dimension
Reduction
2. Analytic approach](https://image.slidesharecdn.com/polonglin-howtoapproachdatascienceproblemsfromstarttoend-160716094311/85/Polong-Lin-how-to-approach-data-science-problems-from-start-to-end-10-320.jpg)
![Data Preparation typically includes:
• Data cleaning
• Merging data
• Transforming data
• Feature engineering
• Text analysis
15
6. Data preparation
6. Data
Preparation
Flights are classified as “delayed” if >15 min late.
• Delayed? [True or False]
Does time of day for departure predict delays?
• Hour](https://image.slidesharecdn.com/polonglin-howtoapproachdatascienceproblemsfromstarttoend-160716094311/85/Polong-Lin-how-to-approach-data-science-problems-from-start-to-end-15-320.jpg)
Polong Lin（林伯龍）/how to approach data science problems from start to end
- 1. How to Approach Data Science Problems from Start to End Polong Lin Data Scientist IBM Analytics, Emerging Technologies @polonglin @bigdatau 台灣資料科學年會
- 2. • Free online courses • Data Science & Data Engineering • A communityinitiative led by IBM • Certificates and Badges • > 450,000 users What is Big Data University (BDU)?
- 3. 3
- 4. 4
- 9. • Every project begins with business understanding. • What is the project objective? • What are we trying to do – what is our goal? 1. Formulate a clear question 2. Define problem and solution requirements 9 1. Business Understanding Flight delays: Create some solution that can help users predict if a flight on a given day will be delayed or not delayed 1. Business understanding
- 10. Using departing & arrival airport, date, carrier, etc., we could predict flight [DELAY] or [NO-DELAY] using logistic regression. • Identify suitable statistical/machine learning technique(s) 10 2. Analytic Approach • Linear regression • Logistic regression • Clustering • Decision Trees • Principal component analysis • Text analysis • SVM/SVR • Neural networks • Dimension Reduction 2. Analytic approach
- 11. 11 3. Data Requirements 4. Data Collection 5. Data Understanding What data is required? What format? Collect the data What does the data look like? What are initial insights? Can we visualize the data? Are missing anything? • Flight data • Open data available • All domestic US flights per year • CSV format • Which airports are busiest? • Which flights are most delayed? • Which airports are best/worst?
- 15. Data Preparation typically includes: • Data cleaning • Merging data • Transforming data • Feature engineering • Text analysis 15 6. Data preparation 6. Data Preparation Flights are classified as “delayed” if >15 min late. • Delayed? [True or False] Does time of day for departure predict delays? • Hour
- 18. Modeling is a: • Highly iterative process • Multiple models may be used and tested 18 Modelling Modeling Using inputs: • Year • Month • Day of Month • Hour of departure • Distance • Destination airport Predict: Delay (True/False) Logistic Regression
- 20. • Once finalized, the model is deployed into a production environment. • May be in a limited / test environment until model is proven • Involves additional groups, skills, and technologies • Solution owner • Marketing • Application developers and designers • IT administration • Feedback to assess model performance • Gathering and analysis of feedback for assessment of the model’s performance and impact • Iterative process for model refinement and redeployment • Accelerate through automated processes 20 Deployment Feedback Prediction Interpretation Justification Testing Deployment and feedback
- 27. A. Based on the ingredients alone, can we predict what cuisine a food dish belongs to? B. Which cuisines are similar to each other based on their ingredients? 27 Business 1. Research Understanding Japanese American British Indian Chinese French Italian Vietnamese Canadian Food and ingredients
- 28. 28 Rice? ALL CUISINES NON-ASIAN FOOD ASIAN FOOD NO YES Wasabi? NO YES NOT JAPANESE JAPANESE A. Based on the ingredients alone, can we predict what cuisine a food dish belongs to? 2. Analytic Approach Decision trees
- 29. B. Which cuisines are similar to each other based on their ingredients? Analytic Approach K-means Clustering Group similar cuisines together into k number of clusters.