Data Preparation for Data Science
0 likes296 views
This document discusses the importance of data preparation and understanding for data science projects. It notes that most of the work in any data project involves cleaning and preparing the data. The document emphasizes gaining both syntactic and semantic understanding of data through techniques like determining true data types, analyzing data density and distributions, identifying outliers over time, and leveraging domain knowledge to understand equivalences. It also demonstrates some data preparation techniques and discusses implications for team structure, including the need for data scientists to be involved in data transformation and for teams to have both data science and software engineering skills.
Data Preparation for Data Science
- 1. Data Preparation for Data Science Casey Stella @casey_stella 2016 Casey Stella@casey_stella (Hortonworks) Data Preparation for Data Science 2016
- 2. Table of Contents Preliminaries Demo Questions Casey Stella@casey_stella (Hortonworks) Data Preparation for Data Science 2016
- 3. Introduction Hi, I’m Casey Stella! Casey Stella@casey_stella (Hortonworks) Data Preparation for Data Science 2016
- 4. Garbage In =⇒ Garbage Out “80% of the work in any data project is in cleaning the data.” — D.J. Patel in Data Jujitsu Casey Stella@casey_stella (Hortonworks) Data Preparation for Data Science 2016
- 5. Data Cleansing =⇒ Data Understanding There are two ways to understand your data • Syntactic Understanding • Semantic Understanding If you hope to get anything out of your data, you have to have a handle on both. Casey Stella@casey_stella (Hortonworks) Data Preparation for Data Science 2016
- 6. Syntactic Understanding: True Types A true type is a label applied to data points xi such that xi are mutually comparable. • Schemas type != true data type • A specific column can have many different types Casey Stella@casey_stella (Hortonworks) Data Preparation for Data Science 2016
- 7. Syntactic Understanding: Density Data density is an indication of how data is clumped together. Casey Stella@casey_stella (Hortonworks) Data Preparation for Data Science 2016
- 8. Syntactic Understanding: Density Data density is an indication of how data is clumped together. • For numerical data, distributions and statistical characteristics are informative • For non-numeric data, counts and distinct counts of a canonical representation are extremely useful. Casey Stella@casey_stella (Hortonworks) Data Preparation for Data Science 2016
- 9. Syntactic Understanding: Density Data density is an indication of how data is clumped together. • For numerical data, distributions and statistical characteristics are informative • For non-numeric data, counts and distinct counts of a canonical representation are extremely useful. Canonical representations are representations which give you an idea at a glance of the data format Casey Stella@casey_stella (Hortonworks) Data Preparation for Data Science 2016
- 10. Syntactic Understanding: Density Data density is an indication of how data is clumped together. • For numerical data, distributions and statistical characteristics are informative • For non-numeric data, counts and distinct counts of a canonical representation are extremely useful. Canonical representations are representations which give you an idea at a glance of the data format • Replacing digits with the character ‘d’ • Stripping whitespace • Normalizing punctuation Casey Stella@casey_stella (Hortonworks) Data Preparation for Data Science 2016
- 11. Syntactic Understanding: Density Data density is an indication of how data is clumped together. • For numerical data, distributions and statistical characteristics are informative • For non-numeric data, counts and distinct counts of a canonical representation are extremely useful. Canonical representations are representations which give you an idea at a glance of the data format • Replacing digits with the character ‘d’ • Stripping whitespace • Normalizing punctuation Data density is an assumption underlying any conclusions drawn from your data. Casey Stella@casey_stella (Hortonworks) Data Preparation for Data Science 2016
- 12. Syntactic Understanding: Density over Time ∆Density ∆t is how data clumps change over time. Casey Stella@casey_stella (Hortonworks) Data Preparation for Data Science 2016
- 13. Syntactic Understanding: Density over Time ∆Density ∆t is how data clumps change over time. This kind of analysis can show • Problems in the data pipeline Casey Stella@casey_stella (Hortonworks) Data Preparation for Data Science 2016
- 14. Syntactic Understanding: Density over Time ∆Density ∆t is how data clumps change over time. This kind of analysis can show • Problems in the data pipeline • Whether the assumptions of your analysis are violated Casey Stella@casey_stella (Hortonworks) Data Preparation for Data Science 2016
- 15. Syntactic Understanding: Density over Time ∆Density ∆t is how data clumps change over time. This kind of analysis can show • Problems in the data pipeline • Whether the assumptions of your analysis are violated ∆Density ∆t =⇒ • Automation Casey Stella@casey_stella (Hortonworks) Data Preparation for Data Science 2016
- 16. Syntactic Understanding: Density over Time ∆Density ∆t is how data clumps change over time. This kind of analysis can show • Problems in the data pipeline • Whether the assumptions of your analysis are violated ∆Density ∆t =⇒ • Automation • Outlier Alerting Casey Stella@casey_stella (Hortonworks) Data Preparation for Data Science 2016
- 17. Semantic Understanding: “Do what I mean, not what I say” Semantic understanding is understanding based on how the data is used rather than how it is stored. Casey Stella@casey_stella (Hortonworks) Data Preparation for Data Science 2016
- 18. Semantic Understanding: “Do what I mean, not what I say” Semantic understanding is understanding based on how the data is used rather than how it is stored. • Finding equivalences based on semantic understanding are often context sensitive. Casey Stella@casey_stella (Hortonworks) Data Preparation for Data Science 2016
- 19. Semantic Understanding: “Do what I mean, not what I say” Semantic understanding is understanding based on how the data is used rather than how it is stored. • Finding equivalences based on semantic understanding are often context sensitive. • May come from humans (e.g. domain experience and ontologies) Casey Stella@casey_stella (Hortonworks) Data Preparation for Data Science 2016
- 20. Semantic Understanding: “Do what I mean, not what I say” Semantic understanding is understanding based on how the data is used rather than how it is stored. • Finding equivalences based on semantic understanding are often context sensitive. • May come from humans (e.g. domain experience and ontologies) • May come from machine learning (e.g. analyzing usage patterns to find synonyms) Casey Stella@casey_stella (Hortonworks) Data Preparation for Data Science 2016
- 21. Semantic Understanding: “Do what I mean, not what I say” Semantic understanding is understanding based on how the data is used rather than how it is stored. • Finding equivalences based on semantic understanding are often context sensitive. • May come from humans (e.g. domain experience and ontologies) • May come from machine learning (e.g. analyzing usage patterns to find synonyms) Semantic understanding does not imply SkyNet Casey Stella@casey_stella (Hortonworks) Data Preparation for Data Science 2016
- 22. DEMO Casey Stella@casey_stella (Hortonworks) Data Preparation for Data Science 2016
- 27. Implications for Team Structure To be successful, Casey Stella@casey_stella (Hortonworks) Data Preparation for Data Science 2016
- 28. Implications for Team Structure To be successful, • Your data science teams have to be integrally involved in the data transformation and understanding. Casey Stella@casey_stella (Hortonworks) Data Preparation for Data Science 2016
- 29. Implications for Team Structure To be successful, • Your data science teams have to be integrally involved in the data transformation and understanding. • Your data science teams have to be willing to get their hands dirty Casey Stella@casey_stella (Hortonworks) Data Preparation for Data Science 2016
- 30. Implications for Team Structure To be successful, • Your data science teams have to be integrally involved in the data transformation and understanding. • Your data science teams have to be willing to get their hands dirty • Your data science teams have to be allowed to get their hands dirty Casey Stella@casey_stella (Hortonworks) Data Preparation for Data Science 2016
- 31. Implications for Team Structure To be successful, • Your data science teams have to be integrally involved in the data transformation and understanding. • Your data science teams have to be willing to get their hands dirty • Your data science teams have to be allowed to get their hands dirty • Your data science teams need software engineering chops. Casey Stella@casey_stella (Hortonworks) Data Preparation for Data Science 2016
- 32. Questions Thanks for your attention! Questions? • Code & scripts for this talk available on my github presentation page.1 • Find me at http://caseystella.com • Twitter handle: @casey_stella • Email address: cstella@hortonworks.com 1 http://github.com/cestella/presentations/ Casey Stella@casey_stella (Hortonworks) Data Preparation for Data Science 2016