The Joys of Clean Data with Matt Dowle
8 likes2,548 views
The document discusses the importance of clean data in the context of machine intelligence, featuring the speaker's experiences from various financial firms and the tools used for data cleaning and analysis. It highlights issues encountered with messy data and explores tools like kdb and R’s tidyr for data manipulation. A live demo showcases the process of data munging and emphasizes the need for the R community to prioritize addressing questions about packages rather than solely focusing on download statistics.
The Joys of Clean Data with Matt Dowle
- 1. H2O.ai Machine Intelligence The joy of Clean Data! San Francisco Big Data Science Meetup 15 Dec 2015 Matt Dowle
- 2. H2O.ai Machine Intelligence 2 Overview ● For beginners ● Examples from my background ● Tools along the way ● Live demo of “80% munging” ● How H2O fits in ● Q & A
- 3. H2O.ai Machine Intelligence 3 1996 – Lehman Brothers ● Just graduated – Applied Maths & Computing ● Dividend claims ● Cleaning at source; e.g. data entry typos ● Estimate cash flows, alerts etc ● Nothing fancy ● Tools: VB & Sybase ● How I accidentally created messy data
- 4. H2O.ai Machine Intelligence 4 1999 - Salomon Brothers ● Equity risk model – Multiple time series regression (10 year) – DEM proxy for EUR prior to 1 Jan 1999 – IPOs get their sector's median; e.g. France Telecom – Abbey National X0004455 – 90% of the lines of code was not the regression
- 5. H2O.ai Machine Intelligence 5 2002 - Citigroup ● Pairs Trading ● 200 most liquid stocks ● 200 x 199 / 2 = 19,900 pairs ● Stock splits, id changes ● Dickey Fuller test for stationarity ● Bollinger bands => buy/sell signal ● Excel spreadsheet to clients with embedded S- PLUS plot, daily, 50 custom variants ● Rebalance => orphan & surrogate pairs
- 6. H2O.ai Machine Intelligence 6 2004 moved to fund management Bigger data e.g. 25TB
- 8. H2O.ai Machine Intelligence 8 Over cleaning 1. I queried for intra-day auctions select from quote where bid>ask No results; i.e. all bid<ask. Asked data provider Grrrrr 2. Negative prices can be correct
- 9. H2O.ai Machine Intelligence 9 Tools KDB http://kx.com/ @kxsystems OneTick https://www.onetick.com/ @OneMarketData
- 12. H2O.ai Machine Intelligence 12 alias awk aws bc bigmler body cat cols csvcut csvgrep csvjoin csvlook csvsort csvsql csvstack csvstat curl cut dseq find for grep head header in2csv jq json2csv less parallel paste pbc python, R and r Rio Rio-scatter run_experiment sample scrape sed seq shuf sort split sql2csv tail tapkee tee tr tree uniq wc weka xml2json ● Can be faster than loading the whole file into R or Python ● Can be faster workflow ● Pre-processing before loading into R or Python
- 13. H2O.ai Machine Intelligence 13 tidyr by Hadley Wickham https://cran.r-project.org/web/packages/tidyr/vignettes/tidy-data.html ● Untidy data defined as : – Column headers are values, not variable names. – Multiple variables are stored in one column. – Variables are stored in both rows and columns. – Multiple types of observational units stored in the same table. – A single observational unit is stored in multiple tables. ● Solves by: gathering, separating and spreading ● That's the shape of the data. Yes, good, but not the kind of messy data I'm talking about in this presentation.
- 14. H2O.ai Machine Intelligence 14 To illustrate ● In June 2013, RStudio made available download logs from their CRAN mirror http://blog.rstudio.org/2013/06/10/rstudio-cran-mirror/ ● R-Bloggers search “CRAN download stats” 154 results; e.g. http://www.r-bloggers.com/finally-tracking-cran-packages-downloads/ https://github.com/metacran/cranlogs http://www.r-bloggers.com/working-with-the-rstudio-cran-logs/ http://www.r-bloggers.com/cran-download-statistics-of-any-packages-rstats/ http://www.r-bloggers.com/my-r-packages-worldmap-of-downloads/
- 17. H2O.ai Machine Intelligence 17 ● Comparing the top downloaded packages with the most discussed packages shows little correlations between them. For Instance, ggplot2 has the most questions asked & is the second highest downloaded package but data.table package (the second highest ranked R package for questions asked) is not even in the top 100 packages downloaded. Knitr is another example which is in the top 5 questions asked, but is 27th ranked in downloaded packages. ● So- does the R community need to focus on packages that have the highest questions to resolve their issues rather than the ones with the most downloads?
- 18. H2O.ai Machine Intelligence 18 Let's look at the data! Live demo of munging Observations and comments on meetup video recording https://youtu.be/4VWQEvYIfV8 ( ~ 22 mins in )
- 19. H2O.ai Machine Intelligence 19 “Big data” 1. Data > 240GB needle-in-haystack e.g. fraud 2. Data < 240GB compute intensive, parallel 100's cores 3. Data < 240GB feature engineering > 240GB Speed for i) production and ii) interaction NB: 240GB is currently largest available on EC2
- 21. H2O.ai Machine Intelligence 21 Dell PowerEdge R920 60 core ( 4 * Intel® Xeon® E7-8880L 2.2GHz, 37.5M Cache, 15 Core ) with 1.5TB RAM $60k ( 96 * 16GB ) with 6TB $150k-$200k? ( 96 * 64GB ) But, still “only” 60 cores In the office here we already have 2.5TB RAM and 320 cores on 10 machines. So do many businesses.
- 22. H2O.ai Machine Intelligence 22 ● data.table's radix join ● Now parallel and distributed ● e.g. high cardinality 1bn/1bn/1bn row join data.table 10 min H2O 1 node 32 core 3.5 min H2O 4 node 128 core 1.5 min => demo H2O 10 node 320 core 2.0 min ● Known improvements to be made