Light up Your Dark Data by Lance Ransom at QuantCon 2016
1 like227 views
This document discusses dark data, or data that is unused and unanalyzed. It presents QuantCon's solution to illuminate dark data through the Lux, Blaze, and Anaconda Mosaic projects. Lux provides a unified view of disparate data sources. Blaze enables storage-independent data analysis expressions. Anaconda Mosaic allows interactive data exploration, visualization, and governance.
Light up Your Dark Data by Lance Ransom at QuantCon 2016
- 1. QuantCon “Light Up Your Dark Data” April 2016
- 2. What is dark data? 2 SQL CSV REST JSON SQL CSV REST JSON SQL CSV SQL CSV
- 6. Sample Environment 6 Oracle MySQL MSSQL KDB ZIPCSV SQL Python DSL R Matlab C++ Java Storage ETL Analysis REST
- 7. Independent First Class Citizens 7 Expression ComputeData
- 9. Datashape Example 9 daily_bars: var * { date: string, symbol: string, open: float64, high: float64, low: float64, close: float64, volume: int64, } Language, compute, and storage independent
- 13. Flat File Repositories 13 Many directories and files Dictated structure Naming convention part of dataset Requires one off ad-hoc scripts
- 14. Vendor - directory structure /daily/us/nasdaq stocks/ /daily/us/nasdaq stocks/1/ /daily/us/nasdaq stocks/2/ osn.us.txt ostk.us.txt … zyne.us.txt /daily/us/nyse etfs/ /daily/us/nyse stocks/1/ /daily/us/nyse stocks/2/ Contains ~8400 individual files 14
- 15. Vendor – file contents 15 Date,Open,High,Low,Close,Volume,OpenInt 20151111,18.5,25.9,18,24.5,1584600,0 20151112,24.25,27.12,22.5,25,83000,0 20151113,25.47,26.2,24.55,25.26,67300,0 20151116,25.01,26.19,24.13,25.02,16900,0 20151117,24.46,25.51,24.38,24.62,25900,0 20151118,24.62,26.31,24.06,25,111100,0 20151119,24.85,26,24.71,25.9,113100,0 … Symbol is not contained within the individual data files /daily/us/nasdaq stocks/1/aaap.us.txt
- 16. Lux 16 source: "lux://global-equities/data/daily/us/nasdaq stocks" extractor: "{}/{Symbol}.{Region}.txt" Date,Open,High,Low,Close,Volume,OpenInt,Symbol,Region 20151111,18.5,25.9,18,24.5,1584600,0,aaap,us 20151112,24.25,27.12,22.5,25,83000,0,aaap,us 20151113,25.47,26.2,24.55,25.26,67300,0,aaap,us … 20160322,11.56,11.98,10.8894,11.09,517604,0,zyne,us 20160323,11.3,11.72,9.5,9.75,489743,0,zyne,us 20160324,9.5,10.24,9.22,9.64,188512,0,zyne,us One dataset with ~5.5 million rows
- 20. Project References • Anaconda Mosaic - http://know.continuum.io/Anaconda-Mosaic • Blaze Ecosystem - http://blaze.pydata.org • Bokeh - http://bokeh.pydata.org 20