Slides 111017220255-phpapp01
1 like236 views
This document discusses using Python and the pandas library for financial data analysis. It provides an overview of pandas, describing it as providing rich data structures like DataFrame for working with financial time series and panel data. It highlights pandas' features for fast data alignment, time series functionality, and SQL-like operations which make it well-suited for financial analysis tasks. The document also presents pandas as addressing weaknesses that Python previously had for statistical analysis and filling gaps relative to data analysis tools like R.
![A simple example: quicksort
First try Python implementation:
def qsort ( array ):
if len ( array ) < 2:
return array
less , greater = [] , []
pivot , rest = array [0] , array [1:]
for x in rest :
if x < pivot :
less . append ( x )
else :
greater . append ( x )
return qsort ( less ) + [ pivot ] + qsort ( greater )
@wesmckinn () Data analysis with pandas 10/17/2011 9 / 22](https://image.slidesharecdn.com/slides-111017220255-phpapp01-121215103930-phpapp01/85/Slides-111017220255-phpapp01-9-320.jpg)
![A simple example: quicksort
Use list comprehensions:
def qsort ( array ):
if len ( array ) < 2:
return array
pivot , rest = array [0] , array [1:]
less = [ x for x in rest if x < pivot ]
greater = [ x for x in rest if x >= pivot ]
return qsort ( less ) + [ pivot ] + qsort ( greater )
@wesmckinn () Data analysis with pandas 10/17/2011 10 / 22](https://image.slidesharecdn.com/slides-111017220255-phpapp01-121215103930-phpapp01/85/Slides-111017220255-phpapp01-10-320.jpg)
![A simple example: quicksort
Heck, fit it onto one line!
qs = lambda r : ( r if len ( r ) < 2
else ( qs ([ x for x in r [1:] if x < r [0]])
+ [ r [0]]
+ qs ([ x for x in r [1:] if x >= r [0]])))
Though that’s starting to look like Lisp code...
@wesmckinn () Data analysis with pandas 10/17/2011 11 / 22](https://image.slidesharecdn.com/slides-111017220255-phpapp01-121215103930-phpapp01/85/Slides-111017220255-phpapp01-11-320.jpg)
![A simple example: quicksort
A quicksort using NumPy arrays
def qsort ( array ):
if len ( array ) < 2:
return array
pivot , rest = array [0] , array [1:]
less = rest [ rest < pivot ]
greater = rest [ rest >= pivot ]
return np . r_ [ qsort ( less ) , [ pivot ] , qsort ( greater )]
Of course no need for this when you can just do:
sorted_array = np.sort(array)
@wesmckinn () Data analysis with pandas 10/17/2011 12 / 22](https://image.slidesharecdn.com/slides-111017220255-phpapp01-121215103930-phpapp01/85/Slides-111017220255-phpapp01-12-320.jpg)
![Why did Python become popular in science?
NumPy traces its roots to 1995
Extremely easy to integrate C/C++/Fortran code
Access fast low level algorithms in a high level, interpreted language
The language itself
“It fits in your head”
“It [Python] doesn’t get in my way” - Robert Kern
Python is good at all the things other scientific programming
languages are not good at (e.g. networking, string processing, OOP)
Liberal BSD license: can use Python for commercial applications
@wesmckinn () Data analysis with pandas 10/17/2011 15 / 22](https://image.slidesharecdn.com/slides-111017220255-phpapp01-121215103930-phpapp01/85/Slides-111017220255-phpapp01-15-320.jpg)
Slides 111017220255-phpapp01
- 1. Financial data analysis in Python with pandas Wes McKinney @wesmckinn 10/17/2011 @wesmckinn () Data analysis with pandas 10/17/2011 1 / 22
- 2. My background 3 years as a quant hacker at AQR, now consultant / entrepreneur Math and statistics background with the zest of computer science Active in scientific Python community My blog: http://blog.wesmckinney.com Twitter: @wesmckinn @wesmckinn () Data analysis with pandas 10/17/2011 2 / 22
- 3. Bare essentials for financial research Fast time series functionality Easy data alignment Date/time handling Moving window statistics Resamping / frequency conversion Fast data access (SQL databases, flat files, etc.) Data visualization (plotting) Statistical models Linear regression Time series models: ARMA, VAR, ... @wesmckinn () Data analysis with pandas 10/17/2011 3 / 22
- 4. Would be nice to have Portfolio and risk analytics, backtesting Easy enough to write yourself, though most people do a bad job of it Portfolio optimization Most financial firms use a 3rd party library anyway Derivative pricing Can use QuantLib in most languages @wesmckinn () Data analysis with pandas 10/17/2011 4 / 22
- 5. What are financial firms using? HFT: a C++ and hardware arms race, a different topic Research Mainstream: R, MATLAB, Python, ... Econometrics: Stata, eViews, RATS, etc. Non-programmatic environments: ClariFI, Palantir, ... Production Popular: Java, C#, C++ Less popular, but growing: Python Fringe: Functional languages (Ocaml, Haskell, F#) @wesmckinn () Data analysis with pandas 10/17/2011 5 / 22
- 6. What are financial firms using? Many hybrid languages environments (e.g. Java/R, C++/R, C++/MATLAB, Python/C++) Which is the main implementation language? If main language is Java/C++, result is lower productivity and higher cost to prototyping new functionality Trends Banks and hedge funds are realizing that Java-based production systems can be replaced with 20% as much Python code (or less) MATLAB is being increasingly ditched in favor of Python. R and Python use for research generally growing @wesmckinn () Data analysis with pandas 10/17/2011 6 / 22
- 7. Python language Simple, expressive syntax Designed for readability, like “runnable pseudocode” Easy-to-use, powerful built-in types and data structures: Lists and tuples (fixed-size, immutable lists) Dicts (hash maps / associative arrays) and sets Everything’s an object, including functions “There should be one, and preferably only one way to do it” “Batteries included”: great general purpose standard library @wesmckinn () Data analysis with pandas 10/17/2011 7 / 22
- 8. A simple example: quicksort Pseudocode from Wikipedia: function qsort(array) if length(array) < 2 return array var list less, greater select and remove a pivot value pivot from array for each x in array if x < pivot then append x to less else append x to greater return concat(qsort(less), pivot, qsort(greater)) @wesmckinn () Data analysis with pandas 10/17/2011 8 / 22
- 9. A simple example: quicksort First try Python implementation: def qsort ( array ): if len ( array ) < 2: return array less , greater = [] , [] pivot , rest = array [0] , array [1:] for x in rest : if x < pivot : less . append ( x ) else : greater . append ( x ) return qsort ( less ) + [ pivot ] + qsort ( greater ) @wesmckinn () Data analysis with pandas 10/17/2011 9 / 22
- 10. A simple example: quicksort Use list comprehensions: def qsort ( array ): if len ( array ) < 2: return array pivot , rest = array [0] , array [1:] less = [ x for x in rest if x < pivot ] greater = [ x for x in rest if x >= pivot ] return qsort ( less ) + [ pivot ] + qsort ( greater ) @wesmckinn () Data analysis with pandas 10/17/2011 10 / 22
- 11. A simple example: quicksort Heck, fit it onto one line! qs = lambda r : ( r if len ( r ) < 2 else ( qs ([ x for x in r [1:] if x < r [0]]) + [ r [0]] + qs ([ x for x in r [1:] if x >= r [0]]))) Though that’s starting to look like Lisp code... @wesmckinn () Data analysis with pandas 10/17/2011 11 / 22
- 12. A simple example: quicksort A quicksort using NumPy arrays def qsort ( array ): if len ( array ) < 2: return array pivot , rest = array [0] , array [1:] less = rest [ rest < pivot ] greater = rest [ rest >= pivot ] return np . r_ [ qsort ( less ) , [ pivot ] , qsort ( greater )] Of course no need for this when you can just do: sorted_array = np.sort(array) @wesmckinn () Data analysis with pandas 10/17/2011 12 / 22
- 13. Python: drunk with power This comic has way too much airtime but: @wesmckinn () Data analysis with pandas 10/17/2011 13 / 22
- 14. Staples of Python for science: MINS (M) matplotlib: plotting and data visualization (I) IPython: rich interactive computing and development environment (N) NumPy: multi-dimensional arrays, linear algebra, FFTs, random number generation, etc. (S) SciPy: optimization, probability distributions, signal processing, ODEs, sparse matrices, ... @wesmckinn () Data analysis with pandas 10/17/2011 14 / 22
- 15. Why did Python become popular in science? NumPy traces its roots to 1995 Extremely easy to integrate C/C++/Fortran code Access fast low level algorithms in a high level, interpreted language The language itself “It fits in your head” “It [Python] doesn’t get in my way” - Robert Kern Python is good at all the things other scientific programming languages are not good at (e.g. networking, string processing, OOP) Liberal BSD license: can use Python for commercial applications @wesmckinn () Data analysis with pandas 10/17/2011 15 / 22
- 16. Some exciting stuff in the last few years Cython “Augmented” Python language with type declarations, for generating compiled extensions C-like speedups with Python-like development time IPython: enhanced interactive Python interpreter The best research and software development env for Python An integrated parallel / distributed computing backend GUI console with inline plotting and a rich HTML notebook (more on this later) PyCUDA / PyOpenCL: GPU computing in Python Transformed Python overnight into one of the best languages for doing GPU computing @wesmckinn () Data analysis with pandas 10/17/2011 16 / 22
- 17. Where has Python historically been weak? Rich data structures for data analysis and statistics NumPy arrays, while powerful, feel distinctly “lower level” if you’re used to R’s data.frame pandas has filled this gap over the last 2 years Statistics libraries Nowhere near the depth of R’s CRAN repository statsmodels provides tested implementations a lot of standard regression and time series models Turns out that most financial data analysis requires only fairly elementary statistical models @wesmckinn () Data analysis with pandas 10/17/2011 17 / 22
- 18. pandas library Began building at AQR in 2008, open-sourced late 2009 Why R / MATLAB, while good for research / data analysis, are not suitable implementation languages for large-scale production systems (I personally don’t care for them for data analysis) Existing data structures for time series in R / MATLAB were too limited / not flexible enough my needs Core idea: indexed data structures capable of storing heterogeneous data Etymology: panel data structures @wesmckinn () Data analysis with pandas 10/17/2011 18 / 22
- 19. pandas in a nutshell A clean axis indexing design to support fast data alignment, lookups, hierarchical indexing, and more High-performance data structures Series/TimeSeries: 1D labeled vector DataFrame: 2D spreadsheet-like structure Panel: 3D labeled array, collection of DataFrames SQL-like functionality: GroupBy, joining/merging, etc. Missing data handling Time series functionality @wesmckinn () Data analysis with pandas 10/17/2011 19 / 22
- 20. pandas design philosophy “Think outside the matrix”: stop thinking about shape and start thinking about indexes Indexing and data alignment are essential Fault-tolerance: save you from common blunders caused by coding errors (specifically misaligned data) Lift the best features of other data analysis environments (R, MATLAB, Stata, etc.) and make them better, faster Performance and usability equally important @wesmckinn () Data analysis with pandas 10/17/2011 20 / 22
- 21. The pandas killer feature: indexing Each axis has an index Automatic alignment between differently-indexed objects: makes it nearly impossible to accidentally combine misaligned data Hierarchical indexing provides an intuitive way of structuring and working with higher-dimensional data Natural way of expressing “group by” and join-type operations Better integrated and more flexible indexing than anything available in R or MATLAB @wesmckinn () Data analysis with pandas 10/17/2011 21 / 22
- 22. Tutorial time To the IPython console! @wesmckinn () Data analysis with pandas 10/17/2011 22 / 22