PyData Boston 2013
8 likes3,940 views
Continuum Analytics focuses on developing open-source software for scientific computing, data processing, and analysis, with a strong emphasis on Python and related tools like Conda and Anaconda. They aim to facilitate easier package management and support high-performance computing through projects such as Numba and Blaze. The initiative also includes the creation of collaborative platforms like Wakari, which enhance data analysis capabilities for large datasets and promote interactive visualizations.
![Numba
@jit('void(f8[:,:],f8[:,:],f8[:,:])')
def filter(image, filt, output):
M, N = image.shape
m, n = filt.shape
for i in range(m//2, M-m//2):
for j in range(n//2, N-n//2):
result = 0.0
for k in range(m):
for l in range(n):
result += image[i+k-m//2,j+l-n//2]*filt[k, l]
output[i,j] = result
~1500x speed-up](https://image.slidesharecdn.com/pydataboston2013-130728164901-phpapp02/85/PyData-Boston-2013-22-320.jpg)
![Laplace Example
@jit('void(double[:,:], double, double)')
def numba_update(u, dx2, dy2):
nx, ny = u.shape
for i in xrange(1,nx-1):
for j in xrange(1, ny-1):
u[i,j] = ((u[i+1,j] + u[i-1,j]) * dy2 +
(u[i,j+1] + u[i,j-1]) * dx2) / (2*(dx2+dy2))
Adapted from http://www.scipy.org/PerformancePython
originally by Prabhu Ramachandran
@jit('void(double[:,:], double, double)')
def numbavec_update(u, dx2, dy2):
u[1:-1,1:-1] = ((u[2:,1:-1]+u[:-2,1:-1])*dy2 +
(u[1:-1,2:] + u[1:-1,:-2])*dx2) / (2*(dx2+dy2))](https://image.slidesharecdn.com/pydataboston2013-130728164901-phpapp02/85/PyData-Boston-2013-24-320.jpg)
PyData Boston 2013
- 1. Continuum Analytics: What we are doing and why Travis Oliphant, PhD Continuum Analytics, Inc
- 2. This talk will be about (mostly) the free and / or open source software we are building. Enterprise Python Scientific Computing Data Processing Data Analysis Visualisation Scalable Computing • Products • Training • Support • Consulting
- 3. Began operations in January of 2012 with 5 people with big dreams Python
- 4. We are big backers of NumFOCUS and organizers of PyData Spyder
- 5. Our Team Scientists Developers +5 contractors +4 interns +1 Part-time Business
- 7. NumPy and SciPy are quite successful Thanks to large and diverse community around it (Matplotlib, IPython, SymPy, Pandas, etc.) I estimate 1.5 million to 2 million users Only incremental improvements possible with these projects at this point. Thus, we needed to start new projects...
- 8. Related Open Source Projects Blaze: High-performance Python library for modern vector computing, distributed and streaming data Numba:Vectorizing Python compiler for multicore and GPU, using LLVM Bokeh: Interactive, grammar-based visualization system for large datasets Common Thread: High-level, expressive language for domain experts; innovative compilers & runtimes for efficient, powerful data transformation
- 9. Conda and Anaconda • Cross-platform package management • Multiple environments allows you to have multiple versions of packages installed in system • Easy app-deployment • Taming open-source Free for all users Enterprise support available!
- 10. Why Conda? • Linux users stopped complaining about Python deployment • I made major mistakes in management of NumPy/SciPy: • too much in SciPy (SciPy as distribution) --- scikits model is much better (tighter libraries developed by smaller teams) • gave in to community desire for binary ABI compatibility in NumPy motivated by difficulty of reproducing Python install • Need for a cross-platform way to install major Python extensions (many with dependencies on large C or C++ libraries • Python can’t be ubiquitous if people struggle to just get it and then manage it.
- 12. What is Conda • Full package management (like yum or apt-get) but cross-platform • Control over environments (using hard-link farms) --- better than virtual-env. virtualenv today is like distutils and setuptools of several years ago (great at first but will end up hating it) • Architected to be able to manage any packages (R, Scala, Clojure, Haskell, Ruby, JS) • SAT solver to manage dependencies • User-definable repositories
- 13. New Features and Binstar • Build command from recipe --- many recipes here: https:// github.com/ContinuumIO/conda-recipes • Upload recipes to Binstar (last mile in binary package hosting and deployment for any language). • “binstar in beta” is the beta code • Personal conda repositories --- https://conda.binstar.org/ travis • Free Continuum Anaconda repo will be on binstar.org • Private packages and behind-the-firewall satellites available • *Free build queue on Linux (Mac and Windows coming soon) for hosted conda recipes
- 14. Demo create Python 3 environment with IPython and scipy create new recipe from PyPI (yunomi)
- 15. Packaging and Distribution Solved • conda and binstar solve most of the problems that we have seen people encounter in managing Python installations (especially in large-scale institutions). • they are supported solutions that can remove the technology pain of managing Python • some problems, though, are people
- 16. Anaconda (open) Free enterprise-ready Python distribution of open- source tools for large-scale data processing, predictive analytics, and scientific computing
- 17. Anaconda Add-Ons (paid-for) •Revolutionary Python to GPU compiler •Extends Numba to take a subset of Python to the GPU (program CUDA in Python) •CUDA FFT / BLAS interfaces Fast, memory-efficient Python interface for SQL databases, NoSQL stores,Amazon S3, and large data files. NumPy, SciPy, scikit-learn, NumExpr compiled against Intel’s Math Kernel Library (MKL)
- 18. Launcher
- 19. Why Numba? •Python is too slow for loops •Most people are not learning C/C++/Fortran today •Cython is an improvment (but still verbose and needs C-compiler) •NVIDIA using LLVM for the GPU •Many people working with large typed-containers (NumPy arrays) •We want to take high-level, tarray-oriented expressions and compile it to fast code
- 20. NumPy + Mamba = Numba LLVM Library Intel Nvidia AppleAMD OpenCLISPC CUDA CLANGOpenMP LLVMPY Python Function Machine Code ARM
- 21. Example Numba
- 22. Numba @jit('void(f8[:,:],f8[:,:],f8[:,:])') def filter(image, filt, output): M, N = image.shape m, n = filt.shape for i in range(m//2, M-m//2): for j in range(n//2, N-n//2): result = 0.0 for k in range(m): for l in range(n): result += image[i+k-m//2,j+l-n//2]*filt[k, l] output[i,j] = result ~1500x speed-up
- 23. Numba changes the game! LLVM IR x86 C++ ARM PTX C Fortran Python Numba turns (a subset of) Python into a “compiled language” as fast as C (but much more flexible). You don’t have to reach for C/C++
- 24. Laplace Example @jit('void(double[:,:], double, double)') def numba_update(u, dx2, dy2): nx, ny = u.shape for i in xrange(1,nx-1): for j in xrange(1, ny-1): u[i,j] = ((u[i+1,j] + u[i-1,j]) * dy2 + (u[i,j+1] + u[i,j-1]) * dx2) / (2*(dx2+dy2)) Adapted from http://www.scipy.org/PerformancePython originally by Prabhu Ramachandran @jit('void(double[:,:], double, double)') def numbavec_update(u, dx2, dy2): u[1:-1,1:-1] = ((u[2:,1:-1]+u[:-2,1:-1])*dy2 + (u[1:-1,2:] + u[1:-1,:-2])*dx2) / (2*(dx2+dy2))
- 25. Results of Laplace example Version Time Speed Up NumPy 3.19 1.0 Numba 2.32 1.38 Vect. Numba 2.33 1.37 Cython 2.38 1.34 Weave 2.47 1.29 Numexpr 2.62 1.22 Fortran Loops 2.30 1.39 Vect. Fortran 1.50 2.13 https://github.com/teoliphant/speed.git
- 26. LLVMPy worth looking at LLVM (via LLVMPy) has done much heavy lifting LLVMPy = Compilers for everybody
- 27. What is wrong with NumPy? • Dtype system is difficult to extend • Many Dtypes needed (missing data, enums, variable length strings) • Immediate mode creates huge temporaries (spawning Numexpr) • “Almost” an in-memory data-base comparable to SQL- lite (missing indexes) • Integration with sparse arrays • Standard structure of arrays representation... • Missing Multi-methods • Optimization / Minimal support for multi-core / GPU
- 28. Now What? After watching NumPy and SciPy get used all over Wall Street and by many scientists / engineers in industry --- what would we do differently?
- 29. New Project Blaze NumPy Out of Core, Distributed and Optimized NumPy
- 30. Blaze Array or Table Data Descriptor Data Buffer Index Operation NumPy BLZ Persistent Format RDBMS CSVData Stream
- 31. Blaze Deferred Arrays +" A" *" B" C" A + B*C • Symbolic objects which build a graph • Represents deferred computation Usually what you have when you have a Blaze Array
- 32. DataShape Type System • A data description language • A super-set of NumPy’s dtype • Provides more flexibility Shape DType DataShape
- 33. Blaze Database GPU Node Array Server NFS Array Server Array Server Blaze Client Synthesized Array/Table view array+sql:// array:// file:// array:// Python REPL, Scripts Viz Data Server C, C++, FORTRAN JVM languages
- 34. Progress • Basic calculations work out-of-core (via Numba and LLVM) • Hard dependency on dynd and dynd-python (a dynamic C++-only multi-dimensional library like NumPy but with many improvements) • Persistent arrays from BLZ • Basic array-server functionality for layering over CSV files • 0.2 release in 1-2 weeks. 0.3 within a month after that (first usable release)
- 35. DARPA providing help DARPA-BAA-12-38: XDATA TA-1: Scalable analytics and data processing technology TA-2: Visual user interface technology
- 36. Bokeh Plotting Library • Interactive graphics for the web • Designed for large datasets • Designed for streaming data • Native interface in Python • Fast JavaScript component • DARPA funded • v0.1 release imminent
- 37. Reasons for Bokeh 1. Plotting must happen near the data too 2. Quick iteration is essential => interactive visualization 3. Interactive visualization on remote-data => use the browser 4. Almost all web plotting libraries are either: 1. Designed for javascript programmers 2. Designed to output static graphs 5. We designed Bokeh to be dynamic graphing in the web for Python programmers 6. Will include “Abstract” or “synthetic” rendering (working on Hadoop and Spark compatibility)
- 38. Wakari • Browser-based data analysis and visualization platform • Wordpress /YouTube / Github for data analysis • Full Linux environment with Anaconda Python • Can be installed on internal clusters & servers
- 39. Why Wakari? • Data is too big to fit on your desktop • You need compute power but don’t have easy access to a large cluster (cloud is sitting there with lots of power) • Configuration of software on a new system stinks (especially a cluster). • Collaborative Data Analytics --- you want to build a complex technical workflow and then share it with others easily (without requiring they do painful configuration to see your results) • IPython Notebook is awesome --- let’s share it (but we also need the dependencies and data).
- 40. Wakari • Free account has 512 MB RAM / 10 GB disk and shared multi-core CPU • Easily spin-up map-reduce (Disco and Hadoop clusters) • Use IPython Parallel on many-nodes in the cloud • Develop GUI apps (possibly in Anaconda) and publish them easily to Wakari (based on full power of scientific python --- complex technical workflows (IPython notebook for now)
- 42. Continuum Data Explorer (CDX) • Open Source • Goal is interactivity • Combination of IPython REPL, Bokeh, and tables • Tight integration between GUI elements and REPL • Current features - Namespace viewer (mapped to IPython namespace) - DataTable widget with group-by, computed columns, advanced- filters - Interactive Plots connected to tables
- 43. CDX
- 44. Conclusion Projects circle around giving tools to experts (occasional programmers or domain experts) to enable them to move their expertise to the data to get insights --- keep data where it is and move high-level but performant code) Join us or ask how we can help you!