Apache Arrow: Leveling Up the Analytics Stack
2 likes1,457 views
This document discusses the development of Apache Arrow, an open source in-memory data format designed for efficient analytical data processing on modern hardware. It provides a brief history of big data and analytics technologies leading to the need for Arrow. Key points about Arrow include that it aims to eliminate data serialization, enable code sharing across languages, and has over 400 contributors representing 11 programming languages. Notable subcomponents include DataFusion, Gandiva, and Plasma; and development is supported by organizations like Ursa Labs.
Apache Arrow: Leveling Up the Analytics Stack
- 2. Leveling up the Analytics Stack Wes McKinney @wesmckinn
- 3. Apache ibis
- 4. The Need for Speed
- 7. A semi-revisionist history of Big Data
- 8. Cloudera’s Jeff Hammerbacher “Data first, ask questions later” Enable everyone to “party on the data” (paraphrasing!)
- 9. Decoupling Storage from Processing
- 10. Simplified MapReduce Arch Storage Step 1 Step 2 Step 3 ...
- 11. “Scalability! But at what COST?” McSherry, Isard, Murray 2015 https://www.usenix.org/system/files/conference/hotos15/hotos15-paper-mcsherry.pdf
- 14. Storage Feasibility Computational Feasibility Resource Utilization Interactivity Up the hierarchy of needs...
- 15. Rise of “End-to-end” Execution Engines Storage Step 1 Step 2 Step 3 ... INPUT OUTPUT
- 16. End-to-end engines: drawbacks • Example: SQL on Hadoop systems, Apache Spark, others • Serve some use cases well, others less well • Fall short in ML/AI domain
- 17. The Interoperability Conundrum Engine A Engine B Data Handoff
- 18. Some Hardware Trends • Manycore processor architectures • Much faster disk • Much faster networking • Beyond CPUs
- 19. “Why Modern CPUs Are Starving and What Can Be Done about It” Francesc Alted, IEEE 2010
- 20. Recognizing Serialization as an Enemy
- 21. Serialization Translation of data into a form that can be stored or transmitted, and reconstructed later
- 22. How to Eliminate Serialization “Serialized” and In-Memory Format must be the same (or nearly so)
- 23. A Collective Realization in 2015 Many open source developers had noted the absence of an in-memory standard for structured data analytics
- 24. ● Language-agnostic in-memory format for analytical query processing on modern hardware ● Low-overhead data sharing and transport ● A cross-language development platform to build Arrow-powered applications Mission
- 25. Why Column-oriented? • Reduce unnecessary IO • Increase memory throughput • Better parallelism • Leverage SIMD instructions
- 26. Apache Arrow “meta” goals • Forge collaborations between database systems and data science / ML / AI communities • Eliminate barriers to code sharing between application ecosystems and programming languages
- 27. Community over Code • ASF open governance model • ~400 unique contributors • 49 committers, 28 PMC members • 11 programming languages represented
- 28. Arrow Development in Practice • “Core” format and protocol implementations • “Batteries-included” standard libraries • Common build / test / package infrastructure and compatibility testing
- 30. Some Arrow Success Stories Apache
- 31. • gRPC-based framework for custom data services • High-speed network dataset transfer • Now available for C++, Java, Python Arrow Flight: Fast Data Services Development Partners
- 32. Flight key ideas • Zero-serialization • Bidirectional streaming transfers • Parallel transfers + horizontal scalability designed into the protocol • Reap benefits of Google’s work on gRPC
- 33. Flight use cases • Replacing slow database protocols like JDBC / ODBC • General network data movement • Retrofit legacy systems with fast Arrow IO
- 34. Notable Arrow subcomponents Rust DataFusion Arrow-native Rust query engine Gandiva LLVM analytical expression compiler Plasma Shared memory object store
- 35. Funding Arrow Development • Apache projects are technically communities of volunteers • Much development contributed by direct users of Arrow • Ursa Labs: not-for-profit group I founded in 2018 with initial support of RStudio and Two Sigma