Apache Arrow Flight: A New Gold Standard for Data Transport
2 likes2,354 views
This document discusses how structured data is often moved inefficiently between systems, causing waste. It introduces Apache Arrow, an open standard for in-memory data, and how Arrow can help make data movement more efficient. Systems like Snowflake and BigQuery are now using Arrow to help speed up query result fetching by enabling zero-copy data transfers and sharing file formats between query processing and storage.
Apache Arrow Flight: A New Gold Standard for Data Transport
- 2. • • • • •
- 3. Some Partners ● https://ursalabs.org ● Apache Arrow-powered Data Science Tools ● Funded by corporate partners ● Built in collaboration with RStudio
- 4. Systems that move structured data often cause significant waste
- 5. • • •
- 6. • • … •
- 8. Server 1 Server 2 Server 3 Client 1 Client 2
- 9. Scalable Blob Storage System 1 System 2 System 3
- 10. Executor Executor Executor Executor / Coordinator Client Result Set Result Set Result Set Result Set Result Set Result Set
- 11. Executor Executor Executor Executor Client Result Set Result Set Result Set
- 12. • • • • •
- 14. • • • • •
- 15. • • • • •
- 16. • • • • • •
- 17. • • • •
- 20. • metadata body
- 23. • • •
- 24. • • • • • •
- 25. • • •
- 26. Client Planner GetFlightInfo FlightInfo DoGet Data Nodes FlightData DoGet FlightData ...
- 27. • • • message SQLQuery { binary database_uri = 1; binary query = 2; } Commands.proto GetFlightInfo RPC type: CMD cmd: <serialized command>
- 28. Client DoGet Data Node FlightData Row Batch Row Batch Row Batch Row Batch Row Batch ... Data transported in a Protocol Buffer, but reads can be made zero-copy by writing a custom gRPC “deserializer”
- 29. Mainlining Databases: Supporting Fast Transactional Workloads on Universal Columnar Data File Formats Li, Butrovich, Ngom, Lim, Pavlo, McKinney https://arxiv.org/pdf/2004.14471.pdf
- 31. • •