Apache Arrow
0 likes113 views
Apache Arrow is a development platform designed for in-memory data processing that utilizes a columnar memory format for efficient analytic operations. It supports various programming languages and is open source, addressing the challenges of disparate memory formats across systems and reducing overhead through a shared data model. Arrow works alongside Parquet, which is optimized for disk storage, facilitating data serialization and efficient data access.
Apache Arrow
- 1. What Is Apache Arrow ? ● A development platform for in-memory data ● It has a columnar memory format ● It provides efficient analytic operations on modern hardware ● Used for in memory processing ● Cross language support ● Open source / Apache 2.0 license ● Supports zero-copy reads for lightning fast data access
- 2. Languages supported ● Arrow supports many languages ● C ● C++ ● C# ● Go ● Java ● JavaScript ● MATLAB ● Python ● R ● Ruby ● Rust
- 3. OS Community Support ● Many open source projects support Arrow ● Calcite ● Cassandra ● Drill ● Hadoop ● HBase ● Ibis ● Impala ● Kudu ● Pandas ● Parquet ● Phoenix ● Spark ● Storm
- 4. The problem Arrow tackles ● Each system has its own internal memory format ● 70-80% computation wasted – on serialization and de-serialization ● Similar functionality implemented in multiple projects ● Overheads for cross-system communication ● All systems utilize different memory formats
- 5. The problem Arrow tackles ● No shared in memory data model
- 6. Arrow solves this problem ● All systems utilize the same memory format – In memory – Columnar format – Optimized for modern CPUs and GPUs ● No overhead for cross-system communication ● Projects can share functionality
- 7. Arrow solves this problem ● Arrow shared data model
- 8. Arrow works with Parquet ● Arrow is an in memory format ● Parquet is designed for disk storage ● Arrow and Parquet are intended to be used together ● Parquet is a columnar file format ● Used for data serialization ● Parquet is a streaming format ● Data must be decoded from start-to-end ● Files are compressed and encoded ● Means smaller files on disk
- 9. Arrow Memory Buffer ● Arrow supports data adjacency for sequential access
- 10. Available Books ● See “Big Data Made Easy” – Apress Jan 2015 ● See “Mastering Apache Spark” – Packt Oct 2015 ● See “Complete Guide to Open Source Big Data Stack – “Apress Jan 2018” ● Find the author on Amazon – www.amazon.com/Michael-Frampton/e/B00NIQDOOM/ ● Connect on LinkedIn – www.linkedin.com/in/mike-frampton-38563020
- 11. Connect ● Feel free to connect on LinkedIn – www.linkedin.com/in/mike-frampton-38563020 ● See my open source blog at – open-source-systems.blogspot.com/ ● I am always interested in – New technology – Opportunities – Technology based issues – Big data integration