DataFusion-and-Arrow_Supercharge-Your-Data-Analytical-Tool-with-a-Rusty-Query-Engine_iteblog.pdf

Supercharge Your Data
Analytical Tool with a Rusty
Query Engine
Andrew Lamb
Staff Engineer, InfluxData
Apache Arrow PMC
DataFusion and
Apache Arrow
Daniël Heres
Data Engineer, GoDataDriven
Apache Arrow PMC

Introduction
Your Speakers
Staff Engineer @ InﬂuxData
Previously
● Query Optimizer @ Vertica, Oracle
Database server, embedded Compilers
● Chief Architect + VP Engineering roles
at ML startups
Andrew
Data/ML Engineer @ GoDataDriven
Previously
• Data / ML Engineer @ bol.com
• Startups
2
Daniël

Recent Proliferation of Big Data systems
4
…

Recent Proliferation of Databases
5
DB

What is going on?
COTS → Totally Custom
7
IT FANG
“Buy and Operate”
● Buy software from
vendors
● Operate on your own
hardware, with
sysadmins
“Build and Operate”
● Write software for, and
operate all components
● Optimized for exact needs
✓
Current Trend
“Assemble and Operate”
● Assemble from open
source technologies
● Operate on resources
in a public cloud

Apache Arrow
Multi-language toolkit for Processing and Interchange
Founded in 2016
Apache Software Foundation
Low level / foundational technology to build fast and
interoperable analytic systems
Open standard, implementations in 12+ languages
Adopted widely in industry products and open
source projects
8

“DataFusion is an extensible query
execution framework, written in Rust,
that uses Apache Arrow as its
in-memory format.”
- DataFusion Website
DataFusion: A Query Engine
9

DataFusion: A Query Engine
SQL Query
SELECT status, COUNT(1)
FROM http_api_requests_total
WHERE path = '/api/v2/write'
GROUP BY status;
Data
Batches
DataFrame
ctx.read_table("http")?
.filter(...)?
.aggregate(..)?;
Data Batches
Catalog information:
tables, schemas, etc
10

Implementation timeline for a new
Database system
Client
API
In memory
storage
In-Memory
filter + aggregation
Durability /
persistence
Metadata Catalog +
Management
Query
Language
Parser
Optimized /
Compressed
storage
Execution on
Compressed
Data
Joins!
Additional Client
Languages
Outer
Joins
Subquery
support
More advanced
analytics
Cost
based
optimizer
Out of core
algorithms
Storage
Rearrangement
Heuristic
Query
Planner
Arithmetic
expressions
Date / time
Expressions
Concurrency
Control
Data Model /
Type System
Distributed query
execution
Resource
Management
“Lets Build
a Database”
🤔
“Ok now this
is pretty
good”
😐
“Look mom!
I have a
database!”
😃
Online
recovery
Window functions
11

LLVM-like Infrastructure for Databases
Inputs
Logical Plan Execution Plan
Plan Representations
(DataFlow Graphs)
Expression Eval
Optimizations /
Transformations
Optimizations /
Transformations
HashAggregate
Sort
…
Optimized Execution
Operators
(Arrow Based)
Join
Data
(Parquet, CSV, statistics, …) DataFusion
Query
(SQL, code, DataFrame, …)
Code
(UDF, UDA, etc)
Resources
(Cores, memory, etc)
13

DataFusion: Totally Customizable
Inputs
Logical Plan Execution Plan
Plan Representations
(DataFlow Graphs)
Expression Eval
Optimizations /
Transformations
Optimizations /
Transformations
HashAggregate
Sort
…
Optimized Execution
Operators
(Arrow Based)
Join
Data
(Parquet, CSV, statistics, …) DataFusion
Query
(SQL, code, DataFrame, …)
Code
(UDF, UDA, etc)
Resources
(Cores, memory, etc)
Extend ✅
Extend ✅
Extend ✅
Extend ✅
Extend ✅
Extend ✅
Extend ✅
Extend ✅
14

DataFusion Project Growth
15
5
.
0
.
0
6
.
0
.
0
7
.
0
.
0
8
.
0
.
0
Number
of
Unique
Contributors
Date

DataFusion Project Growth
16
https://star-history.com/#apache/arrow-datafusion&Date

DataFusion Milestones: Time to Mature
17
5+ year labor of love
Dec 2016 Initial
DataFusion Commit By
Andy Grove
Feb 2019 Donation to
Apache Arrow
Apr/May 2022:
Subqueries
Apr 2021 Ballista is
donated to Apache
Arrow
2020-2021: (hash) joins,
window functions,
performance &
parallelization, etc.
Mar 2018 Arrow in Rust
started, DataFusion
switches to Arrow
Nov 2021
DataFusion Contrib

Overview of
Apache Arrow
DataFusion
18
Daniël

From Query to Results
ExecutionPlan
LogicalPlan
Optimize Optimize
Execute!
Query / DataFrame
19

an example
20
select
count(*) num_visitors,
job_title
from
visitors
where
city = "San Francisco"
group by
job_title
1
2
3
4
5
6
7
8
9
10

datafusion package available via PyPI
21
visitors = ctx.table("visitors")
df = (
visitors.filter(col("city") == literal("San Francisco"))
.aggregate([col("job_title")], [f.count(literal(1))])
)
batches = df.collect() # collect results into memory (Arrow batches)
1
2
3
4
5
6
7
8
9
10

ExecutionPlan
Optimize Optimize
Execute!
Query / DataFrame LogicalPlan
22
Logical Plan represents the what

Initial Logical Plan
SQL is parsed, then translated into a initial Logical Plan.
23
Projection: #COUNT(UInt8(1)) AS num_visitors, #visitors.job_title
Aggregate: groupBy=[[#visitors.job_title]], aggr=[[COUNT(UInt8(1))]]
Filter: #visitors.city = Utf8("San Francisco")
TableScan: visitors projection=None
select count(*) num_visitors, job_title
from visitors
where city = "San Francisco"
group by job_title
visitors = ctx.table("visitors")
df = (
visitors.filter(col("city") == literal("San Francisco"))
.aggregate([col("job_title")], [f.count(literal(1))])
)
(Read plan from bottom to top)

Let's Optimize!
ExecutionPlan
Optimize
Execute!
Optimize
24
• Massively speed up execution times (10x, 100x, 1000x) by rewriting
queries to a equivalent, optimized version
• 14 built-in optimization passes in DataFusion, adding more each version
• Add custom optimization passes

Let's Optimize!
Projection Pushdown
Minimizing IO (especially useful for formats like Parquet), processing
25
num_visitors

Let's Optimize!
Projection Pushdown
26
TableScan: visitors projection=Some([0, 1])
projection_push_down
num_visitors

Let's Optimize!
Filter Pushdown
27
Projection: #COUNT(UInt8(1)) AS n, #visitors.job_title
num_visitors

Let's Optimize!
Filter Pushdown
28
filter_push_down
TableScan: visitors projection=Some([0, 1]), partial_filters=[#visitors.city = Utf8("San Francisco")]
num_visitors

Let's Create...
The ExecutionPlan
Optimize
Execute!
Query / DataFrame LogicalPlan ExecutionPlan
Optimize
29
The Execution Plan represents the where and how

The Initial Execution Plan
30
ProjectionExec: expr=[COUNT(UInt8(1))@1 as number_visitors, job_title@0 as job_title]
HashAggregateExec: mode=FinalPartitioned, gby=[job_title@0 as job_title], aggr=[COUNT(UInt8(1))]
RepartitionExec: partitioning=Hash([Column { name: "job_title", index: 0 }], 16)
HashAggregateExec: mode=Partial, gby=[job_title@0 as job_title], aggr=[COUNT(UInt8(1))]
FilterExec: city@1 = San Francisco
CsvExec: files=[./data/visitors.csv], has_header=true, limit=None, projection=[job_title, city]

And... Optimize!
Execute!
Optimize Optimize
ExecutionPlan
31

Optimize
CoalesceBatches: Avoiding small batch size
32

Optimize
CoalesceBatches: Avoiding small batch size
33
CoalesceBatchesExec: target_batch_size=4096
coalesce_batches

Optimize
Repartition: Introducing parallelism
34

Optimize
Repartition: Introducing parallelism
35
repartition
RepartitionExec: partitioning=RoundRobinBatch(16)

Getting results
Return record batches (or write results)
36
Optimize
ExecutionPlan Execute!
Optimize
Arrow Batches

DataFusion Features
● Mostly complete SQL implementation (aggregates, joins, window
functions, etc)
● DataFrame API (Python, Rust)
● High performance vectorized, native, safe, multi-threaded execution
● Common ﬁle formats: Parquet, CSV, JSON, Avro
● Highly extensible / customizable
● Large, growing community driving project forward
37

SQL Support
https://arrow.apache.org/datafusion/user-guide/sql/sql_sta
tus.html#supported-sql
Projection (SELECT), Filtering (WHERE), Ordering (ORDER BY), Aggregation
(GROUP BY)
Aggregation functions (COUNT, SUM, MIN, MAX, AVG, APPROX_PERCENTILE, etc)
Window functions (OVER .. ([ORDER BY ...] [PARTITION BY ..])
Set functions: UNION (ALL), INTERSECT (ALL), EXCEPT
Scalar functions: string, Date/time,... (basic)
Joins (INNER, LEFT, RIGHT, FULL OUTER, SEMI, ANTI)
Subqueries, Grouping Sets
38

39
Extensibility
Customize DataFusion to your needs
User Defined Functions
User Defined Aggregates
User Defined Optimizer passes
User Defined LogicalPlan nodes
User Defined ExecutionPlan nodes
User Defined TableProvider
User Defined FileFormat
User Defined ObjectStore

Systems Powered
by DataFusion
40
Andrew

FLOCK
https://github.com/flock-lab/flock
● Overview:
○ Low-Cost Streaming Query Engine on FaaS Platforms
○ Project from UMD Database Group, runs streaming queries on AWS Lambda (x86
and arm64/graviton2).
● Use of DataFusion
○ SQL API:
○ DataFrame API: To build plans
○ Optimized native plan execution
41

ROAPI
https://roapi.github.io/
● Overview:
○ read-only APIs for static datasets without code
○ columnq-cli: run sql queries against CSV ﬁles
○ SQL API:
○ DataFrame API: (to build plans for GraphQL)
○ File formats: CSV, JSON, Parquet, Avro
42

VegaFusion
https://vegafusion.io/
● Overview:
○ Accelerates execution of (interactive) data
visualizations
○ Compiles Vega data transforms into
DataFusion query plans.
● Use of DataFusion:
○ DataFrame API: To build plans
○ UDFs: to implement some Vega expressions
43

Cube.js / Cube Store
https://cube.dev/
● Overview:
○ Headless Business Intelligence
○ cubestore pre-aggregation storage layer
● Use of DataFusion (fork)
○ SQL API (with custom extensions)
○ Custom Logical and Physical Operators
○ UDFs: custom functions
44

InfluxDB IOx
https://github.com/influxdata/influxdb_iox
● Overview:
○ In-memory columnar store using object storage, future
core of InfluxDB; support SQL, InfluxQL, and Flux
○ Query and data reorganization built with DataFusion
○ Table Provider: Custom data sources
○ SQL API
○ PlanBuilder API: Plans for custom query language
○ UD Logical and Execution Plans
○ UDFs: to implement the precise semantics of influxRPC
45

Coralogix
https://coralogix.com/
● Overview:
○ Stateful streaming analytics with machine learning enables teams to monitor and
visualize observability data in real-time before indexing
○ Table Provider: custom data source
○ User Deﬁned Logical and Execution Plans: to implement a custom query language
○ User Deﬁned ObjectStore: for queries over data in object storage
○ UDFs: for working with semi-structured data
46

blaze-rs
https://github.com/blaze-init/blaze
● Overview:
○ High performance, low-cost native execution layer for Spark: execute the physical
operators with Rust
○ Translates Spark Exec nodes into DataFusion Execution Plans
○ HDFS Object Store Extension
47

Ballista Distributed Compute
https://github.com/apache/arrow-ballista
● Overview:
○ Spark-like distributed Query Engine (part of Arrow Project)
○ Adds distributed execution to DataFusion plans
○ SQL API
○ DataFrame API
48

Future Directions
● Embeddability
○ More regular releases to crates.io, more modularity
● Broader SQL features
○ Subqueries, more date/time functions, struct / array types
● Improved Performance
○ Query directly from Object Storage
○ More state of the art tech: JIT, NUMA aware scheduling, hybrid row/columnar exec
● Ecosystem integration
○ FlightSQL, Substrait.io
○ Databases
● GPU support
50

Come Join Us
We ❤ Our Contributors
● Contributions at all levels are encouraged and welcomed.
● Learn Rust!
● Learn Database Internals!
● Have a great time with a welcoming community!
More details:
https://arrow.apache.org/datafusion/community/communication.html
51

52
Thank you
arrow.apache.org/datafusion
github.com/apache/arrow-datafusion
Andrew Lamb
Staff Engineer, InfluxData
Apache Arrow PMC
Daniël Heres
Data Engineer, GoDataDriven
Apache Arrow PMC

Thank You!
arrow.apache.org/datafusion
github.com/apache/arrow-datafusion
Andrew Lamb
Staff Engineer,
InﬂuxData
Apache Arrow
PMC
Daniël Heres
Data Engineer,
GoDataDriven
Apache Arrow
PMC

DataFusion / Arrow / Parquet
Parquet
Arrow
sqlparser-rs
DataFusion

A Virtuous Cycle
Increased Use of Drives Increased Contribution
57
Increased use of
open source systems
Increased capacity
for maintenance and
contribution
DataFusion, and Apache Arrow are key open source technologies
for building interoperable open source systems

delta-rs
https://github.com/delta-io/delta-rs
● Overview:
○ Native Delta Lake implementation in Rust
○ Table Provider API: allows other DataFusion users
to read from Delta tables
58
DISCLAIMER: Not yet cleared / verified with project team

Cloudfuse Buzz
https://github.com/cloudfuse-io/buzz-rust
● Serverless cloud-based query engine
○ map using cloud functions (AWS Lambda)
○ aggregate using containers (AWS Fargate)
● Project (expected to be) continued from june
59

dask-sql
https://github.com/dask-contrib/dask-sql
● Overview:
○ TBD
●
○ WIP https://github.com/dask-contrib/dask-sql/issues/474
60

Apache Arrow Analytics Toolkit
Where does DataFusion fit?
61
Parquet (“Disk”) Arrow (“Memory”)
Compute
Kernels
Arrow Flight
Arrow FlightSQL DataFusion
Data Formats
Low Level
Calculations +
Interchange
Runtime
Subsystems
IPC
C ABI
Analytics /
Database
Systems
C++ Query Engine
Analytic systems built using some of this stack
Native
Implementations
Language
Bindings

Query Engines
What is it and why do you need one?
1. Add SQL or DataFrame interface to your application’s data
2. Implement a custom query language / DSL
3. Implement a new data analytic system
4. Implement a new database system (natch)
Maps Desired Computations: SQL and DataFrame (ala Pandas)
To Efficient Calculation: projection pushdown, filter pushdown, joins,
expression simplification, parallelization, etc
62

datafusion-python
https://github.com/datafusion-contrib/datafusion-python
● Overview:
○ Python dataframe library (modeled after pyspark)
○ SQL API
○ DataFrame API
63

Common Themes
Come for the performance, stay for the features (?)
Native execution
Native (non JVM) of Spark/Spark like behavior
SQL interface
Projects are leveraging properties of Rustlang
SQL / DataFrame API
64

Better, Faster, Cheaper
The DataFusion Query Engine is part of the commoditization of advanced
analytic database technologies
Transform analytic systems over the next decade
65
Better Faster Cheaper

Andrew’s Notes
Proposal: Data + AI Summit talk
Desired Takeaways:
1. If you need a query engine (in Rust?), you should use DataFusion
Thesis: DataFusion is part of a larger trend (spearheaded by Apache Arrow) in the
commoditization of analytic database technologies, which will lead to many faster / cheaper /
better analytic systems over the next decade
Other decks for inspiration:
DataFusion: An Embeddable Query Engine Written in Rust
xA Rusty Introduction to Apache Arrow and how it Applies to a Time Series Database
2021-04-20: Apache Arrow and its Impact on the Database industry.pptx 66

67
Instructions: Read me!
Getting started with our slide template
When using this template, create your new slides at the very top of the
slide order, above this slide. Explore the advice and example slides below to
ﬁnd useful layouts and graphics to pull into your design. When your slide
deck is complete, delete this slide and every slide below it.

There are great baseline
slides in this template, but
it may not have everything
you need. Don’t be afraid
to craft your own layouts!
Just pay attention to the
font and grid guidelines,
and take advantage of
starter shapes.
Get creative
Use text hierarchy to
create order and keep your
content scannable. No
walls of text! Try to keep
headlines short.
Make it scannable
Presentation best practices
Less is more
Don’t try to cram
everything onto a limited
number of slides. More
slides with less text per
slide is easier to digest.
Clarity over density
68

69
Font Guidance
Font selection
All text in our slide decks should use
one of two available event brand
fonts: DM Sans or DM Mono.
If you do not see these fonts in your font
selection menu, they can be added by
selecting “More fonts” and searching for
“dm.” Click on DM Sans and DM Mono, then
hit OK.
1
2

70
Font Guidance (Cont.)
Font sizing
Using consistent type sizing is a
good way to help your slides feel
uniform. When selecting type sizes,
try to stick to multiple of 8, with the
exceptions of 12 and 20 as
in-betweens.
DATA+AI Summit
12 DATA+AI Summit
DATA+AI Summit
DATA+AI Summit
DATA+AI Summit
DATA+AI Summit
DATA+AI Summi
DATA+AI Summ
16
20
24
32
40
56
64

71
Grid Guidance
Keep it orderly
Your presentation template has a 12
column grid to help you organize the
elements on your slides. When laying
out objects, consider using the grid
to help.
Toggle the grid visibility by navigating
to View > Guides > Show Guides.

72
Color Guidance
Keep it on brand
When customizing charts or adding other visual
elements, do your best to stay within our deﬁned
event color palette. This will ensure that all your
content looks great together and doesn’t clash with
the slide template design.
Always use black text when placing content over a
colored background. The only exception is when
using a black background. Any color text is
acceptable on black.
10121E 00B6E0 85DDB5 F16047
EDEEF1 8FDDEF AFE9CF F3A89B

Eighteen colorful title slide
options with varying shapes
74
Add your Name
Add your title, company
Choose Your
Title Slide

75
Add your Name
Choose Your
Title Slide

76
Add your Name
Choose Your
Title Slide

Basic Content Slide
Your all-purpose zone
Use this slide as a starting point for crafting your own layouts, or for simple
text slides.
77

Activate Dark Mode
Mix in black slides to add contrast and variety
Or make your whole presentation dark!
78

Insert your charts or images
Take advantage of the content panels
79
If you want to insert a gif or
other image from the web,
simply navigate to
Insert > Image > by URL.
Crop and resize your image to ﬁt
within content panels, if you’re
feeling fancy.
Insert Image by URL

“With just a few adjustments to text size and
alignment, you can use the basic content slide for
other types of content such as quotes.”
80
Andrew Pons
Slide Designer

81
Column A Column B Column C Column D Column E Column F
Row A You can create
simple tables to
help organize
information.
Row B
Row C
Row D
Row E
Row F
Row G
Row H

Timeline Style One
82
Your subtitle here
Timeline Item Timeline Item Timeline Item Timeline Item
Timeline Item Timeline Item Timeline Item

Timeline Style Two
83
Your subtitle here
Your gantt chart item
Q1 Q2 Q3 Q4

Single Column
Content Tile
Use this panel for content, images, diagrams, or whatever else you want to include. You can
use the line tool to divide this panel into multiple sections if you want.
Multi-purpose
84

Two Column
Content Tile
Use these slides for comparing two topics
or just for splitting your content into
multiple pieces.
Multi-purpose
85
Use these slides for comparing two topics
or just for splitting your content into
multiple pieces.
Multi-purpose

Column 3
Column 2
Three Column
Column 1
86

Four Column
Column 1
87
Column 4
Column 3
Column 2

Half Panel
Right aligned
88
This space is great for supporting text that
compliments whatever content is inside
the panel.
Open Content
This space can be for text content, images,
diagrams, or whatever you need
Panel Content

Half Panel
Left aligned
89
This space can be for text content, images,
diagrams, or whatever you need
Panel Content
This space is great for supporting text that
compliments whatever content is inside
the panel.
Open Content

⅔ Panel
Right aligned
90
This space can be for text content, images, diagrams, or
whatever you need
Panel Content
This space is great for
supporting text that
compliments whatever
content is inside the panel.
Open Content

⅔ Panel
Left aligned
91
This space can be for text content, images, diagrams, or
whatever you need
Panel Content
This space is great for
supporting text that
compliments whatever
content is inside the panel.
Open Content

Code Display
Paste snippets
92
select
count (*),
age
from
visitors
where location="SanFrancisco"
group by job_title
1
2
3
4
5
6
7
8
9
10

Use breaker
slides to divide
your deck into
sections
93

94
Use breaker
slides to divide
your deck into
sections

Starter Shapes
Copy and paste these wherever you need them
95
Floating panel for text or graphics
❤ ⚠ ★
✓ ✗
Medium pill label
SMALL PILL LABEL
✓ ✗
Medium pill label
SMALL PILL LABEL

Logos
Partners and cloud platforms
96

DataFusion-and-Arrow_Supercharge-Your-Data-Analytical-Tool-with-a-Rusty-Query-Engine_iteblog.pdf

More Related Content

What's hot (20)

Similar to DataFusion-and-Arrow_Supercharge-Your-Data-Analytical-Tool-with-a-Rusty-Query-Engine_iteblog.pdf (20)

Recently uploaded (20)

DataFusion-and-Arrow_Supercharge-Your-Data-Analytical-Tool-with-a-Rusty-Query-Engine_iteblog.pdf