SlideShare a Scribd company logo

Apache Calcite Tutorial - BOSS 21

S
Stamatis Zampetakis

The document is a tutorial from the BOSS'21 conference led by Stamatis Zampetakis and Julian Hyde, detailing the setup of a coding environment using Apache Calcite. It includes instructions for cloning a GitHub repository, compiling the project, and several coding modules covering components like hybrid planning, custom operators, and the Volcano planner internals. The tutorial also discusses exercises and coding challenges aimed at enhancing skills in SQL query processing and optimization using the Calcite framework.

Engineering
Related topics:
Query Optimization
1 of 83
Downloaded 13 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Most read
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
Most read
55
56
57
58
59
60
61
62
63
64
Most read
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
Tutorial @BOSS’21 Copenhagen Stamatis Zampetakis, Julian Hyde • August 16, 2021
Apache Calcite Tutorial - BOSS 21
Tutorial @BOSS’21 Copenhagen Stamatis Zampetakis, Julian Hyde • August 16, 2021 # Follow these steps to set up your environment # (The first time, it may take ~3 minutes to download dependencies.) git clone --branch boss21 https://github.com/zabetak/calcite-tutorial.git java -version # need Java 8 or higher cd calcite-tutorial ./mvnw package -DskipTests
Setup Environment Requirements 1. Git 2. JDK version ≥ 1.8 Steps 1. Clone GitHub repository 2. Load to IDE (preferred IntelliJ) a. Click Open b. Navigate to calcite-tutorial c. Select pom.xml file d. Choose “Open as Project” 3. Compile the project git clone --branch boss21 https://github.com/zabetak/calcite-tutorial.git java -version cd calcite-tutorial ./mvnw package -DskipTests
Julian Hyde @julianhyde Senior Staff Engineer @ Google / Looker Creator of Apache Calcite PMC member of Apache Arrow, Drill, Eagle, Incubator and Kylin Stamatis Zampetakis @szampetak Senior Software Engineer @ Cloudera, Hive query optimizer team PMC member of Apache Calcite; Hive committer PhD in Data Management, INRIA & Paris-Sud University About us
Outline 1. Introduction 2. CSV Adapter Demo 3. Coding module I: Main components 4. Coding module I Exercises (Homework) 5. Hybrid planning 6. Coding module II: Custom operators/rules (Homework) 7. Volcano Planner internals 8. Dialects 9. Materialized views 10. Working with spatial data 11. Research using Apache Calcite optional optional optional optional
1. Calcite introduction
1. Retrieve books and authors 2. Display image, title, price of the book along with firstname & lastname of the author 3. Sort the books based on their id (price or something else) 4. Show results in groups of five Motivation: Data views
What, where, how data are stored? author 0..1 AUTHOR id int fname string lname string birth date BOOK id int title string price decimal year int
XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN FILESYSTEM What, where, how data are stored? author 0..1 AUTHOR id int fname string lname string birth date BOOK id int title string price decimal year int
BIN What, where, how data are stored? XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN FILESYSTEM author 0..1 AUTHOR id int fname string lname string birth date BOOK id int title string price decimal year int
What, where, how data are stored? 360+ DBMS XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN FILESYSTEM author 0..1 AUTHOR id int fname string lname string birth date BOOK id int title string price decimal year int
★ Open-source search engine ★ Java library ★ Powerful indexing & search features ★ Spell checking, hit highlighting ★ Advanced analysis/tokenization capabilities ★ ACID transactions ★ Ultra compact memory/disk format Apache Lucene
How to query the data? SELECT b.id, b.title, b.year, a.fname, a.lname FROM Book b LEFT OUTER JOIN Author a ON b.author=a.id ORDER BY b.id LIMIT 5 1. Retrieve books and authors 2. Display image, title, price of the book along with firstname & lastname of the author 3. Sort the books based on their id (price or something else) 4. Show results in groups of five
Query processor architecture Query Relational Algebra Query planner CatalogReader Schema Metadata (Cost, Statistics) Execution engine Results Parser Rules Relational Algebra
Apache Calcite SQL query SqlToRelConverter RelNode RelOptPlanner CatalogReader Schema RelMetadata Provider RelBuilder RelRunner Results API calls SqlParser SqlValidator RelNode SqlNode SqlNode RelRule
Apache Calcite SQL query SqlToRelConverter RelNode RelOptPlanner CatalogReader Schema RelRule RelBuilder RelRunner Results API calls SqlParser SqlValidator RelNode SqlNode SqlNode Dev & extensions RelMetadata Provider
2. CSV Adapter Demo
Adapter Implement SchemaFactory interface Connect to a data source using parameters Extract schema - return a list of tables Push down processing to the data source: ● A set of planner rules ● Calling convention (optional) ● Query model & query generator (optional) { "schemas": [ { "name": "BOOKSTORE", "type": "custom", "factory": "org.apache.calcite.adapter.file.FileSchemaFactory", "operand": { "directory": "bookstore" } } ] ]
3. Coding module I: Main components
Setup schema & type factory <<Interface>> Schema +getTable(name: Table): Table <<Interface>> Table +getRowType(typeFactory: RelDataTypeFactory): RelDataType <<Interface>> RelDataTypeFactory +createJavaType(clazz: Class): RelDataType +createSqlType(typeName: SqlTypeName): RelDataType * tables
Apache Calcite Tutorial - BOSS 21
SELECT b.id, b.title, b.year, a.fname, a.lname FROM Book AS b LEFT OUTER JOIN Author a ON b.author = a.id WHERE b.year > 1830 ORDER BY b.id LIMIT 5 Query to Abstract Syntax Tree (AST) SQL query SqlParser SqlNode Numeric Literal 5 Identifier b.id NodeList Select OrderBy Join BasicCall As Operator Identifier author Identifier a Literal LEFT Literal ON Identifier b.author Identifier a.id Binary Operator = As Operator Identifier book Identifier b BasicCall BasicCal l Identifier b.year Numeric Literal 1830 Binary Operator > BasicCall Identifier b.id NodeList Identifier b.title Identifier b.year Identifier a.fname Identifier a.lname
AST to logical plan SqlNode Join Select b.id, b.title, b.year, a.fname, a.lname BasicCall b.author = a.id BasicCall book AS b BasicCall author AS a Literal LEFT Literal ON LogicalJoin [$3==$4,type=left] LogicalTableScan [Book] LogicalTableScan [Author] LogicalFilter [$2>1830] LogicalProject [ID=$0,TITLE=$1,YEAR=$2, FNAME=$5,LNAME=$6] LogicalSort [sort0=$0,dir=ASC,fetch=5] SqlValidator SqlToRelConverter SqlNode RelNode OrderBy b.id FETCH 5 BasicCall b.year > 1830
Logical to physical plan RelRule EnumerableSortRule EnumerableProjectRule EnumerableFilterRule EnumerableJoinRule EnumerableTableScanRule RelOptPlanner RelNode RelNode LogicalJoin [$3==$4,type=left] LogicalTableScan [Book] LogicalTableScan [Author] LogicalFilter [$2>1830] LogicalProject [ID=$0,TITLE=$1,YEAR=$2, FNAME=$5,LNAME=$6] LogicalSort [sort0=$0,dir=ASC,fetch=5] EnumerableJoin [$3==$4,type=left] EnumerableTableScan [Book] EnumerableTableScan [Author] EnumerableFilter [$2>1830] EnumerableProject [ID=$0,TITLE=$1,YEAR=$2, FNAME=$5,LNAME=$6] EnumerableSort [sort0=$0,dir=ASC,fetch=5]
Physical to Executable plan EnumerableInterpretable Java code RelNode EnumerableJoin [$3==$4,type=left] EnumerableTableScan [Book] EnumerableTableScan [Author] EnumerableFilter [$2>1830] EnumerableProject [ID=$0,TITLE=$1,YEAR=$2, FNAME=$5,LNAME=$6] EnumerableSort [sort0=$0,dir=ASC,fetch=5]
4. Coding module I: Exercises (Homework)
Exercise I: Execute more SQL queries Include GROUP BY and other types of clauses: SELECT o.o_custkey, COUNT(*) FROM orders AS o GROUP BY o.o_custkey
Exercise I: Execute more SQL queries Include GROUP BY and other types of clauses: ● Missing rule to convert LogicalAggregate to EnumerableAggregate ● Add EnumerableRules.ENUMERABLE_AGGREGATE_RULE to the planner SELECT o.o_custkey, COUNT(*) FROM orders AS o GROUP BY o.o_custkey
Push filter below the join: Exercise II: Improve performance by applying more optimization rules SELECT c.c_name, o.o_orderkey, o.o_orderdate FROM customer AS c INNER JOIN orders AS o ON c.c_custkey = o.o_custkey WHERE c.c_custkey < 3 ORDER BY c.c_name, o.o_orderkey
Push filter below the join: 1. Add rule CoreRules.FILTER_INTO_JOIN to the planner 2. Compare plans before and after (or logical and physical) 3. Check cost estimates by using SqlExplainLevel.ALL_ATTRIBUTES Exercise II: Improve performance by applying more optimization rules SELECT c.c_name, o.o_orderkey, o.o_orderdate FROM customer AS c INNER JOIN orders AS o ON c.c_custkey = o.o_custkey WHERE c.c_custkey < 3 ORDER BY c.c_name, o.o_orderkey
Exercise II: Improve performance by applying more optimization rules LogicalJoin [$0=$9,type=inner] LogicalTableScan [CUSTOMER] LogicalTableScan [ORDERS] LogicalFilter [$0<3] LogicalProject [C_NAME=$1,O_ORDERKEY=$8, O_ORDERDATE=$12] LogicalSort [sort0=$0,dir0=ASC,sort1=$1,dir1=ASC] RelRule FilterIntoJoinRule LogicalJoin [$0=$9,type=inner] LogicalTableScan [CUSTOMER] LogicalTableScan [ORDERS] LogicalFilter [$0<3] LogicalProject [C_NAME=$1,O_ORDERKEY=$8, O_ORDERDATE=$12] LogicalSort [sort0=$0,dir0=ASC,sort1=$1,dir1=ASC]
Open LuceneBuilderProcessor.java and complete TODOs Exercise III: Use RelBuilder API to construct the logical plan SELECT o.o_custkey, COUNT(*) FROM orders AS o GROUP BY o.o_custkey SELECT o.o_custkey, COUNT(*) FROM orders AS o WHERE o.o_totalprice > 220388.06 GROUP BY o.o_custkey Q1: Q2:
Exercise III: Use RelBuilder API to construct the logical plan builder .scan("orders") .filter( builder.call( SqlStdOperatorTable.GREATER_THAN, builder.field("o_totalprice"), builder.literal(220388.06))) .aggregate( builder.groupKey("o_custkey"), builder.count());
5. Hybrid planning
Calling convention Scan Scan Join Filter Join Scan Initially all nodes belong to “logical” calling convention. Logical calling convention cannot be implemented, so has infinite cost
Calling convention Scan Scan Join Filter Join Scan Tables can’t be moved so there is only one choice of calling convention for each table. Examples: ● Enumerable ● Druid ● Drill ● HBase ● JDBC
Calling convention Scan Scan Join Filter Join Scan Rules fire to convert nodes to particular calling conventions. The calling convention propagates through the tree. Because this is Volcano, each node can have multiple conventions.
Calling convention Scan Scan Join Filter Join Scan Rules fire to convert nodes to particular calling conventions. The calling convention propagates through the tree. Because this is Volcano, each node can have multiple conventions.
Calling convention Scan Scan Join Filter Join Scan Rules fire to convert nodes to particular calling conventions. The calling convention propagates through the tree. Because this is Volcano, each node can have multiple conventions.
Converters Scan Scan Join Filter Join To keep things honest, we need to insert a converter at each point where the convention changes. (Recall: Volcano has an enforcer for each trait. Convention is a physical property, and converter is the enforcer.) Blue to Logical BlueFilterRule: LogicalFilter(BlueToLogical(Blue b)) → BlueToLogical(BlueFilter(b)) Green to Logical Scan
Converters Scan Scan Join Join Green to Logical To keep things honest, we need to insert a converter at each point where the convention changes. (Recall: Volcano has an enforcer for each trait. Convention is a physical property, and converter is the enforcer.) Blue to Logical Filter BlueFilterRule: LogicalFilter(BlueToLogical(Blue b)) → BlueToLogical(BlueFilter(b)) Scan
Generating programs to implement hybrid plans Hybrid plans are glued together using an engine - a convention that does not have a storage format. (Example engines: Drill, Spark, Presto.) To implement, we generate a program that calls out to query1 and query2. The "Blue-to-Orange" converter is typically a function in the Orange language that embeds a Blue query. Similarly "Green-to-Orange". Scan Scan Join Join Green to Orange Blue to Orange Filter Scan
6. Coding module II: Custom operators/rules (Homework)
What we want to achieve? EnumerableJoin [$0=$9,type=inner] EnumerableTableScan [CUSTOMER] EnumerableCalc [$0<3] EnumerableCalc [C_NAME=$1,O_ORDERKEY=$8, O_ORDERDATE=$12] EnumerableSort [sort0=$0,dir0=ASC,sort1=$1,dir1=ASC] EnumerableTableScan [ORDERS] EnumerableJoin [$0=$9,type=inner] LuceneTableScan [CUSTOMER] LuceneFilter [$0<3] EnumerableCalc [C_NAME=$1,O_ORDERKEY=$8, O_ORDERDATE=$12] EnumerableSort [sort0=$0,dir0=ASC,sort1=$1,dir1=ASC] LuceneTableScan [ORDERS] LuceneToEnumerableConverter LuceneToEnumerableConverter
What do we need? EnumerableJoin [$0=$9,type=inner] LuceneTableScan [CUSTOMER] LuceneFilter [$0<3] EnumerableCalc [C_NAME=$1,O_ORDERKEY=$8, O_ORDERDATE=$12] EnumerableSort [sort0=$0,dir0=ASC,sort1=$1,dir1=ASC] LuceneTableScan [ORDERS] LuceneToEnumerableConverter LuceneToEnumerableConverter Two calling conventions: 1. Enumerable 2. Lucene Three custom operators: 1. LuceneTableScan 2. LuceneToEnumerableConverter 3. LuceneFilter Three custom conversion rules: 1. LogicalTableScan → LuceneTableScan 2. LogicalFilter → LuceneFilter 3. LuceneANY → LuceneToEnumerableConverter
What do we need? EnumerableJoin [$0=$9,type=inner] LuceneTableScan [CUSTOMER] LuceneFilter [$0<3] EnumerableCalc [C_NAME=$1,O_ORDERKEY=$8, O_ORDERDATE=$12] EnumerableSort [sort0=$0,dir0=ASC,sort1=$1,dir1=ASC] LuceneTableScan [ORDERS] LuceneToEnumerableConverter LuceneToEnumerableConverter Two calling conventions: 1. Enumerable 2. Lucene Three custom operators: 1. LuceneTableScan 2. LuceneToEnumerableConverter 3. LuceneFilter Three custom conversion rules: 1. LogicalTableScan → LuceneTableScan 2. LogicalFilter → LuceneFilter 3. LuceneANY → LuceneToEnumerableConverter STEP 1 STEP 2 STEP 3 STEP 4 STEP 5 STEP 6
7. Volcano Planner Internals
Volcano planning algorithm Based on two papers by Goetz Graefe in the 1990s (Volcano, Cascades), now the industry standard for cost-based optimization. Dynamic programming: to optimize a relational expression R0 , convert it into equivalent expressions {R1 , R2 , …}, and pick the one with the lowest cost. Much of the cost of R is the cost of its input(s). So we apply dynamic programming to its inputs, too. R0 S0 T0
Volcano planning algorithm Based on two papers by Goetz Graefe in the 1990s (Volcano, Cascades), now the industry standard for cost-based optimization. Dynamic programming: to optimize a relational expression R0 , convert it into equivalent expressions {R1 , R2 , …}, and pick the one with the lowest cost. Much of the cost of R is the cost of its input(s). So we apply dynamic programming to its inputs, too. R0 S0 R1 T0
Volcano planning algorithm Based on two papers by Goetz Graefe in the 1990s (Volcano, Cascades), now the industry standard for cost-based optimization. Dynamic programming: to optimize a relational expression R0 , convert it into equivalent expressions {R1 , R2 , …}, and pick the one with the lowest cost. Much of the cost of R is the cost of its input(s). So we apply dynamic programming to its inputs, too. R0 S0 R1 T0 T1
Volcano planning algorithm Based on two papers by Goetz Graefe in the 1990s (Volcano, Cascades), now the industry standard for cost-based optimization. Dynamic programming: to optimize a relational expression R0 , convert it into equivalent expressions {R1 , R2 , …}, and pick the one with the lowest cost. Much of the cost of R is the cost of its input(s). So we apply dynamic programming to its inputs, too. R0 S0 R1 T0 T1 R2 Uo
Volcano planning algorithm We keep equivalence sets of expressions (class RelSet). Each input of a relational expression is an equivalence set + required physical properties (class RelSubset). R0 S0 R1 T0 T1 R2 Uo
Volcano planning algorithm Each relational expression has a memo (digest), so we will recognize it if we generate it again. R0 S0 R1 T0 T1 R2 Uo
Volcano planning algorithm If an expression transforms to an expression in another equivalence set, we can merge those equivalence sets. R0 S0 R1 T0 T1 R2 Uo
Matches and queues We register a new RelNode by adding it to a RelSet. Each rule instance declares a pattern of RelNode types (and other properties) that it will match. Suppose we have: ● Filter-on-Project ● Project-on-Project ● Project-on-Join On register, we detect rules that are newly matched. Project Filter Union Join Scan Project
Matches and queues We register a new RelNode by adding it to a RelSet. Each rule instance declares a pattern of RelNode types (and other properties) that it will match. Suppose we have: ● Filter-on-Project ● Project-on-Project ● Project-on-Join On register, we detect rules that are newly matched. Project Filter Union Project Join Scan Project
Matches and queues We register a new RelNode by adding it to a RelSet. Each rule instance declares a pattern of RelNode types (and other properties) that it will match. Suppose we have: ● Filter-on-Project ● Project-on-Project ● Project-on-Join On register, we detect rules that are newly matched. (4 matches.) Project Filter Union Project Join Scan Project
Matches and queues Should we fire these matched rules immediately? No! Because rule match #1 would generate new matches… which would generate new matches… and we'd never get to match #2. Instead, we put the matched rules on a queue. The queue allows us to: ● Search breadth-first (rather than depth-first) ● Prioritize (fire more "important" rules first) ● Potentially terminate when we have a "good enough" plan Equivalence sets containing registered RelNodes 0. Register each RelNode in the initial tree. Rule match queue 1. Each time a RelNode is registered, find rule matches, and put RuleMatch objects on the queue. 3. Pop the top rule match. Fire the rule. Register each RelNode generated by the rule, merging sets if equivalences are found. Goto 1. 2. If the queue is empty, or the cost is good enough, we're done.
Other planner engines, same great rules Three planner engines: ● Volcano ● Volcano top-down (Cascades style) ● Hep applies rules in a strict "program" The same rules are used by all engines. It takes a lot of time effort to write a high-quality rule. Rules can be reused, tested, improved, and they compose with other rules. Calcite's library of rules is valuable.
8. Dialects
Calcite architecture Apache Calcite JDBC adapter Pluggable rewrite rules Enumerable adapter MongoDB adapter File adapter (CSV, JSON, Http) Apache Kafka adapter Apache Spark adapter SQL SQL parser & validator Query planner Relational algebra At what points in the Calcite stack do ‘languages’ exist? ● Incoming SQL ● Validating SQL against built-in operators ● Type system (e.g. max size of INTEGER type) ● JDBC adapter generates SQL ● Other adapters generate other languages
Parsing & validating SQL - what knobs can I turn? SELECT deptno AS d, SUM(sal) AS [sumSal] FROM [HR].[Emp] WHERE ename NOT ILIKE "A%" GROUP BY d ORDER BY 1, 2 DESC SqlConformance.isGroupByAlias() = true SqlConformance.isSortByOrdinal() = true Lex.quoting = Quoting.BRACKET Lex.charLiteralStyle = CharLiteralStyle.BQ_DOUBLE SqlValidator.Config.defaultNullCollation = HIGH Lex.unquotedCasing = Casing.TO_UPPER Lex.quotedCasing = Casing.UNCHANGED PARSER_FACTORY = "org.apache.calcite.sql.parser.impl.SqlParserImpl.FACTORY" FUN = "postgres" (ILIKE is not standard SQL)
SQL dialect - APIs and properties Pluggable rewrite rules Pluggable parser, lexical, conformance, operators Pluggable SQL dialect SQL SQL SQL parser & validator Query planner Relational algebra JDBC adapter interface SqlParserImplFactory CalciteConnectionProperty.LEX enum Lex enum Quoting enum Casing enum CharLiteralStyle CalciteConnectionProperty.CONFORMANCE interface SqlConformance CalciteConnectionProperty.FUN interface SqlOperatorTable class SqlStdOperatorTable class SqlLibraryOperators class SqlOperator class SqlFunction extends SqlOperator class SqlAggFunction extends SqlFunction class RelRule class SqlDialect interface SqlDialectFactory
Contributing a dialect (or anything!) to Calcite For your first code contribution, pick a small bug or feature. Introduce yourself! Email dev@, saying what you plan to do. Create a JIRA case describing the problem. To understand the code, find similar features. Run their tests in a debugger. Write 1 or 2 tests for your feature. Submit a pull request (PR).
Other front-end languages Pig RelBuilder Adapter Physical operators Morel Storage Query planner Relational algebra Datalog SQL parser & validator SQL Calcite is an excellent platform for implementing your own data language Write a parser for your language, use RelBuilder to translate to relational algebra, and you can use any of Calcite's back-end implementations
9. Materialized views
Forwards planning Backwards planning Backwards planning Until now, we have seen forward planning. Forward planning transforms an expression (R0 ) to many equivalent forms and picks the one with lowest cost (Ropt ). Backwards planning transforms an expression to an equivalent form (RN ) that contains a target expression (T). R0 R0 T Ropt
Forwards planning Backwards planning Equivalence set Backwards planning Ropt R2 R1 R0 Equivalence set RN R2 R1 R0 T S Until now, we have seen forward planning. Forward planning transforms an expression (R0 ) to many equivalent forms and picks the one with lowest cost (Ropt ). Backwards planning transforms an expression to an equivalent form (RN ) that contains a target expression (T).
Applications of backwards planning Indexes (e.g. b-tree indexes). An index is a derived data structure whose contents can be described as a relational expression (generally project-sort). When we are planning a query, it already exists (i.e. the cost has already been paid). Summary tables. A summary table is a derived data structure (generally filter-project-join-aggregate). Replicas with different physical properties (e.g. copy the table from New York to Tokyo, or copy the table and partition by month(orderDate), sort by productId). Incremental view maintenance. Materialized view V is populated from base table T. Yesterday, we populated V with V0 = Q(T0 ). Today we want to make its contents equal to V1 = Q(T1 ). Can we find and apply a delta query, dQ = Q(T1 - T0 )?
Materialized views in Calcite { "schemas": { "name": "HR", "tables": [ { "name": "emp" } ], "materializations": [ { "table": "i_emp_job", "sql": "SELECT job, empno FROM emp ORDER BY job, empno" }, { "table": "add_emp_deptno", "sql": "SELECT deptno, SUM(sal) AS ss, COUNT(*) AS c FROM emp GROUP BY deptno" } ] } } /** Transforms a relational expression into a * semantically equivalent relational expression, * according to a given set of rules and a cost * model. */ public interface RelOptPlanner { /** Defines an equivalence between a table and * a query. */ void addMaterialization( RelOptMaterialization materialization); /** Finds the most efficient expression to * implement this query. */ RelNode findBestExp(); } /** Records that a particular query is materialized * by a particular table. */ public class RelOptMaterialization { public final RelNode tableRel; public final List<String> qualifiedTableName; public final RelNode queryRel; } You can define materializations in a JSON model, via the planner API, or via CREATE MATERIALIZED VIEW DDL (not shown).
More about materialized views ● There are several algorithms to rewrite queries to match materialized views ● A lattice is a data structure to model a star schema ● Calcite has algorithms to recommend an optimal set of summary tables for a lattice (given expected queries, and statistics about column cardinality) ● Data profiling algorithms estimate the cardinality of all combinations of columns
10. Working with spatial data
Spatial query Find all restaurants within 1.5 distance units of my current location: We cannot use a B-tree index (it can sort points by x or y coordinates, but not both) and specialized spatial indexes (such as R*-trees) are not generally available. restaurant x y Zachary’s pizza 3 1 King Yen 7 7 Filippo’s 7 4 Station burger 5 6 SELECT * FROM Restaurants AS r WHERE ST_Distance( ST_MakePoint(r.x, r.y), ST_MakePoint(6, 7)) < 1.5 • • • • Zachary’s pizza Filippo’s King Yen Station burger
Hilbert space-filling curve ● A space-filling curve invented by mathematician David Hilbert ● Every (x, y) point has a unique position on the curve ● Points near to each other typically have Hilbert indexes close together
• • • • Add restriction based on h, a restaurant’s distance along the Hilbert curve Must keep original restriction due to false positives Using Hilbert index restaurant x y h Zachary’s pizza 3 1 5 King Yen 7 7 41 Filippo’s 7 4 52 Station burger 5 6 36 Zachary’s pizza Filippo’s SELECT * FROM Restaurants AS r WHERE (r.h BETWEEN 35 AND 42 OR r.h BETWEEN 46 AND 46) AND ST_Distance( ST_MakePoint(r.x, r.y), ST_MakePoint(6, 7)) < 1.5 King Yen Station burger
Telling the optimizer 1. Declare h as a generated column 2. Sort table by h Planner can now convert spatial range queries into a range scan Does not require specialized spatial index such as R*-tree Very efficient on a sorted table such as HBase There are similar techniques for other spatial patterns (e.g. region-to-region join) CREATE TABLE Restaurants ( restaurant VARCHAR(20), x DOUBLE, y DOUBLE, h DOUBLE GENERATED ALWAYS AS ST_Hilbert(x, y) STORED) SORT KEY (h); restaurant x y h Zachary’s pizza 3 1 5 Station burger 5 6 36 King Yen 7 7 41 Filippo’s 7 4 52
11. Research using Apache Calcite
Apache Calcite Tutorial - BOSS 21
Yes, VLDB 2021! Go to the talk! 0900 Wednesday.
Apache Calcite Tutorial - BOSS 21
Thank you! @julianhyde @szampetak https://calcite.apache.org
Resources ● Calcite project https://calcite.apache.org ● Materialized view algorithms https://calcite.apache.org/docs/materialized_views.html ● JSON model https://calcite.apache.org/docs/model.html ● Lazy beats smart and fast (DataEng 2018) - MVs, spatial, profiling https://www.slideshare.net/julianhyde/lazy-beats-smart-and-fast ● Efficient spatial queries on vanilla databases (ApacheCon 2018) https://www.slideshare.net/julianhyde/spatial-query-on-vanilla-databases ● Graefe, McKenna. The Volcano Optimizer Generator, 1991 ● Graefe. The Cascades Framework for Query Optimization, 1995 ● Slideshare (past presentations by Julian Hyde, including several about Apache Calcite) https://www.slideshare.net/julianhyde

More Related Content

PPTX
iceberg introduction.pptx
Dori Waldman
 
PDF
Introduction to Apache Calcite
Jordan Halterman
 
PDF
Apache Calcite: A Foundational Framework for Optimized Query Processing Over ...
Julian Hyde
 
PDF
Measures of query cost
Hitesh Mohapatra
 
PDF
Apache Calcite (a tutorial given at BOSS '21)
Julian Hyde
 
PDF
Fast federated SQL with Apache Calcite
Chris Baynes
 
PPTX
SQL Query Optimization: Why Is It So Hard to Get Right?
Brent Ozar
 
PDF
如何快速实现数据编织架构
Denodo
 
iceberg introduction.pptx
Dori Waldman
 
Introduction to Apache Calcite
Jordan Halterman
 
Apache Calcite: A Foundational Framework for Optimized Query Processing Over ...
Julian Hyde
 
Measures of query cost
Hitesh Mohapatra
 
Apache Calcite (a tutorial given at BOSS '21)
Julian Hyde
 
Fast federated SQL with Apache Calcite
Chris Baynes
 
SQL Query Optimization: Why Is It So Hard to Get Right?
Brent Ozar
 
如何快速实现数据编织架构
Denodo
 

What's hot (20)

PDF
Apache Calcite: One planner fits all
Julian Hyde
 
PPTX
Apache Calcite overview
Julian Hyde
 
PDF
Smarter Together - Bringing Relational Algebra, Powered by Apache Calcite, in...
Julian Hyde
 
PPTX
Presto query optimizer: pursuit of performance
DataWorks Summit
 
PDF
A Deep Dive into Query Execution Engine of Spark SQL
Databricks
 
PDF
Spark shuffle introduction
colorant
 
PDF
Cost-based Query Optimization in Apache Phoenix using Apache Calcite
Julian Hyde
 
PPTX
HBase and HDFS: Understanding FileSystem Usage in HBase
enissoz
 
PDF
A Practical Introduction to Handling Log Data in ClickHouse, by Robert Hodges...
Altinity Ltd
 
PPTX
High Performance, High Reliability Data Loading on ClickHouse
Altinity Ltd
 
PPTX
Apache Spark Architecture
Alexey Grishchenko
 
PDF
Intro to HBase
alexbaranau
 
PDF
ClickHouse Data Warehouse 101: The First Billion Rows, by Alexander Zaitsev a...
Altinity Ltd
 
PDF
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
Databricks
 
PDF
ClickHouse Deep Dive, by Aleksei Milovidov
Altinity Ltd
 
PPTX
The Basics of MongoDB
valuebound
 
PPTX
Real-time Analytics with Trino and Apache Pinot
Xiang Fu
 
PPTX
Centralized log-management-with-elastic-stack
Rich Lee
 
PPTX
Introduction to Apache Spark
Rahul Jain
 
PPTX
quick intro to elastic search
medcl
 
Apache Calcite: One planner fits all
Julian Hyde
 
Apache Calcite overview
Julian Hyde
 
Smarter Together - Bringing Relational Algebra, Powered by Apache Calcite, in...
Julian Hyde
 
Presto query optimizer: pursuit of performance
DataWorks Summit
 
A Deep Dive into Query Execution Engine of Spark SQL
Databricks
 
Spark shuffle introduction
colorant
 
Cost-based Query Optimization in Apache Phoenix using Apache Calcite
Julian Hyde
 
HBase and HDFS: Understanding FileSystem Usage in HBase
enissoz
 
A Practical Introduction to Handling Log Data in ClickHouse, by Robert Hodges...
Altinity Ltd
 
High Performance, High Reliability Data Loading on ClickHouse
Altinity Ltd
 
Apache Spark Architecture
Alexey Grishchenko
 
Intro to HBase
alexbaranau
 
ClickHouse Data Warehouse 101: The First Billion Rows, by Alexander Zaitsev a...
Altinity Ltd
 
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
Databricks
 
ClickHouse Deep Dive, by Aleksei Milovidov
Altinity Ltd
 
The Basics of MongoDB
valuebound
 
Real-time Analytics with Trino and Apache Pinot
Xiang Fu
 
Centralized log-management-with-elastic-stack
Rich Lee
 
Introduction to Apache Spark
Rahul Jain
 
quick intro to elastic search
medcl
 
Ad

Similar to Apache Calcite Tutorial - BOSS 21 (20)

PPTX
Iac d.damyanov 4.pptx
Dimitar Damyanov
 
PDF
Incremental View Maintenance with Coral, DBT, and Iceberg
Walaa Eldin Moustafa
 
PDF
Apache DeltaSpike the CDI toolbox
Antoine Sabot-Durand
 
PDF
Apache DeltaSpike: The CDI Toolbox
Virtual JBoss User Group
 
PDF
Data Exploration with Apache Drill: Day 1
Charles Givre
 
PPTX
Webinar: What's new in the .NET Driver
MongoDB
 
PDF
DumpsBoss Your Key to Passing the DP 203 Exam
jackjohnson9842
 
PDF
Top DP-203 Exam Dumps PDF Free Download for Easy Preparation
jackjohnson9842
 
PDF
6 tips for improving ruby performance
Engine Yard
 
PDF
Bye bye $GLOBALS['TYPO3_DB']
Jan Helke
 
PPTX
Scale By The Bay | 2020 | Gimel
Deepak Chandramouli
 
PDF
LinuxCon 2013 Steven Dake on Using Heat for autoscaling OpenShift on Openstack
OpenShift Origin
 
PDF
22nd Athens Big Data Meetup - 1st Talk - MLOps Workshop: The Full ML Lifecycl...
Athens Big Data
 
PDF
The Ring programming language version 1.8 book - Part 95 of 202
Mahmoud Samir Fayed
 
PDF
Key projects Data Science and Engineering
Vijayananda Mohire
 
PDF
Key projects Data Science and Engineering
Vijayananda Mohire
 
PPTX
Document validation in MongoDB 3.2
Andrew Morgan
 
PPT
Database Development Replication Security Maintenance Report
nyin27
 
PDF
What's New in MariaDB Server 10.2 and MariaDB MaxScale 2.1
MariaDB plc
 
PDF
What's New in MariaDB Server 10.2 and MariaDB MaxScale 2.1
MariaDB plc
 
Iac d.damyanov 4.pptx
Dimitar Damyanov
 
Incremental View Maintenance with Coral, DBT, and Iceberg
Walaa Eldin Moustafa
 
Apache DeltaSpike the CDI toolbox
Antoine Sabot-Durand
 
Apache DeltaSpike: The CDI Toolbox
Virtual JBoss User Group
 
Data Exploration with Apache Drill: Day 1
Charles Givre
 
Webinar: What's new in the .NET Driver
MongoDB
 
DumpsBoss Your Key to Passing the DP 203 Exam
jackjohnson9842
 
Top DP-203 Exam Dumps PDF Free Download for Easy Preparation
jackjohnson9842
 
6 tips for improving ruby performance
Engine Yard
 
Bye bye $GLOBALS['TYPO3_DB']
Jan Helke
 
Scale By The Bay | 2020 | Gimel
Deepak Chandramouli
 
LinuxCon 2013 Steven Dake on Using Heat for autoscaling OpenShift on Openstack
OpenShift Origin
 
22nd Athens Big Data Meetup - 1st Talk - MLOps Workshop: The Full ML Lifecycl...
Athens Big Data
 
The Ring programming language version 1.8 book - Part 95 of 202
Mahmoud Samir Fayed
 
Key projects Data Science and Engineering
Vijayananda Mohire
 
Key projects Data Science and Engineering
Vijayananda Mohire
 
Document validation in MongoDB 3.2
Andrew Morgan
 
Database Development Replication Security Maintenance Report
nyin27
 
What's New in MariaDB Server 10.2 and MariaDB MaxScale 2.1
MariaDB plc
 
What's New in MariaDB Server 10.2 and MariaDB MaxScale 2.1
MariaDB plc
 
Ad

Recently uploaded (20)

PPTX
OOP with Java - Java Introduction (Basics)
Rashmi T V
 
PDF
Mohammad Mahdi Farshadian CV - Prospective PhD Student 2026
Educational Group Mohammad Farshadian
 
PDF
Arduino robotics embedded978-1-4302-3184-4.pdf
AbdullahEmam4
 
PPTX
Lesson 3_Tessellation.pptx finite Mathematics
quakeplayz54
 
PPTX
Fluid Mechanics, Module 3: Basics of Fluid Mechanics
Dr. Rahul Kumar
 
PPTX
CH1 Production IntroductoryConcepts.pptx
RacemMellouli1
 
PDF
SM_6th-Sem__Cse_Internet-of-Things.pdf IOT
smceramu
 
PDF
Model Code of Practice - Construction Work - 21102022 .pdf
AinieButt1
 
PPTX
MET 305 MODULE 1 KTU 2019 SCHEME 25.pptx
VinayB68
 
PPTX
Engineering Ethics, Safety and Environment [Autosaved] (1).pptx
MehrabTaseenTalukder
 
PPTX
IOT PPTs Week 10 Lecture Material.pptx of NPTEL Smart Cities contd
ssusera400e8
 
PPTX
M Tech Sem 1 Civil Engineering Environmental Sciences.pptx
ShriNRPrasad
 
PDF
PRIZ Academy - 9 Windows Thinking Where to Invest Today to Win Tomorrow.pdf
PRIZ Guru
 
PPTX
Recipes for Real Time Voice AI WebRTC, SLMs and Open Source Software.pptx
Alberto González Trastoy
 
PPTX
MET 305 2019 SCHEME MODULE 2 COMPLETE.pptx
VinayB68
 
PPTX
CYBER-CRIMES AND SECURITY A guide to understanding
Davies Chacko
 
PPTX
Foundation to blockchain - A guide to Blockchain Tech
Davies Chacko
 
PPTX
Infosys Presentation by1.Riyan Bagwan 2.Samadhan Naiknavare 3.Gaurav Shinde 4...
bapushinde7218
 
PPTX
web development for engineering and engineering
keshriji700
 
PDF
Digital Logic Computer Design lecture notes
CHINNASUNKAIAH1
 
OOP with Java - Java Introduction (Basics)
Rashmi T V
 
Mohammad Mahdi Farshadian CV - Prospective PhD Student 2026
Educational Group Mohammad Farshadian
 
Arduino robotics embedded978-1-4302-3184-4.pdf
AbdullahEmam4
 
Lesson 3_Tessellation.pptx finite Mathematics
quakeplayz54
 
Fluid Mechanics, Module 3: Basics of Fluid Mechanics
Dr. Rahul Kumar
 
CH1 Production IntroductoryConcepts.pptx
RacemMellouli1
 
SM_6th-Sem__Cse_Internet-of-Things.pdf IOT
smceramu
 
Model Code of Practice - Construction Work - 21102022 .pdf
AinieButt1
 
MET 305 MODULE 1 KTU 2019 SCHEME 25.pptx
VinayB68
 
Engineering Ethics, Safety and Environment [Autosaved] (1).pptx
MehrabTaseenTalukder
 
IOT PPTs Week 10 Lecture Material.pptx of NPTEL Smart Cities contd
ssusera400e8
 
M Tech Sem 1 Civil Engineering Environmental Sciences.pptx
ShriNRPrasad
 
PRIZ Academy - 9 Windows Thinking Where to Invest Today to Win Tomorrow.pdf
PRIZ Guru
 
Recipes for Real Time Voice AI WebRTC, SLMs and Open Source Software.pptx
Alberto González Trastoy
 
MET 305 2019 SCHEME MODULE 2 COMPLETE.pptx
VinayB68
 
CYBER-CRIMES AND SECURITY A guide to understanding
Davies Chacko
 
Foundation to blockchain - A guide to Blockchain Tech
Davies Chacko
 
Infosys Presentation by1.Riyan Bagwan 2.Samadhan Naiknavare 3.Gaurav Shinde 4...
bapushinde7218
 
web development for engineering and engineering
keshriji700
 
Digital Logic Computer Design lecture notes
CHINNASUNKAIAH1
 

Apache Calcite Tutorial - BOSS 21

  • 1. Tutorial @BOSS’21 Copenhagen Stamatis Zampetakis, Julian Hyde • August 16, 2021
  • 3. Tutorial @BOSS’21 Copenhagen Stamatis Zampetakis, Julian Hyde • August 16, 2021 # Follow these steps to set up your environment # (The first time, it may take ~3 minutes to download dependencies.) git clone --branch boss21 https://github.com/zabetak/calcite-tutorial.git java -version # need Java 8 or higher cd calcite-tutorial ./mvnw package -DskipTests
  • 4. Setup Environment Requirements 1. Git 2. JDK version ≥ 1.8 Steps 1. Clone GitHub repository 2. Load to IDE (preferred IntelliJ) a. Click Open b. Navigate to calcite-tutorial c. Select pom.xml file d. Choose “Open as Project” 3. Compile the project git clone --branch boss21 https://github.com/zabetak/calcite-tutorial.git java -version cd calcite-tutorial ./mvnw package -DskipTests
  • 5. Julian Hyde @julianhyde Senior Staff Engineer @ Google / Looker Creator of Apache Calcite PMC member of Apache Arrow, Drill, Eagle, Incubator and Kylin Stamatis Zampetakis @szampetak Senior Software Engineer @ Cloudera, Hive query optimizer team PMC member of Apache Calcite; Hive committer PhD in Data Management, INRIA & Paris-Sud University About us
  • 6. Outline 1. Introduction 2. CSV Adapter Demo 3. Coding module I: Main components 4. Coding module I Exercises (Homework) 5. Hybrid planning 6. Coding module II: Custom operators/rules (Homework) 7. Volcano Planner internals 8. Dialects 9. Materialized views 10. Working with spatial data 11. Research using Apache Calcite optional optional optional optional
  • 8. 1. Retrieve books and authors 2. Display image, title, price of the book along with firstname & lastname of the author 3. Sort the books based on their id (price or something else) 4. Show results in groups of five Motivation: Data views
  • 9. What, where, how data are stored? author 0..1 AUTHOR id int fname string lname string birth date BOOK id int title string price decimal year int
  • 10. XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN FILESYSTEM What, where, how data are stored? author 0..1 AUTHOR id int fname string lname string birth date BOOK id int title string price decimal year int
  • 11. BIN What, where, how data are stored? XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN FILESYSTEM author 0..1 AUTHOR id int fname string lname string birth date BOOK id int title string price decimal year int
  • 12. What, where, how data are stored? 360+ DBMS XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN FILESYSTEM author 0..1 AUTHOR id int fname string lname string birth date BOOK id int title string price decimal year int
  • 13. ★ Open-source search engine ★ Java library ★ Powerful indexing & search features ★ Spell checking, hit highlighting ★ Advanced analysis/tokenization capabilities ★ ACID transactions ★ Ultra compact memory/disk format Apache Lucene
  • 14. How to query the data? SELECT b.id, b.title, b.year, a.fname, a.lname FROM Book b LEFT OUTER JOIN Author a ON b.author=a.id ORDER BY b.id LIMIT 5 1. Retrieve books and authors 2. Display image, title, price of the book along with firstname & lastname of the author 3. Sort the books based on their id (price or something else) 4. Show results in groups of five
  • 15. Query processor architecture Query Relational Algebra Query planner CatalogReader Schema Metadata (Cost, Statistics) Execution engine Results Parser Rules Relational Algebra
  • 17. Apache Calcite SQL query SqlToRelConverter RelNode RelOptPlanner CatalogReader Schema RelRule RelBuilder RelRunner Results API calls SqlParser SqlValidator RelNode SqlNode SqlNode Dev & extensions RelMetadata Provider
  • 19. Adapter Implement SchemaFactory interface Connect to a data source using parameters Extract schema - return a list of tables Push down processing to the data source: ● A set of planner rules ● Calling convention (optional) ● Query model & query generator (optional) { "schemas": [ { "name": "BOOKSTORE", "type": "custom", "factory": "org.apache.calcite.adapter.file.FileSchemaFactory", "operand": { "directory": "bookstore" } } ] ]
  • 20. 3. Coding module I: Main components
  • 21. Setup schema & type factory <<Interface>> Schema +getTable(name: Table): Table <<Interface>> Table +getRowType(typeFactory: RelDataTypeFactory): RelDataType <<Interface>> RelDataTypeFactory +createJavaType(clazz: Class): RelDataType +createSqlType(typeName: SqlTypeName): RelDataType * tables
  • 23. SELECT b.id, b.title, b.year, a.fname, a.lname FROM Book AS b LEFT OUTER JOIN Author a ON b.author = a.id WHERE b.year > 1830 ORDER BY b.id LIMIT 5 Query to Abstract Syntax Tree (AST) SQL query SqlParser SqlNode Numeric Literal 5 Identifier b.id NodeList Select OrderBy Join BasicCall As Operator Identifier author Identifier a Literal LEFT Literal ON Identifier b.author Identifier a.id Binary Operator = As Operator Identifier book Identifier b BasicCall BasicCal l Identifier b.year Numeric Literal 1830 Binary Operator > BasicCall Identifier b.id NodeList Identifier b.title Identifier b.year Identifier a.fname Identifier a.lname
  • 24. AST to logical plan SqlNode Join Select b.id, b.title, b.year, a.fname, a.lname BasicCall b.author = a.id BasicCall book AS b BasicCall author AS a Literal LEFT Literal ON LogicalJoin [$3==$4,type=left] LogicalTableScan [Book] LogicalTableScan [Author] LogicalFilter [$2>1830] LogicalProject [ID=$0,TITLE=$1,YEAR=$2, FNAME=$5,LNAME=$6] LogicalSort [sort0=$0,dir=ASC,fetch=5] SqlValidator SqlToRelConverter SqlNode RelNode OrderBy b.id FETCH 5 BasicCall b.year > 1830
  • 25. Logical to physical plan RelRule EnumerableSortRule EnumerableProjectRule EnumerableFilterRule EnumerableJoinRule EnumerableTableScanRule RelOptPlanner RelNode RelNode LogicalJoin [$3==$4,type=left] LogicalTableScan [Book] LogicalTableScan [Author] LogicalFilter [$2>1830] LogicalProject [ID=$0,TITLE=$1,YEAR=$2, FNAME=$5,LNAME=$6] LogicalSort [sort0=$0,dir=ASC,fetch=5] EnumerableJoin [$3==$4,type=left] EnumerableTableScan [Book] EnumerableTableScan [Author] EnumerableFilter [$2>1830] EnumerableProject [ID=$0,TITLE=$1,YEAR=$2, FNAME=$5,LNAME=$6] EnumerableSort [sort0=$0,dir=ASC,fetch=5]
  • 26. Physical to Executable plan EnumerableInterpretable Java code RelNode EnumerableJoin [$3==$4,type=left] EnumerableTableScan [Book] EnumerableTableScan [Author] EnumerableFilter [$2>1830] EnumerableProject [ID=$0,TITLE=$1,YEAR=$2, FNAME=$5,LNAME=$6] EnumerableSort [sort0=$0,dir=ASC,fetch=5]
  • 27. 4. Coding module I: Exercises (Homework)
  • 28. Exercise I: Execute more SQL queries Include GROUP BY and other types of clauses: SELECT o.o_custkey, COUNT(*) FROM orders AS o GROUP BY o.o_custkey
  • 29. Exercise I: Execute more SQL queries Include GROUP BY and other types of clauses: ● Missing rule to convert LogicalAggregate to EnumerableAggregate ● Add EnumerableRules.ENUMERABLE_AGGREGATE_RULE to the planner SELECT o.o_custkey, COUNT(*) FROM orders AS o GROUP BY o.o_custkey
  • 30. Push filter below the join: Exercise II: Improve performance by applying more optimization rules SELECT c.c_name, o.o_orderkey, o.o_orderdate FROM customer AS c INNER JOIN orders AS o ON c.c_custkey = o.o_custkey WHERE c.c_custkey < 3 ORDER BY c.c_name, o.o_orderkey
  • 31. Push filter below the join: 1. Add rule CoreRules.FILTER_INTO_JOIN to the planner 2. Compare plans before and after (or logical and physical) 3. Check cost estimates by using SqlExplainLevel.ALL_ATTRIBUTES Exercise II: Improve performance by applying more optimization rules SELECT c.c_name, o.o_orderkey, o.o_orderdate FROM customer AS c INNER JOIN orders AS o ON c.c_custkey = o.o_custkey WHERE c.c_custkey < 3 ORDER BY c.c_name, o.o_orderkey
  • 32. Exercise II: Improve performance by applying more optimization rules LogicalJoin [$0=$9,type=inner] LogicalTableScan [CUSTOMER] LogicalTableScan [ORDERS] LogicalFilter [$0<3] LogicalProject [C_NAME=$1,O_ORDERKEY=$8, O_ORDERDATE=$12] LogicalSort [sort0=$0,dir0=ASC,sort1=$1,dir1=ASC] RelRule FilterIntoJoinRule LogicalJoin [$0=$9,type=inner] LogicalTableScan [CUSTOMER] LogicalTableScan [ORDERS] LogicalFilter [$0<3] LogicalProject [C_NAME=$1,O_ORDERKEY=$8, O_ORDERDATE=$12] LogicalSort [sort0=$0,dir0=ASC,sort1=$1,dir1=ASC]
  • 33. Open LuceneBuilderProcessor.java and complete TODOs Exercise III: Use RelBuilder API to construct the logical plan SELECT o.o_custkey, COUNT(*) FROM orders AS o GROUP BY o.o_custkey SELECT o.o_custkey, COUNT(*) FROM orders AS o WHERE o.o_totalprice > 220388.06 GROUP BY o.o_custkey Q1: Q2:
  • 34. Exercise III: Use RelBuilder API to construct the logical plan builder .scan("orders") .filter( builder.call( SqlStdOperatorTable.GREATER_THAN, builder.field("o_totalprice"), builder.literal(220388.06))) .aggregate( builder.groupKey("o_custkey"), builder.count());
  • 36. Calling convention Scan Scan Join Filter Join Scan Initially all nodes belong to “logical” calling convention. Logical calling convention cannot be implemented, so has infinite cost
  • 37. Calling convention Scan Scan Join Filter Join Scan Tables can’t be moved so there is only one choice of calling convention for each table. Examples: ● Enumerable ● Druid ● Drill ● HBase ● JDBC
  • 38. Calling convention Scan Scan Join Filter Join Scan Rules fire to convert nodes to particular calling conventions. The calling convention propagates through the tree. Because this is Volcano, each node can have multiple conventions.
  • 39. Calling convention Scan Scan Join Filter Join Scan Rules fire to convert nodes to particular calling conventions. The calling convention propagates through the tree. Because this is Volcano, each node can have multiple conventions.
  • 40. Calling convention Scan Scan Join Filter Join Scan Rules fire to convert nodes to particular calling conventions. The calling convention propagates through the tree. Because this is Volcano, each node can have multiple conventions.
  • 41. Converters Scan Scan Join Filter Join To keep things honest, we need to insert a converter at each point where the convention changes. (Recall: Volcano has an enforcer for each trait. Convention is a physical property, and converter is the enforcer.) Blue to Logical BlueFilterRule: LogicalFilter(BlueToLogical(Blue b)) → BlueToLogical(BlueFilter(b)) Green to Logical Scan
  • 42. Converters Scan Scan Join Join Green to Logical To keep things honest, we need to insert a converter at each point where the convention changes. (Recall: Volcano has an enforcer for each trait. Convention is a physical property, and converter is the enforcer.) Blue to Logical Filter BlueFilterRule: LogicalFilter(BlueToLogical(Blue b)) → BlueToLogical(BlueFilter(b)) Scan
  • 43. Generating programs to implement hybrid plans Hybrid plans are glued together using an engine - a convention that does not have a storage format. (Example engines: Drill, Spark, Presto.) To implement, we generate a program that calls out to query1 and query2. The "Blue-to-Orange" converter is typically a function in the Orange language that embeds a Blue query. Similarly "Green-to-Orange". Scan Scan Join Join Green to Orange Blue to Orange Filter Scan
  • 44. 6. Coding module II: Custom operators/rules (Homework)
  • 45. What we want to achieve? EnumerableJoin [$0=$9,type=inner] EnumerableTableScan [CUSTOMER] EnumerableCalc [$0<3] EnumerableCalc [C_NAME=$1,O_ORDERKEY=$8, O_ORDERDATE=$12] EnumerableSort [sort0=$0,dir0=ASC,sort1=$1,dir1=ASC] EnumerableTableScan [ORDERS] EnumerableJoin [$0=$9,type=inner] LuceneTableScan [CUSTOMER] LuceneFilter [$0<3] EnumerableCalc [C_NAME=$1,O_ORDERKEY=$8, O_ORDERDATE=$12] EnumerableSort [sort0=$0,dir0=ASC,sort1=$1,dir1=ASC] LuceneTableScan [ORDERS] LuceneToEnumerableConverter LuceneToEnumerableConverter
  • 46. What do we need? EnumerableJoin [$0=$9,type=inner] LuceneTableScan [CUSTOMER] LuceneFilter [$0<3] EnumerableCalc [C_NAME=$1,O_ORDERKEY=$8, O_ORDERDATE=$12] EnumerableSort [sort0=$0,dir0=ASC,sort1=$1,dir1=ASC] LuceneTableScan [ORDERS] LuceneToEnumerableConverter LuceneToEnumerableConverter Two calling conventions: 1. Enumerable 2. Lucene Three custom operators: 1. LuceneTableScan 2. LuceneToEnumerableConverter 3. LuceneFilter Three custom conversion rules: 1. LogicalTableScan → LuceneTableScan 2. LogicalFilter → LuceneFilter 3. LuceneANY → LuceneToEnumerableConverter
  • 47. What do we need? EnumerableJoin [$0=$9,type=inner] LuceneTableScan [CUSTOMER] LuceneFilter [$0<3] EnumerableCalc [C_NAME=$1,O_ORDERKEY=$8, O_ORDERDATE=$12] EnumerableSort [sort0=$0,dir0=ASC,sort1=$1,dir1=ASC] LuceneTableScan [ORDERS] LuceneToEnumerableConverter LuceneToEnumerableConverter Two calling conventions: 1. Enumerable 2. Lucene Three custom operators: 1. LuceneTableScan 2. LuceneToEnumerableConverter 3. LuceneFilter Three custom conversion rules: 1. LogicalTableScan → LuceneTableScan 2. LogicalFilter → LuceneFilter 3. LuceneANY → LuceneToEnumerableConverter STEP 1 STEP 2 STEP 3 STEP 4 STEP 5 STEP 6
  • 48. 7. Volcano Planner Internals
  • 49. Volcano planning algorithm Based on two papers by Goetz Graefe in the 1990s (Volcano, Cascades), now the industry standard for cost-based optimization. Dynamic programming: to optimize a relational expression R0 , convert it into equivalent expressions {R1 , R2 , …}, and pick the one with the lowest cost. Much of the cost of R is the cost of its input(s). So we apply dynamic programming to its inputs, too. R0 S0 T0
  • 50. Volcano planning algorithm Based on two papers by Goetz Graefe in the 1990s (Volcano, Cascades), now the industry standard for cost-based optimization. Dynamic programming: to optimize a relational expression R0 , convert it into equivalent expressions {R1 , R2 , …}, and pick the one with the lowest cost. Much of the cost of R is the cost of its input(s). So we apply dynamic programming to its inputs, too. R0 S0 R1 T0
  • 51. Volcano planning algorithm Based on two papers by Goetz Graefe in the 1990s (Volcano, Cascades), now the industry standard for cost-based optimization. Dynamic programming: to optimize a relational expression R0 , convert it into equivalent expressions {R1 , R2 , …}, and pick the one with the lowest cost. Much of the cost of R is the cost of its input(s). So we apply dynamic programming to its inputs, too. R0 S0 R1 T0 T1
  • 52. Volcano planning algorithm Based on two papers by Goetz Graefe in the 1990s (Volcano, Cascades), now the industry standard for cost-based optimization. Dynamic programming: to optimize a relational expression R0 , convert it into equivalent expressions {R1 , R2 , …}, and pick the one with the lowest cost. Much of the cost of R is the cost of its input(s). So we apply dynamic programming to its inputs, too. R0 S0 R1 T0 T1 R2 Uo
  • 53. Volcano planning algorithm We keep equivalence sets of expressions (class RelSet). Each input of a relational expression is an equivalence set + required physical properties (class RelSubset). R0 S0 R1 T0 T1 R2 Uo
  • 54. Volcano planning algorithm Each relational expression has a memo (digest), so we will recognize it if we generate it again. R0 S0 R1 T0 T1 R2 Uo
  • 55. Volcano planning algorithm If an expression transforms to an expression in another equivalence set, we can merge those equivalence sets. R0 S0 R1 T0 T1 R2 Uo
  • 56. Matches and queues We register a new RelNode by adding it to a RelSet. Each rule instance declares a pattern of RelNode types (and other properties) that it will match. Suppose we have: ● Filter-on-Project ● Project-on-Project ● Project-on-Join On register, we detect rules that are newly matched. Project Filter Union Join Scan Project
  • 57. Matches and queues We register a new RelNode by adding it to a RelSet. Each rule instance declares a pattern of RelNode types (and other properties) that it will match. Suppose we have: ● Filter-on-Project ● Project-on-Project ● Project-on-Join On register, we detect rules that are newly matched. Project Filter Union Project Join Scan Project
  • 58. Matches and queues We register a new RelNode by adding it to a RelSet. Each rule instance declares a pattern of RelNode types (and other properties) that it will match. Suppose we have: ● Filter-on-Project ● Project-on-Project ● Project-on-Join On register, we detect rules that are newly matched. (4 matches.) Project Filter Union Project Join Scan Project
  • 59. Matches and queues Should we fire these matched rules immediately? No! Because rule match #1 would generate new matches… which would generate new matches… and we'd never get to match #2. Instead, we put the matched rules on a queue. The queue allows us to: ● Search breadth-first (rather than depth-first) ● Prioritize (fire more "important" rules first) ● Potentially terminate when we have a "good enough" plan Equivalence sets containing registered RelNodes 0. Register each RelNode in the initial tree. Rule match queue 1. Each time a RelNode is registered, find rule matches, and put RuleMatch objects on the queue. 3. Pop the top rule match. Fire the rule. Register each RelNode generated by the rule, merging sets if equivalences are found. Goto 1. 2. If the queue is empty, or the cost is good enough, we're done.
  • 60. Other planner engines, same great rules Three planner engines: ● Volcano ● Volcano top-down (Cascades style) ● Hep applies rules in a strict "program" The same rules are used by all engines. It takes a lot of time effort to write a high-quality rule. Rules can be reused, tested, improved, and they compose with other rules. Calcite's library of rules is valuable.
  • 62. Calcite architecture Apache Calcite JDBC adapter Pluggable rewrite rules Enumerable adapter MongoDB adapter File adapter (CSV, JSON, Http) Apache Kafka adapter Apache Spark adapter SQL SQL parser & validator Query planner Relational algebra At what points in the Calcite stack do ‘languages’ exist? ● Incoming SQL ● Validating SQL against built-in operators ● Type system (e.g. max size of INTEGER type) ● JDBC adapter generates SQL ● Other adapters generate other languages
  • 63. Parsing & validating SQL - what knobs can I turn? SELECT deptno AS d, SUM(sal) AS [sumSal] FROM [HR].[Emp] WHERE ename NOT ILIKE "A%" GROUP BY d ORDER BY 1, 2 DESC SqlConformance.isGroupByAlias() = true SqlConformance.isSortByOrdinal() = true Lex.quoting = Quoting.BRACKET Lex.charLiteralStyle = CharLiteralStyle.BQ_DOUBLE SqlValidator.Config.defaultNullCollation = HIGH Lex.unquotedCasing = Casing.TO_UPPER Lex.quotedCasing = Casing.UNCHANGED PARSER_FACTORY = "org.apache.calcite.sql.parser.impl.SqlParserImpl.FACTORY" FUN = "postgres" (ILIKE is not standard SQL)
  • 64. SQL dialect - APIs and properties Pluggable rewrite rules Pluggable parser, lexical, conformance, operators Pluggable SQL dialect SQL SQL SQL parser & validator Query planner Relational algebra JDBC adapter interface SqlParserImplFactory CalciteConnectionProperty.LEX enum Lex enum Quoting enum Casing enum CharLiteralStyle CalciteConnectionProperty.CONFORMANCE interface SqlConformance CalciteConnectionProperty.FUN interface SqlOperatorTable class SqlStdOperatorTable class SqlLibraryOperators class SqlOperator class SqlFunction extends SqlOperator class SqlAggFunction extends SqlFunction class RelRule class SqlDialect interface SqlDialectFactory
  • 65. Contributing a dialect (or anything!) to Calcite For your first code contribution, pick a small bug or feature. Introduce yourself! Email dev@, saying what you plan to do. Create a JIRA case describing the problem. To understand the code, find similar features. Run their tests in a debugger. Write 1 or 2 tests for your feature. Submit a pull request (PR).
  • 66. Other front-end languages Pig RelBuilder Adapter Physical operators Morel Storage Query planner Relational algebra Datalog SQL parser & validator SQL Calcite is an excellent platform for implementing your own data language Write a parser for your language, use RelBuilder to translate to relational algebra, and you can use any of Calcite's back-end implementations
  • 68. Forwards planning Backwards planning Backwards planning Until now, we have seen forward planning. Forward planning transforms an expression (R0 ) to many equivalent forms and picks the one with lowest cost (Ropt ). Backwards planning transforms an expression to an equivalent form (RN ) that contains a target expression (T). R0 R0 T Ropt
  • 69. Forwards planning Backwards planning Equivalence set Backwards planning Ropt R2 R1 R0 Equivalence set RN R2 R1 R0 T S Until now, we have seen forward planning. Forward planning transforms an expression (R0 ) to many equivalent forms and picks the one with lowest cost (Ropt ). Backwards planning transforms an expression to an equivalent form (RN ) that contains a target expression (T).
  • 70. Applications of backwards planning Indexes (e.g. b-tree indexes). An index is a derived data structure whose contents can be described as a relational expression (generally project-sort). When we are planning a query, it already exists (i.e. the cost has already been paid). Summary tables. A summary table is a derived data structure (generally filter-project-join-aggregate). Replicas with different physical properties (e.g. copy the table from New York to Tokyo, or copy the table and partition by month(orderDate), sort by productId). Incremental view maintenance. Materialized view V is populated from base table T. Yesterday, we populated V with V0 = Q(T0 ). Today we want to make its contents equal to V1 = Q(T1 ). Can we find and apply a delta query, dQ = Q(T1 - T0 )?
  • 71. Materialized views in Calcite { "schemas": { "name": "HR", "tables": [ { "name": "emp" } ], "materializations": [ { "table": "i_emp_job", "sql": "SELECT job, empno FROM emp ORDER BY job, empno" }, { "table": "add_emp_deptno", "sql": "SELECT deptno, SUM(sal) AS ss, COUNT(*) AS c FROM emp GROUP BY deptno" } ] } } /** Transforms a relational expression into a * semantically equivalent relational expression, * according to a given set of rules and a cost * model. */ public interface RelOptPlanner { /** Defines an equivalence between a table and * a query. */ void addMaterialization( RelOptMaterialization materialization); /** Finds the most efficient expression to * implement this query. */ RelNode findBestExp(); } /** Records that a particular query is materialized * by a particular table. */ public class RelOptMaterialization { public final RelNode tableRel; public final List<String> qualifiedTableName; public final RelNode queryRel; } You can define materializations in a JSON model, via the planner API, or via CREATE MATERIALIZED VIEW DDL (not shown).
  • 72. More about materialized views ● There are several algorithms to rewrite queries to match materialized views ● A lattice is a data structure to model a star schema ● Calcite has algorithms to recommend an optimal set of summary tables for a lattice (given expected queries, and statistics about column cardinality) ● Data profiling algorithms estimate the cardinality of all combinations of columns
  • 73. 10. Working with spatial data
  • 74. Spatial query Find all restaurants within 1.5 distance units of my current location: We cannot use a B-tree index (it can sort points by x or y coordinates, but not both) and specialized spatial indexes (such as R*-trees) are not generally available. restaurant x y Zachary’s pizza 3 1 King Yen 7 7 Filippo’s 7 4 Station burger 5 6 SELECT * FROM Restaurants AS r WHERE ST_Distance( ST_MakePoint(r.x, r.y), ST_MakePoint(6, 7)) < 1.5 • • • • Zachary’s pizza Filippo’s King Yen Station burger
  • 75. Hilbert space-filling curve ● A space-filling curve invented by mathematician David Hilbert ● Every (x, y) point has a unique position on the curve ● Points near to each other typically have Hilbert indexes close together
  • 76. • • • • Add restriction based on h, a restaurant’s distance along the Hilbert curve Must keep original restriction due to false positives Using Hilbert index restaurant x y h Zachary’s pizza 3 1 5 King Yen 7 7 41 Filippo’s 7 4 52 Station burger 5 6 36 Zachary’s pizza Filippo’s SELECT * FROM Restaurants AS r WHERE (r.h BETWEEN 35 AND 42 OR r.h BETWEEN 46 AND 46) AND ST_Distance( ST_MakePoint(r.x, r.y), ST_MakePoint(6, 7)) < 1.5 King Yen Station burger
  • 77. Telling the optimizer 1. Declare h as a generated column 2. Sort table by h Planner can now convert spatial range queries into a range scan Does not require specialized spatial index such as R*-tree Very efficient on a sorted table such as HBase There are similar techniques for other spatial patterns (e.g. region-to-region join) CREATE TABLE Restaurants ( restaurant VARCHAR(20), x DOUBLE, y DOUBLE, h DOUBLE GENERATED ALWAYS AS ST_Hilbert(x, y) STORED) SORT KEY (h); restaurant x y h Zachary’s pizza 3 1 5 Station burger 5 6 36 King Yen 7 7 41 Filippo’s 7 4 52
  • 78. 11. Research using Apache Calcite
  • 80. Yes, VLDB 2021! Go to the talk! 0900 Wednesday.
  • 83. Resources ● Calcite project https://calcite.apache.org ● Materialized view algorithms https://calcite.apache.org/docs/materialized_views.html ● JSON model https://calcite.apache.org/docs/model.html ● Lazy beats smart and fast (DataEng 2018) - MVs, spatial, profiling https://www.slideshare.net/julianhyde/lazy-beats-smart-and-fast ● Efficient spatial queries on vanilla databases (ApacheCon 2018) https://www.slideshare.net/julianhyde/spatial-query-on-vanilla-databases ● Graefe, McKenna. The Volcano Optimizer Generator, 1991 ● Graefe. The Cascades Framework for Query Optimization, 1995 ● Slideshare (past presentations by Julian Hyde, including several about Apache Calcite) https://www.slideshare.net/julianhyde