When Apache Spark Meets TiDB with Xiaoyu Ma
3 likes847 views
The document discusses TiSpark, an integration of Apache Spark with TiDB, highlighting its architecture and benefits for Hybrid Transactional/Analytical Processing (HTAP). It details how TiSpark allows complex SQL queries on TiKV, improves data retrieval efficiency, and supports various indexing strategies. The future developments for TiSpark including batch writing, JSON type support, and optimization strategies are also outlined.
![Construct Schema Transformation Rules
TiDB has totally different type system and infer rules
Aggregates Processing
select class, avg(score) from student
…….
group by class ;
Region 1
StudentId
[0-5000)
Region 2
StudentId
[5000-10000)
Region 3
StudentId
[10000-15000)
AVG(score) Group BY class
AVG are rewritten into SUM and COUNT
Map Task 1 Map Task 2
gRPC via Spark worker
Spark SQL plan received in TiStrategy
SUM(score) / COUNT(score) Group BY class
Spark Schema by its own type infer rules
[SUM, COUNT, class]
TiKV Schema to Spark Schema
[groupBy keys as bytes, SUM as Decimal, COUNT as BigInt ]
Reduce Task 1 Reduce Task 2
● After coprocessor preprocessing,
TiSpark still rely on normal Spark
aggregation strategy](https://image.slidesharecdn.com/9xiaoyuma-180614025349/85/When-Apache-Spark-Meets-TiDB-with-Xiaoyu-Ma-17-320.jpg)
When Apache Spark Meets TiDB with Xiaoyu Ma
- 1. When Apache Spark meets TiDB => TiSpark maxiaoyu@pingcap.com
- 2. Who am I ● Shawn Ma@PingCAP ● Tech Lead of OLAP Team ● Working on OLAP related products and features ● Previously tech lead of Big Data infra team@Netease ● Focus on SQL on Hadoop and Big Data related stuff
- 3. Agenda ● A little bit about TiDB / TiKV ● What is TiSpark ● Architecture ● Benefit ● What’s Next
- 4. What’s TiDB ● Open source distributed RDBMS ● Inspired by Google Spanner ● Horizontal Scalability ● ACID Transaction ● High Availability ● Auto-Failover ● SQL at scale ● Widely used in different industries, including Internet, Gaming, Banking, Finance, Manufacture and so on (200+ users)
- 5. A little bit about TiDB and TiKV TiKV TiKV TiKV TiKV Raft Raft Raft TiDB TiDB TiDB ... ...... ... ... Placement Driver (PD) Control flow: Balance / Failover Metadata / Timestamp request Stateless SQL Layer Distributed Storage Layer gRPC gRPC gRPC
- 6. TiKV: The whole picture Client Store 1 Region 1 Region 3 Region 5 Region 4 Store 3 Region 3 Region 5 Region 2 Store 2 Region 1 Region 3 Region 2 Region 4 Store 4 Region 1 Region 5 Region 2 Region 4 RPC RPC RPC RPC TiKV node 1 TiKV node 2 TiKV node 3 TiKV node 4 Placement Driver PD 1 PD 2 PD 3 Raft Group TiKV is powered by RocksDB
- 7. What is TiSpark ● TiSpark = Spark SQL on TiKV ○ Spark SQL directly on top of a distributed Database Storage ● Hybrid Transactional/Analytical Processing (HTAP) rocks ○ Provide strong OLAP capacity together with TiDB
- 8. What is TiSpark ● Complex Calculation Pushdown ● Key Range pruning ● Index support ○ Clustered index / Non-Clustered index ○ Index Only Query ● Cost Based Optimization ○ Histogram ○ Pick up right Access Path
- 9. Spark Exec Architecture Spark Exec Spark Driver Spark Exec TiKV TiKV TiKV TiKV TiSpark TiSpark TiSpark TiSpark TiKV Placement Driver (PD) gRPC Distributed Storage Layer gRPC retrieve data location retrieve real data from TiKV
- 10. Architecture ● On Spark Driver ○ Translate metadata from TiDB into Spark meta info ○ Transform Spark SQL logical plan, pick up elements to be leverage by storage (TiKV) and rewrite the plan ○ Locate Data based on Region info from Placement Driver and split partitions; ● On Spark Executor ○ Encode Spark SQL plan into TiKV’s coprocessor request ○ Decode TiKV / Coprocessor result and transform result into Spark SQL Rows
- 11. How everything made possible ● Extension points for Spark SQL Internal ● Extra Strategies allow us to inject our own physical executor and that’s what we leveraged for TiSpark ● Trying best to keep Spark Internal untouched to avoid compatibility issue
- 12. How everything made possible ● A fat java client module, paying the price of bypassing TiDB ○ Parsing Schema, Type system, encoding / decoding, coprocessor ○ Almost full featured TiKV client (without write support for now) ○ Predicates / Index - Key Range related logic ○ Aggregates pushdown related ○ Limit, Order, Stats related ● A thin layer inside Spark SQL ○ TiStrategy for Spark SQL plan transformation ○ And other utilities for mapping things from Spark SQL to TiKV client library ○ Physical Operators like IndexScan ○ Thin enough for not bothering much of compatibility with Spark SQL
- 13. Too Abstract? Let’s get concrete select class, avg(score) from student WHERE school = ‘engineering’ and lottery(name) = ‘picked’ and studentId >= 8000 and studentId < 10100 group by class ; ● Above is a table on TiDB named student ● Clustered index on StudentId and a secondary index on School column ● Lottery is an Spark SQL UDF which pick up a name and output ‘picked’ if RNG decided so
- 14. Construct Tasks Predicates Processing WHERE school = ‘engineering’ and lottery(name) = ‘picked’ and studentId >= 8000 and studentId < 10100 Region 1 StudentId [0-5000) Region 2 StudentId [5000-10000) Region 3 StudentId [10000-15000) StudentId >= 8000 StudentId < 10100 Key Range: [8000, 10100) Predicates are converted into key ranges based on indexes Spark Task 1 Region2 [8000, 10000) COP Request Spark Task 2 Region3 [10000, 10100) COP Request 1. Append remaining predicates if supported by coprocessor 2. Push back whatever needs to be computed by Spark SQL, e.g. UDFs, prefix index predicates 3. Cut them into tasks according to Region／Range 4. Encode into coprocessor request gRPC via Spark worker school = ‘engineering’ School = ‘engineering’ Lottery(name) = ‘picked’
- 15. WHERE school = ‘engineering’ and lottery(name) = ‘picked’ and (studentId >= 8000 and studentId < 10100) Index Scan TiKV Region Data TiKV Region Data TiKV Region Data Index Data for student_school Row Data for student Executor [1,5) 5,7,9 10 88Batch Scan for index according to predicates range Sort and cut row keys into ranges according to Key range in region ● Secondary Index is encode as key-value pair ○ Key is comparable bytes format of all index keys in defined order ○ Value is the row ID pointing to table row data ● Reading data via Secondary Index usually requires a double read. ○ First, read secondary index in range just like reading primary keys in previous slide. ○ Shuffle Row IDs according to region ○ Sort all row IDs retrieved and combine them into ranges if possible ○ Encoding row IDs into row keys for the table ○ Send those mini requests in batch concurrently ● Optimize away second read operation ○ If all required column covered by index itself already 1,2,3,4,5,7,8,10,88 Executor
- 16. Index Selection WHERE school = ‘engineering’ and lottery(name) = ‘picked’ and (studentId >= 8000 and studentId < 10100) or studentId in (10323, 10327) Histogram Clustered Index on StudentID + predicates related StudentId matched Secondary Index on School + predicates related School matched 1k Rows 800 Rows 1K * Clustered Index Access Cost < 800 * Secondary Index Access Cost ● If the columns referred are all covered by index, then instead of retrieving actual rows, we apply index only query and cost function will be different ● If histogram not exists, TiSpark using pseudo selection logic.
- 17. Construct Schema Transformation Rules TiDB has totally different type system and infer rules Aggregates Processing select class, avg(score) from student ……. group by class ; Region 1 StudentId [0-5000) Region 2 StudentId [5000-10000) Region 3 StudentId [10000-15000) AVG(score) Group BY class AVG are rewritten into SUM and COUNT Map Task 1 Map Task 2 gRPC via Spark worker Spark SQL plan received in TiStrategy SUM(score) / COUNT(score) Group BY class Spark Schema by its own type infer rules [SUM, COUNT, class] TiKV Schema to Spark Schema [groupBy keys as bytes, SUM as Decimal, COUNT as BigInt ] Reduce Task 1 Reduce Task 2 ● After coprocessor preprocessing, TiSpark still rely on normal Spark aggregation strategy
- 18. Benefit ● Analytical / Transactional support all on one platform ○ No need for ETL and query data in real-time ○ High throughput and consistent snapshot read from database ○ Simplify your platform and reduce maintenance cost ● Embrace Apache Spark and its eco-system ○ Support of complex transformation and analytics beyond SQL ○ Cooperate with other projects in eco-system (like Apache Zeppelin) ○ Apache Spark bridges your data sources
- 19. Ease of Use ● Working on your existing Spark Cluster ○ Just a single jar like other Spark connector ● Workable as standalone application, spark-shell, thrift-server, pyspark and R ● Work just like another data source val ti = new org.apache.spark.sql.TiContext(spark) // Map all TiDB tables from database tpch as Spark SQL tables ti.tidbMapDatabase("sampleDB") spark.sql("select count(*) from sampleTable").show
- 20. What’s Next ● Batch Write Support (writing directly as TiKV native format) ● JSON Type support (since TiDB already supported) ● Partition Table support (both Range and Hash) ● Join optimization based on range and partition table ● (Maybe) Join Reorder with TiDB’s own Histogram ● Another separate columnar storage project using Spark as its execution engine (not released yet)