TiDB Introduction - San Francisco MySQL Meetup
1 like297 views
This document provides an overview and agenda for introducing TiDB, an open source distributed SQL database inspired by Google's Spanner and F1 projects. The summary includes: - TiDB is a distributed SQL database that is compatible with MySQL and provides horizontal scalability, high availability, and strong consistency with its key components TiDB, TiKV, and PD. - The agenda covers an introduction to PingCAP, the company behind TiDB, a technical walkthrough of the TiDB architecture, and a use case example with Mobike, one of TiDB's customers with over 200 million users. - A live demo of running TiDB on Google Kubernetes Engine is also included on the agenda along with discussions of
![TiKV: The Foundation [in CNCF]
RocksDB
Raft
Transaction
Txn KV API
Coprocessor
API
RocksDB
Raft
Transaction
Txn KV API
Coprocessor
API
RocksDB
Raft
Transaction
Txn KV API
Coprocessor
API
Raft
Group
Client
gRPC
TiKV Instance TiKV Instance TiKV Instance
gRPC gRPC
PD Cluster](https://image.slidesharecdn.com/tidbintroductionmysql1-181011163820/85/TiDB-Introduction-San-Francisco-MySQL-Meetup-15-320.jpg)
![Index Structure
Row:
Key: tablePrefix_rowPrefix_tableID_rowID (IDs are assigned by TiDB, all int64)
Value: [col1, col2, col3, col4]
Index:
Key: tablePrefix_idxPrefix_tableID_indexID_ColumnsValue_rowID
Value: [null]
Keys are ordered by byte array in TiKV, so can support SCAN
Every key is appended a timestamp, issued by Placement Driver](https://image.slidesharecdn.com/tidbintroductionmysql1-181011163820/85/TiDB-Introduction-San-Francisco-MySQL-Meetup-33-320.jpg)
TiDB Introduction - San Francisco MySQL Meetup
- 1. Introducing TiDB (For those coming from MySQL..) Make Data Creative Morgan Tocker (@pingcap; @morgo) October, 2018
- 2. ● History and Community ● Technical Walkthrough ● Use Case with Mobike ● Live Demo: TiDB on GKE ● MySQL Compatibility ● Q&A Agenda
- 3. ● Sr Product / Community Manager ● ~15+ years MySQL Experience ○ MySQL AB, Sun Microsystems, Percona, Oracle ● Previously Product Manager for MySQL Server A Little About Me...
- 4. A Little About PingCAP... ● Founded in April 2015 by 3 infrastructure engineers ● Offices in China and North America ● Remote Friendly! ○ I work from here ➡
- 5. Recent News
- 6. Recent News
- 7. Our Product is The TiDB Platform ● TiDB platform: (Ti = Titanium) ○ TiDB (stateless SQL layer compatible with MySQL) ○ TiKV (distributed transactional key-value store) ○ TiSpark (Apache Spark plug-in on top of TiKV) ● Open source from Day 1 ○ GA 1.0: October 2017 ○ GA 2.0: April 2018
- 8. TiDB is a NewSQL Database ● 1960s: First Gen Databases ● 1970s: Relational Model + SQL ● 2000s: Sharding + Memcached ● 2010s: NoSQL ● 2010s+: NewSQL Inspired by Google Spanner / F1 MySQL Compatible
- 9. ● Hybrid OLTP & OLAP (Minimize ETL) ● Horizontal Scalability ● MySQL Compatible ● Distributed Transaction (ACID Compliant) ● High Availability ● Cloud-Native TiDB Core Features
- 10. Architecture SparkSQL TiDB TiDB Worker Spark Driver TiKV Cluster (Storage) Metadata TiKV TiKV TiKV Data location Job TiSpark DistSQL API TiKV TiDB TSO/Data location Worker Worker Spark Cluster TiDB Cluster TiDB DistSQL API PD PD Cluster TiKV TiKV TiDB KV API MySQL MySQL PD PD
- 11. 2018 PingCAP Stars ● TiDB: 15,000+ ● TiKV: 3700+ Contributors ● TiDB: 200+ ● TiKV: 100+ Community
- 14. Platform Architecture TiDB TiDB Worker Spark Driver TiKV Cluster (Storage) Metadata TiKV TiKV TiKV Data location Job TiSpark DistSQL API TiKV TiDB TSO/Data location Worker Worker Spark Cluster TiDB Cluster TiDB DistSQL API PD PD Cluster TiKV TiKV TiDB KV API MySQL MySQL SparkSQL PD PD SparkSQL
- 15. TiKV: The Foundation [in CNCF] RocksDB Raft Transaction Txn KV API Coprocessor API RocksDB Raft Transaction Txn KV API Coprocessor API RocksDB Raft Transaction Txn KV API Coprocessor API Raft Group Client gRPC TiKV Instance TiKV Instance TiKV Instance gRPC gRPC PD Cluster
- 16. TiDB: OLTP + Ad Hoc OLAP Node1 Node2 Node3 Node4 MySQL Network Protocol SQL Parser Cost-based Optimizer Distributed Executor (Coprocessor) ODBC/JDBC MySQL Client Any ORM which supports MySQL TiDB TiKV
- 17. ID Name Email 1 Edward h@pingcap.com 2 Tom tom@pingcap.com ... user/1 Edward,h@pingcap.com user/2 Tom,tom@pingcap.com ... In TiKV -∞ +∞ (-∞, +∞) Sorted map “User” Table TiDB: Relational -> KV Some region...
- 18. SQL -> Parser -> Coprocessor
- 19. TiSpark: Complex OLAP Spark ExecSpark Exec Spark Driver Spark Exec TiKV TiKV TiKV TiKV TiSpark TiSpark TiSpark TiSpark TiKV Placement Driver (PD) gRPC Distributed Storage Layer gRPC retrieve data location retrieve real data from TiKV
- 21. 2018 PingCAP Who’s using TiDB? 300+ Companies
- 22. 2018 PingCAP 1. MySQL Scalability 2. Hybrid OLTP/OLAP Architecture 3. Unifying Data Storage/Management Three Big Use Cases
- 23. Mobike + TiDB ● 200 million users ● 200 cities ● 9 milllion smart bikes ● ~30 TB / day
- 24. ● Locking and unlocking of smart bikes generate massive data ● Smooth experience is key to user retention ● TiDB supports this system by alerting administrators when success rate of locking/unlocking drops, within minutes ● Quickly find malfunctioning bikes Scenario #1: Locking/Unlocking
- 25. ● Synchronize TiDB with MySQL instances using Syncer (proprietary tool) ● TiDB + TiSpark empower real-time analysis with horizontal scalability ● No need for Hadoop + Hive Scenario #2: Real-Time Analysis
- 26. ● An innovative loyalty program that must be on 24 x 7 x 365 ● TiDB handles: ○ High-concurrency for peak or promotional season ○ Permanent storage ○ Horizontal scalability ● No interruption as business evolves Scenario #3: Mobike Store
- 27. TiDB on GKE Demo
- 29. ● Compatible with MySQL 5.7 ○ Joins, Subqueries, DML, DDL etc. ● On the roadmap: ○ Views, Window Functions, GIS ● Missing: ○ Stored Procedures, Triggers, Events Summary pingcap.com /docs/sql/mysql-compatibility/
- 30. ● Some features work differently ○ Auto Increment ○ Optimistic Locking ● TiDB works better with smaller transactions ○ Recommended to batch updates, deletes, inserts to 5000 rows Nuanced
- 31. Thank You! Twitter: @PingCAP; @morgo https://github.com/pingcap (Give us a Watch/Star!) Morgan Tocker (morgan@pingcap.com) Early Sign-up: www.pingcap.com/tidb-academy/
- 32. ● Hash Join (fastest; if table <= 50 million rows) ● Sort Merge Join (join on indexed column or ordered data source) ● Index Lookup Join (join on indexed column; ideally after filter, result < 10,000 rows) Chosen based on Cost-based Optimizer: Join Support Network cost Memory cost CPU cost
- 33. Index Structure Row: Key: tablePrefix_rowPrefix_tableID_rowID (IDs are assigned by TiDB, all int64) Value: [col1, col2, col3, col4] Index: Key: tablePrefix_idxPrefix_tableID_indexID_ColumnsValue_rowID Value: [null] Keys are ordered by byte array in TiKV, so can support SCAN Every key is appended a timestamp, issued by Placement Driver
- 34. ● Complex calculation pushdown ● Key-range pruning ● Index support: ○ Clustered index / non-clustered index ○ Index-only query optimization ● Cost-based optimization: ○ Stats gathered from TiDB in histogram TiSpark: Features
- 35. PD: Dynamic Split and Merge Region A Region A Region B Region A Region A Region B Split Region A Region A Region B Merge TiKV_1 TiKV_2 TiKV_2TiKV_1
- 36. PD: Hotspot Removal *Region A* *Region B* Region A Region B Workload *Region A* Region B Region A *Region B* Workload Workload Hotspot Schedule (Raft leader transfer) TiKV_1 TiKV_2 TiKV_2TiKV_1
- 37. Geo-Replication + Data Location *Region A* Region B Region A Region B Seattle_1 Seattle_2 Region A *Region B* New York_1 *Region A* Region B Region A *Region B* Seattle_2Seattle_1 Region A Region B New York_1
- 38. ● Timestamp Oracle service (from Google’s Percolator paper) ● 2-Phase commit protocol (2PC) ● Problem: Single point of failure ● Solution: Placement Driver HA cluster ○ Replicated using Raft Transaction Model
- 39. ● Formal proof using TLA+ ○ a formal specification and verification language to reason about and prove aspects of complex systems ● Raft ● TSO/Percolator ● 2PC ● See details: https://github.com/pingcap/tla-plus Guaranteeing Correctness