Integrating best of breed open source tools to vitess orchestrator pleu21
0 likes82 views
Vitess is an open source database clustering system for horizontally scaling MySQL and MariaDB. It uses a control plane including a proxy server (vtgate), orchestration tools like XtraBackup and Orchestrator (vtorc), and supports features like sharding, replication, and online DDL using tools like gh-ost and pt-online-schema change. The presentation demonstrated how Vitess routes queries to shards in a transparent way using its topology configuration and control servers.
![XtraBackup
● For MySQL 5.7 and MySQL 8.0
● MariaDB 10.3 is not compatible with XtraBackup.
$ vtctl Backup <tablet-alias>
$ vtctl BackupShard [-allow_master=false]
<keyspace/shard>](https://image.slidesharecdn.com/integratingbestofbreedopensourcetoolstovitessorchestrator-pleu21-210816061951/85/Integrating-best-of-breed-open-source-tools-to-vitess-orchestrator-pleu21-23-320.jpg)
Integrating best of breed open source tools to vitess orchestrator pleu21
- 1. Integrating Best of Breed Open Source Tools to Vitess: Orchestrator Alkin Tezuysal PlanetScale Perconalive, May 2021
- 2. About me Sr. Technical Manager at PlanetScale Maintainer for Vitess Open source database evangelist previously at Percona, Pythian and others with Enterprise Background Born to Sail, Forced to Work! @ask_dba
- 3. Founded Feb. 2018 by co-creators of Vitess ~50 employees HQ Mountain View, 100% remote team @ask_dba
- 4. Vitess A database clustering system for horizontal scaling of MySQL / MariaDB ● CNCF graduated project ● Open source, Apache 2.0 licence ● Contributors from around the community ● Written in Golang @ask_dba
- 5. Agenda Vitess architecture overview Vitess use cases and sharding Vitess Open Source Tools ● XtraBackup - Percona ● pt-online-schema change - Percona ● gh-ost - Github ● Orchestrator - Shlomi Noach ● And more ... @ask_dba
- 6. Vitess architecture basics How the Vitess architecture enables transparent database infrastructure operations @ask_dba
- 7. Glossary ● Keyspace: Logical database (sharded) ○ Keyspace ID ○ Primary Vindex ○ Vindex ● VTGate: Proxy server ● VTTablet : Backend server ● Topology : Configuration server (etcd, zookeper) @ask_dba
- 8. Vitess architecture basics Consider a common replication cluster @ask_dba
- 9. Vitess architecture basics Each MySQL server is assigned a vttablet - A daemon/sidecar - Controls the mysqld process - Interacts with the mysqld server - Typically on same host as mysqld @ask_dba
- 10. Vitess architecture basics In production you have multiple clusters @ask_dba
- 11. Vitess architecture basics User and application traffic is routed via vtgate - A smart, stateless proxy - Speaks the MySQL protocol - Impersonates as a monolith MySQL server - Relays queries to vttablets @ask_dba
- 12. Vitess architecture basics A vitess deployment will run multiple vtgate servers for scale out @ask_dba
- 13. Vitess architecture basics vtgate must transparently route queries to correct clusters, to relevant shards app app commerce shard 0 commerce shard 1 internal unsharded ? @ask_dba
- 14. Vitess architecture basics Queries route based on schema & sharding scheme app app commerce shard 0 commerce shard 1 internal unsharded USE commerce; SELECT order_id, price FROM orders WHERE customer_id=4; @ask_dba
- 15. Vitess architecture basics topo: distributed key/value store - Stores the state of vitess: schemas, shards, sharding scheme, tablets, roles, etc. - etcd/consul/zookeeper - Small dataset, mostly cached by vtgate commerce shard 0 commerce shard 1 internal unsharded @ask_dba
- 16. vtctld: control daemon - Runs ad hoc operations - API server - Reads/writes topo - Uses locks - Operates on tablets Vitess architecture basics commerce shard 0 commerce shard 1 internal unsharded @ask_dba
- 17. Vitess knows Vitess keeps known schemas, shards, clusters, server roles, all in topo It keeps a state @ask_dba
- 18. Vitess Controlplane Includes ● Proxy server (vtgate) ● Managed Backup and Recovery (XtraBackup) ● Integrated failover (a.k.a Orchestrator/vtorc) ● Sharding Schemes (Horizontal) ● Advanced Replication (Vreplication,Vstream) ● Online DDL (gh-ost, pt-osc) ● And more @ask_dba
- 20. Supported Backend Databases ● MySQL 5.7 / 8.0 ● MariaDB 10.3 ● Postgresql @ask_dba
- 21. Vitess Use Cases and Sharding ● Part or entire application scaling ● Management of existing MySQL topology ● Sharding and resharding ● Minimizing backup/recovery scenarios
- 22. Vitess Open Source Tools ● XtraBackup - Percona ✔ ● pt-online-schema change - Percona ✔ ● gh-ost - Github ✔ ● Orchestrator - Shlomi Noach → vtorc - in progress ● And more ...
- 23. XtraBackup ● For MySQL 5.7 and MySQL 8.0 ● MariaDB 10.3 is not compatible with XtraBackup. $ vtctl Backup <tablet-alias> $ vtctl BackupShard [-allow_master=false] <keyspace/shard>
- 24. pt-online-schema ● Supported since version 9.0.0 GA ● Automatic handling of flags ● Uses @@ddl_strategy session variable mysql> set @@ddl_strategy='pt-osc --max-load Threads_running=200'; mysql> ALTER TABLE user MODIFY user_id BIGINT $ vtctlclient ApplySchema -ddl_strategy "pt-osc --max-load Threads_running=200" -sql "ALTER TABLE user MODIFY user_id INT " commerce
- 25. gh-ost ● Supported since version 9.0.0 GA ● Automatic handling of flags ● Uses @@ddl_strategy session variable mysql> set @@ddl_strategy='gh-ost --max-load Threads_running=200'; mysql> ALTER TABLE user MODIFY user_id BIGINT $ vtctlclient ApplySchema -ddl_strategy "gh-ost --max-load Threads_running=200" -sql "ALTER TABLE user MODIFY user_id INT " commerce
- 26. Choosing a DDL strategy ● Online / VReplication ● gh-ost or pt-osc Read more on Vitess Documentation.
- 27. Orchestrator - Vtorc ● Experimental ● Development in progress, help needed. Orchestrator Highlights ● Developed by Shlomi Noach , several contributors ● MySQL Replication Topology Management and High Availability solution ● Provides HTTP API & Web Interface ● Supports Discovery, Refactoring, Recovery with pre/post hooks
- 29. Resources Docs: vitess.io/docs/ Code: github.com/vitessio/vitess Slack: vitess.slack.com @ask_dba