MySQL Failover and Orchestrator
4 likes1,872 views
The document summarizes a presentation on MySQL failover and orchestrator software. It discusses how orchestrator can help manage MySQL failover across servers to provide high availability and prevent downtime. Orchestrator automatically discovers MySQL topology, monitors servers, and can relocate masters and notify administrators of failures. It provides both a GUI and CLI to manage failover tasks and supports configurations from simple to complex multi-server setups.
MySQL Failover and Orchestrator
- 2. Content • Handling failover with MySQL • Downtime & Requirements • MySQL Clustering solutions • Non-clustering solutions and considerations • Orchestrator • Questions 105/07/2017 Barcelona MySQL Meetup
- 3. Is Downtime Acceptable? • Do you have a system that needs to run 24 x 7 ? • Not everyone does • If you have a website then generally downtime is not acceptable 205/07/2017 Barcelona MySQL Meetup
- 4. Requirements Goal: Run 24 x 7 x 365 with no downtime • Is this really necessary? • If you ask management they’ll always say yes… • What is the cost? • Shorter downtime requirements mean more effort spent to achieve that • How do you reliably detect failure? Hard problem to solve If you accept downtime how much can you really tolerate? • 1s, 5s, 30s, 1min ? 305/07/2017 Barcelona MySQL Meetup
- 5. What options are available? • MySQL Cluster • carrier grade • very high uptime • Not InnoDB – specialised workloads • Galera • Often with asynchronous replication between datacentres • InnoDB Cluster • Very new • All require clients to take action on failure of a node • If you use a proxy that can fail too… 405/07/2017 Barcelona MySQL Meetup
- 6. What options are available? “Cluster solutions” • Do not work well cross-DC due to latency • If you accept writes into multiple masters there’s a chance of conflict • Slows things down • InnoDB Cluster now does not recommend this behaviour – requires care • Only small setups work in a single data-centre so adaptation here is also needed • Cluster setups do not scale easily to 10 or more servers 505/07/2017 Barcelona MySQL Meetup
- 7. What options are available? • Standard MySQL, MariaDB, Amazon RDS, Google Cloud SQL, … • Read scale-out • Asynchronous replication • Semi-sync helps improve performance and ensure data is “somewhere else” when acknowledging a transaction • If you are out of the cloud then: different setups • SBR or RBR? • No GTID, Oracle or MySQL GTID? • Optional semi-sync? • If you are out of the cloud then: do it yourself • MHA • MariaDB Replication Manager • Orchestrator 605/07/2017 Barcelona MySQL Meetup
- 9. Orchestrator • Handles master failover, but more… • GUI to manage and visualise topology – very handy • CLI to do the same things – good for scripting • API calls to run at a distance (more generic interface) • Needs a DB backend to store state. • Normally MySQL but can be SQLite 805/07/2017 Barcelona MySQL Meetup
- 10. Orchestrator • Written by Shlomi Noach who works at github • He worked previously at booking.com and introduced us to orchestrator, previously working at outbrain. 905/07/2017 Barcelona MySQL Meetup
- 11. Orchestrator What failures does it handle? • Master failures – needs to talk to external systems • Intermediate master failures – can handle on its own • Does not care about slaves or applications • Works with GTID: Oracle or MariaDB • Works without using GTID: Can add Pseudo-GTID (events injected on the master are used to find a match) so no need to migrate to GTID if not wanted • Handles multi-level topologies 1005/07/2017 Barcelona MySQL Meetup
- 15. Orchestrator CLI Over 100 commands you can use • E.g. • relocate • discover • begin-downtime, end-downtime • topology 1405/07/2017 Barcelona MySQL Meetup
- 19. Orchestrator Setup • Source at github.com/github/orchestrator • Binaries written in go • Daemon runs web service and discovery, client on each MySQL server • State stored in MySQL / SQLite • Single json configuration file: /etc/orchestrator.conf.json • How to reach backend database (stores state) • How to recognise delay • Most defaults are good to get you going • Which systems you want to trigger recovery on • Hooks to handle recovery (talking to external systems) • If you need help please ask 1805/07/2017 Barcelona MySQL Meetup
- 20. Orchestrator Characteristics • Discover one server in your cluster and orchestrator will find the others • Detects new servers in the cluster automatically • Notifies you of problems seen • Recovery is optional (per cluster) • Optional selection of candidate masters or servers to blacklist • Global ON / OFF switch – handy if several failures happen at once • For paranoid DBAs, so far orchestrator has always done the right thing 1905/07/2017 Barcelona MySQL Meetup
- 21. Orchestrator HA ? Orchestrator can be run in HA mode • Multiple daemons will co-operate so if one fails another one takes over (they share the database backend) • Use a load balancer to provide an HA GUI service • Use nginx (or similar) for authentication and TLS if needed • Upgrades are easier • Replicate the orchestrator MySQL backend to not lose data 2005/07/2017 Barcelona MySQL Meetup
- 22. Does it Scale? Yes • Booking.com has a large installation with a single cluster monitoring thousands of MySQL servers • Recommended by YouTube for managing Vitess servers • Quite a number of other users but they are not very visible 2105/07/2017 Barcelona MySQL Meetup
- 23. Future work • Simplify configuration and setup so more people can use it • Improve scalability • Make it work on Amazon RDS • Spread the word… 05/07/2017 Barcelona MySQL Meetup 22
- 24. Further help needed? • github.com/github/orchestrator • for Issues (Problems / Questions) and Pull Requests (patches) • Google Group: Orchestrator MySQL • https://groups.google.com/forum/#!forum/orchestrator-mysql • Feel free to contact me and I will try to help provide pointers 2305/07/2017 Barcelona MySQL Meetup
- 25. Oh, and Booking.com is hiring! • Almost any role: • MySQL Engineer / DBA • System Administrator • System Engineer • Site Reliability Engineer • Developer • Designer • Technical Team Lead • Product Owner • Data Scientist • And many more… • https://workingatbooking.com/ 05/07/2017 Barcelona MySQL Meetup 24