MySQL HA Orchestrator Proxysql Consul.pdf

MySQL HA Using
Orchestrator, Proxysql
and Consul

Yunus Shaikh
● MySQL DBA II at Percona
● 15 Years of experience in System
and Database administration
● Enjoys discovering new open
source technologies
● Likes to play Cricket in free time

Kedar Vaijanapurkar
● Living in Vadodara, Gujarat, India
● Major components: SPOF (wife) and two HA (highly active) replicas.
● Educated Masters in Computer Applications (2006)
● Eventual DBA and loving it (VB/VC/C# >> Perl/PHP >> MySQL)
● Enjoys music, photography, chess, cricket, blogging and knowledge
sharing
● Knows a bit about MySQL ecosystem, still learning.
● Proud Perconian

● Introduction
● Whys of High Availability
● Building A High Availability Solution
● Questions
Agenda

Let's try to understand what is HA?
● Eliminate single point of failure
● Reduces interruptions and downtime
● Quick recovery during failures
● Scalable to handle high traffic
● Maintain high level of system uptime

Factors considered for High Availability
● Unplanned Downtime
○ Hardware failures
○ Software failures
● Planned Downtime
○ Hardware maintenance
○ Software maintenance

How many 9’s do you really need?

MySQL
● Async replication: Oldest, popular and
widely deployed, simple.
● Scale, Backup
● GTID v Non-GTID

● SPOF - Single Point of Failure
● Primary is a weak link.
● Downtime or degraded action for application.
MySQL (SPOF)
We need protection against SPOF to
● Let application continue writes
● Cleanup what’s bad in the environment
● Maintain architecture

● Editing app configuration to point to replica
● If we need to failover for any reason, we might need to
do it manually
○ More time in manual effort means more
downtime.
○ Writes need to be stopped during manual failover.
○ Human error prone.
Disadvantage
● How can we improve?

Orchestrator
● Thank you Shlomi
● Replication Topology Manager
● Discover, Visualize Topology
● Restructure Architecture
● Clustering support
● Free and Open Source
● https://github.com/openark/orchestrator

● Prerequisite
○ Orchestrator user
○ binlog & log_slave_updates
○ Master_info_repository - Table
● GTID / Non GTID Support
● Pseudo GTID
● MySQL or SQLite Backend options
● Percona Support
Orchestrator
CREATE USER 'orchestrator'@'orch_host' IDENTIFIED BY 'orch_topology_password';
GRANT SUPER, PROCESS, REPLICATION SLAVE, RELOAD ON *.* TO
'orchestrator'@'orch_host';
GRANT SELECT ON mysql.slave_master_info TO 'orchestrator'@'orch_host';
GRANT DROP ON _pseudo_gtid_.* TO 'orchestrator'@'orch_host';

● Orchestrator HA possible in 2 ways
○ Orchestrator RAFT
○ SHARED backend
● Nodes communicate with Raft consensus protocol
○ Why Raft?
● Only leader makes the changes (though everyone
runs discoveries)
● Little cross-node communication; fit for high
latency cross DC networks
Orchestrator (HA)

Orchestrator (Discovery)
- Image with orchestrator discovery | UI
- Image with orchestrator discovery | CLI

Orchestrator (Restructure)
- Image with instance promotion | UI
- Image with instance promotion | CLI

Orchestrator (Failure Detection)
● Naive Approach without orchestrator
○ Cannot contact master then failover
○ Can cause false positives
○ N checks in some intervals increases response
time.
● The holistic failure detection approach
○ Orchestrator cannot see master
○ Replicas cannot see the master.
● Recommended for faster detection
○ Slave_net_timeout
○ Master connect retry and master_retry count.

Orchestrator (Candidate Selection)
● Promote replica with 0 replication lag (least lag / first
to catch-up)
● Don’t promote servers without bin logs
● Replication filters
● Mysql versions
● Register_candidate

Orchestrator (Recovery)
● Pre-and-Post recovery hooks (once, sequentially)
● Healing of topology
● Events
○ OnFailureDetectionProcesses
○ PreGracefulTakeoverProcesses
○ PreFailoverProcesses
○ PostFailoverProcesses
○ PostUnsuccessfulFailoverProcesses
○ PostMasterFailoverProcesses
○ PostIntermediateMasterFailoverProcesses
○ PostGracefulTakeoverProcesses

Orchestrator (Recovery)
● Automatic recovery
● Manual recovery
{
"RecoveryPeriodBlockSeconds": 3600,
"RecoveryIgnoreHostnameFilters": [],
"RecoverMasterClusterFilters": [
"thiscluster",
"thatcluster"
],
"RecoverIntermediateMasterClusterFilters": [
"*"
],
}

Something is still lacking?
● We have failover tool now with orchestrator
● MySQL is HA, Orchestrator is HA
What is missing?
● We will still have to manually point application after
failover.
● No use of auto failover. As it needs manual intervention
to edit application config to point to new master
● No scaling/distributed read/writes

ProxySQL
● Proxy for database
● Connection pooling / multiplexing
● Load balancing
● Query Caching / Rewrite / Blocks
● Facilitates read write splits
● Easy configuration using SQL
interface with tables
● Clustering support

ProxySQL (Implementation)
On Application Node On Dedicated Node

● Tables and variables to configure
○ mysql_users: define DB users
○ mysql_servers: Backend MySQL Servers
○ mysql_replication_hostgroups: MySQL replication clusters
with servers in RW or RO mode
○ mysql_query_rules: Rules for query to backend databases.
ProxySQL (Configuration)

● Available since version 1.4.x
● ProxySQL nodes watches each other for changes in
configuration checksums (runtime_checksums_values)
● Monitors + Reconfigure - 4 tables and global variables
○ mysql_query_rules
○ mysql_servers
○ mysql_users
○ proxysql_servers
ProxySQL (HA)

What do we have now?
Orchestrator: Hello! Can you talk?
ProxySQL: Not yet!
Orchestrator: Let me hook it up.

● No native communication
● ProxySQL: read_only monitor
● Orchestrator: Failover
● Orchestrator comes to rescue with “Hooks”
○ Scripts executed on events
○ Pre and Post failover execution on recovery
ProxySQL + Orchestrator
Orchestrator: I see fire, inside the mountains
ProxySQL: I see fire, burning the trees
Orchestrator Hooks: And I hope that you remember me.

● Don’t let the dead come alive - STONITH (via hooks)
● Can be used to generate alerts!
● Hooks are executed only once.
○ If ProxySQL is OoO, hook fails!
■ Who can be our messenger?
ProxySQL + Orchestrator (...)

Consul
● Built on Golang
● Service discovery tool
● Key value store
● Lightweight
● Native support in Orchestrator
● HA
● 3 Default ports
○ 8400 - RPC
○ 8500 - HTTP
○ 8600 - DNS

Consul (features)
● Service Discovery − Using either DNS or HTTP, applications
can easily find the services they depend upon.
● Health Check Status − It can provide any number of health
checks. It is used by the service discovery components to
route traffic away from unhealthy hosts.
● Key/Value Store − It can make use of Consul's hierarchical
key/value store for any number of purposes, including
dynamic configuration, feature flagging, coordination, leader
election, etc.
● Multi Datacenter Deployment − Consul supports multiple
datacenters. It is used for building additional layers of
abstraction to grow to multiple regions.
● Web UI − Consul provides its users a beautiful web interface
using which it can be easy to use and manage all of the
features in consul.

● Configure consul on (all) Orchestrator node(s) as Server
● Local install -> faster communication, 0 network latency
● Orchestrator config
“KVClusterMasterPrefix”: ”mysql/master”
“ConsulAddress”: “127.0.0.1:8500”
● Restart orchestrator
● orchestrator-client -c submit-masters-to-kv-stores
Consul (talking to Orchestrator)

● What Key Value
○ Orchestrator writes master info to Consul as per
defined KVClusterMasterPrefix Key.
■ IPV4, IPV6, Host, Port
Consul (What)

● Configure consul on (all) ProxySQL node(s) as
Agent joining Consul server on Orchestrator.
● Orchestrator -> Consul (S) -> Consul (A) ->
ProxySQL
● Apply changes using consul-template.
Consul (talking to ProxySQL)

● Consul Template is a simple and powerful tool that populates
values from consul into the file system.
● Consul templates can also execute some commands.
● Runs as a daemon and queries consul
Consul Template
# Snip from Template
REPLACE mysql_servers (hostgroup_id,hostname,port) values (2, "{{ key
$get_ip }}", "{{ key $get_port }}")
…
# Snip from config
command = "/bin/bash -c 'mysql -P 6032 -h 127.0.0.1 -u admin -padmin
< /opt/consul-template/templates/proxysql.sql'"

● Raft consensus protocol
● Leader elected, all requests processed by leader.
Consul (HA)

Orchestrator
+
ProxySQL
+
Consul

Consul
● Orchestrator writes to consul.
● Consul acts as source of truth to proxysql.
● Consul-template can be used as a script to make
changes in proxysql.
● Service discovery can be used along with DNS using
dnsmasq for discovering proxysql servers
○ Load balancing
○ Fault Tolerance

Service Discovery
root@localhost consul.d]# cat proxysql.json
{"services": [
{
"id": "proxy1",
"name": "proxysql",
"address": "192.168.103.30",
"tags": ["mysql"],
"port": 6033,
"check": {
"args": ["/usr/bin/mysqladmin", "ping", "--host=192.168.103.30", "--port=6033", "--user=test",
"--password=test@123"],
"interval": "3s"}
},
{
"id": "proxy2",
"name": "proxysql",
"address": "192.168.103.40",
"tags": ["mysql"],
"port": 6033,
"check": {
"args": ["/usr/bin/mysqladmin", "ping", "--host=192.168.103.40", "--port=6033", "--user=test",
"--password=test@123"],
"interval": "3s"}
}
]
}

Orchestrator hook and Consul-template
Hook
"PostGracefulTakeoverProcesses": [
"echo 'Planned takeover complete' >> /tmp/recovery.log",
"/root/orchestrator/post_failover2.sh {successorHost} {successorPort} {failedHost} {failedPort} >>
/tmp/recovery.log"
],
cat /root/orchestrator/post_failover2.sh
#!/bin/bash
IFS=' '
echo "-------------------------------"
echo $@
echo "-----------------------------------------"
consul kv put mysql/slave/percona_live/$3/ip $3
consul kv put mysql/slave/percona_live/$3/port $4
consul kv delete -recurse mysql/slave/percona_live/$1/

Logs for Orchestrator hook
Detected DeadMaster on 192.168.103.100:3307. Affected replicas: 1
Will recover from DeadMaster on 192.168.103.100:3307
Recovered from DeadMaster on 192.168.103.100:3307. Failed: 192.168.103.100:3307; Promoted:
192.168.103.10:3306
(for all types) Recovered from DeadMaster on 192.168.103.100:3307. Failed: 192.168.103.100:3307; Successor:
192.168.103.10:3306
Planned takeover about to take place on 192.168.103.10:3306. Master will switch to read_only
Detected DeadMaster on 192.168.103.10:3306. Affected replicas: 1
Will recover from DeadMaster on 192.168.103.10:3306
Recovered from DeadMaster on 192.168.103.10:3306. Failed: 192.168.103.10:3306; Promoted:
192.168.103.20:3306
(for all types) Recovered from DeadMaster on 192.168.103.10:3306. Failed: 192.168.103.10:3306; Successor:
192.168.103.20:3306
Planned takeover complete
-------------------------------
192.168.103.20 3306 192.168.103.10 3306
-----------------------------------------
Success! Data written to: mysql/slave/percona_live/192.168.103.10/ip
Success! Data written to: mysql/slave/percona_live/192.168.103.10/port
Success! Deleted keys with prefix: mysql/slave/percona_live/192.168.103.20/

How does consul-template work
template {
source = "/opt/consul-template/templates/proxysql.ctmpl"
destination = "/opt/consul-template/templates/proxysql.sql"
command = "/bin/bash -c 'mysql -P 6032 -h 127.0.0.1 -u admin -padmin <
/opt/consul-template/templates/proxysql.sql'"
command_timeout = "60s"
perms = 0644
backup = true
wait = "2s:6s"
}
[root@localhost consul-template]# tail -n 6 consul-template.log
2022/04/02 15:14:05.289735 [INFO] (runner) initiating run
2022/04/02 15:14:07.436471 [INFO] (runner) rendered "/opt/consul-template/templates/proxysql.ctmpl" =>
"/opt/consul-template/templates/proxysql.sql"
2022/04/02 15:14:07.436556 [INFO] (runner) executing command "/bin/bash -c 'mysql -P 6032 -h 127.0.0.1 -u admin -padmin <
/opt/consul-template/templates/proxysql.sql'" from "/opt/consul-template/templates/proxysql.ctmpl" =>
"/opt/consul-template/templates/proxysql.sql"
2022/04/02 15:14:07.436668 [INFO] (child) spawning: /bin/bash -c mysql -P 6032 -h 127.0.0.1 -u admin -padmin <
/opt/consul-template/templates/proxysql.sql

Proxysql before Failover
Proxysql after Failover
MySQL [(none)]> select hostgroup_id,hostname,port,status from runtime_mysql_servers;
+--------------+-----------------+------+--------+
| hostgroup_id | hostname | port | status |
+--------------+-----------------+------+--------+
| 1 | 192.168.103.20 | 3306 | ONLINE |
| 2 | 192.168.103.20 | 3306 | ONLINE |
| 2 | 192.168.103.10 | 3306 | ONLINE |
| 2 | 192.168.103.100 | 3307 | ONLINE |
+--------------+-----------------+------+--------+
MySQL [(none)]> select hostgroup_id,hostname,port,status from runtime_mysql_servers;
+--------------+-----------------+------+--------+
| hostgroup_id | hostname | port | status |
+--------------+-----------------+------+--------+
| 1 | 192.168.103.10 | 3306 | ONLINE |
| 2 | 192.168.103.20 | 3306 | ONLINE |
| 2 | 192.168.103.10 | 3306 | ONLINE |
| 2 | 192.168.103.100 | 3307 | ONLINE |
+--------------+-----------------+------+--------+

Conclusions
● Consul Orchestrator ProxySQL saves the day.
● Every component is highly available.
● All components are Open Source and Free.
● Well supported by Percona and community.
● Works with Percona Xtradb Cluster (Galera),
MariaDB Galera Cluster and other MySQL
setups.

MySQL HA Orchestrator Proxysql Consul.pdf

More Related Content

Similar to MySQL HA Orchestrator Proxysql Consul.pdf (20)

Recently uploaded (20)

MySQL HA Orchestrator Proxysql Consul.pdf