ProxySQL in the Cloud

ProxySQL in the cloud
How to conﬁgure and leverage ProxySQL with
AWS Aurora,
Azure Database for MySQL
and CloudSQL for MySQL
May 26th, 2022

Who am I?
René Cannaò
● Founder of ProxySQL LLC
● Author of ProxySQL
● MySQL DBA

ProxySQL LLC
We provide services to help build, support as well as improve the
performance & reliability of your Cloud-Based or On-Premise
MySQL infrastructure:
● ProxySQL Development
● Remote DBRE Consulting
● ProxySQL Support Services

Agenda
● What is ProxySQL
● Overview of its features
● Monitoring
● Failover detection
● Integration with Azure
● Integration with CloudSQL
● Integration with Aurora
We will not cover Kubernetes

What is proxySQL?
MySQL protocol aware data gateway
○ Clients connect to ProxySQL
○ Requests are evaluated
○ Various actions are performed
Visit https://proxysql.com for more information

ProxySQL features
● High Availability and Scalability
● seamless failover
● firewall
● query throttling
● query timeout
● query mirroring
● runtime reconfiguration
● Scheduler
● Support for Async Replication, Galera/PXC, Group Replication, Aurora

ProxySQL features
● on-the-fly rewrite of queries
● caching reads outside the database
● connection pooling and multiplexing
● complex query routing and r/w split
● load balancing
● real time statistics
● monitoring
● Data masking
● Management of hundreds of backend servers
● Native Clustering

Monitoring Availability of backends,
and failover detection and handling

mysql_servers table
CREATE TABLE mysql_servers (
hostgroup_id INT CHECK (hostgroup_id>=0) NOT NULL DEFAULT 0,
hostname VARCHAR NOT NULL,
port INT CHECK (port >= 0 AND port <= 65535) NOT NULL DEFAULT 3306,
…
status VARCHAR CHECK (UPPER(status) IN ('ONLINE','SHUNNED','OFFLINE_SOFT',
'OFFLINE_HARD')) NOT NULL DEFAULT 'ONLINE',
weight INT CHECK (weight >= 0 AND weight <=10000000) NOT NULL DEFAULT 1,
…
max_connections INT CHECK (max_connections >=0) NOT NULL DEFAULT 1000,
max_replication_lag INT CHECK (max_replication_lag >= 0 AND
max_replication_lag <= 126144000) NOT NULL DEFAULT 0,
use_ssl INT CHECK (use_ssl IN(0,1)) NOT NULL DEFAULT 0,
max_latency_ms INT UNSIGNED CHECK (max_latency_ms>=0) NOT NULL DEFAULT 0,
…
PRIMARY KEY (hostgroup_id, hostname, port) )

Backend server states
The monitor module will determine the status of a backend server in the “runtime_mysql_servers” table (these states can
also be set manually in “mysql_servers”):
● ONLINE - backend server is fully operational (manually set / automatically set by monitor)
● SHUNNED - backend server is temporarily taken out of use because of either too many connection errors in a time
that was too short, or replication lag exceeded the allowed threshold (automatically set by monitor)
● OFFLINE_SOFT - when a server is put into OFFLINE_SOFT mode, new incoming connections aren't accepted
anymore, while the existing connections are kept until they became inactive. In other words, connections are kept in
use until the current transaction is completed. This allows to gracefully detach a backend (manually set).
● OFFLINE_HARD - when a server is put into OFFLINE_HARD mode, the existing connections are dropped and no
new incoming connections are accepted. This is equivalent to deleting the server from a hostgroup (indeﬁnitely or
for maintenance) and is done by ProxySQL on failover (manually / automatically set by monitor)

Monitoring Backend Availability and status
It monitors availability of MySQL backends.
It monitors their status:
● can perform various checks against backend servers
● checks are built with dependencies:
○ if a lower level check fails there is no need for a higher level check

The Monitor Module
The Monitor Module is responsible for checking backends and ensuring that trafﬁc is only sent to
operational servers. Monitoring is performed at the following levels:
● Connect – tests new connections to backend servers (MySQL connect)
● Ping – pings backend servers (MySQL ping)
● Replication Lag – checks Seconds_Behind_Master status
● Read only – checks read_only status

Monitor Module Conﬁguration
The following global_variables are used to control the monitor module’s status and credentials:
● mysql-monitor_enabled – { true | false }
● mysql-monitor_username – a MySQL user present in the backend servers with USAGE and
REPLICATION CLIENT privileges
● mysql-monitor_password – the password for the monitoring account

Ping Conﬁguration
The following global_variables are used to control the monitor module’s ping check:
● mysql-monitor_ping_interval – How frequently a ping check is performed, in milliseconds.
● mysql-monitor_ping_timeout – Ping timeout in milliseconds.
● mysql-monitor_ping_max_failures – Maximum failures count before all connections to the backend are killed.
MySQL_Monitor will establish a new connection to perform the ping if no backend connections are available.
The time to mark a node down is:
● mysql-monitor_ping_max_failures * mysql-monitor_ping_timeout
Ping allows to detect network failures.

Connect Conﬁguration
The following global_variables are used to control the monitor module’s connect check:
● mysql-monitor_connect_interval – How frequently a connect check is performed, in milliseconds.
● mysql-monitor_connect_timeout – Connect timeout in milliseconds (value is rounded to the
second, always less than the interval).
Connect check allows to detect network failures or ﬁrewall.

Replication Lag Conﬁg
Automatically enabled when mysql_servers.max_replication_lag is not zero.
Variables used to control the monitor module’s replication lag check:
● mysql-monitor_replication_lag_interval - how frequently the check is performed, in milliseconds
● mysql-monitor_replication_lag_timeout - check timeout in milliseconds
● mysql-monitor_replication_lag_count - number of checks before shunning a node
● mysql-monitor_slave_lag_when_null - value to use if Seconds_behind_master is NULL
And few more.

Read Only Conﬁguration
Automatically enabled for hostgroups enabled in mysql_replication_hostgroups:
CREATE TABLE mysql_replication_hostgroups (
writer_hostgroup INT CHECK (writer_hostgroup>=0) NOT NULL PRIMARY KEY,
reader_hostgroup INT NOT NULL CHECK (reader_hostgroup<>writer_hostgroup
AND reader_hostgroup>=0),
check_type VARCHAR CHECK (LOWER(check_type) IN
('read_only','innodb_read_only','super_read_only','read_only|innodb_read_only
','read_only&innodb_read_only')) NOT NULL DEFAULT 'read_only',
comment VARCHAR NOT NULL DEFAULT '', UNIQUE (reader_hostgroup))

Read Only Conﬁguration
Variables used to control the monitor module’s read only status:
● mysql-monitor_read_only_interval – how frequently the check is performed, in milliseconds
● mysql-monitor_read_only_timeout – read only check timeout in milliseconds
● mysql-monitor_read_only_max_timeout_count - max number of timeouts

Monitor Logging
Each check type is logged to a different table in the monitor database:
connect mysql_server_connect_log
ping mysql_server_ping_log
replication lag mysql_server_replication_lag_log
read only mysql_server_read_only_log

mysql_server_replication_lag_log
CREATE TABLE mysql_server_replication_lag_log (
port INT NOT NULL DEFAULT 3306,
time_start_us INT NOT NULL DEFAULT 0,
success_time_us INT DEFAULT 0,
repl_lag INT DEFAULT 0,
error VARCHAR,
PRIMARY KEY (hostname, port, time_start_us))

mysql_server_read_only_log
CREATE TABLE mysql_server_read_only_log (
read_only INT DEFAULT 1,
error VARCHAR,
PRIMARY KEY (hostname, port, time_start_us))

ProxySQL Failover Detection
ProxySQL transparently detects and adapts to these topology changes via the monitor modules
read-only check
ProxySQL RO Checks:
- select @@read_only

When the status of a Primary server changes to read-only ProxySQL will remove the server from the
writer hostgroup (OFFLINE_HARD)
ProxySQL RO Checks:

Momentarily there are no RW nodes available, ProxySQL holds on to the connections for
mysql-connect_timeout_server_max milliseconds
ProxySQL RO Checks:

Once a RW is detected it is added to the writer_hostgroup (ONLINE) and connections are routed to the
new Primary server
ProxySQL RO Checks:

Important monitor variable that affects failover: mysql-monitor_writer_is_also_reader:
When a node change its read_only value from 1 to 0, this variable determines if the node should be added
to both reader and writer hostgroups
● false : node will be moved in writer_hostgroup and removed from reader_hostgroup
● true : node will be copied in writer_hostgroup and stay also in reader_hostgroup

Horizontal Scaling Azure Database for MySQL

ProxySQL for Azure Database for MySQL
ProxySQL helps to achieve:
● Connection pooling and multiplexing
● Read-Write split to scale reads across replicas
● Intelligent load balancing

Scaling and R/W split across replicas
Prerequisites:
a) A Source/replica replication
b) ProxySQL installed on a VM.
Step 1. Create proxysql’s monitor user in Azure database for MySQL
create user if not exists monitor identified with mysql_native_password by 'xxx'; grant replication client on *.* to monitor;
Step 2. Setup proxysql monitor user
set mysql-monitor_username=’monitor’;
set mysql-monitor_password=’xxx’;
Step 3. Conﬁgure MySQL instances on ProxySQL
INSERT INTO mysql_servers (hostgroup_id,hostname) VALUES (1,'proxysql-test-db.mysql.database.azure.com’);
INSERT INTO mysql_servers (hostgroup_id,hostname) VALUES (2,'proxysql-test-db0.mysql.database.azure.com');
The server in the hostgroup 1 is the writer and the servers in the hostgroup 2 is the reader.
Note if the require_secure_transport =ON (By default is on Azure) then you need to also to have the columns use_ssl=1.
UPDATE mysql_servers SET use_ssl=1;

Step 4. Create query rules to split you read and write trafﬁc: (example not suitable for production)
insert into mysql_query_rules(rule_id,active,match_digest,destination_hostgroup,apply)values(1,1,'^SELECT.*FOR UPDATE$',1,1);
insert into mysql_query_rules(rule_id,active,match_digest,destination_hostgroup,apply)values(2,1,'^SELECT',2,1);
Step 5. Conﬁgure application user in proxySQL. (Assuming that app user already exists in the DB)
INSERT INTO mysql_users (username,password,active,default_hostgroup) values ('sbtest','sbtestsbtest',1,1);
Step 6. Apply the changes to runtime and save them to disk.
LOAD MYSQL SERVERS TO RUNTIME;
LOAD MYSQL QUERY RULES TO RUNTIME;
LOAD MYSQL VARIABLES TO RUNTIME;
LOAD MYSQL USERS TO RUNTIME;
—---------------------------------
SAVE MYSQL SERVERS TO DISK;
SAVE MYSQL QUERY RULES TO DISK;
SAVE MYSQL VARIABLES TO DISK;
SAVE MYSQL USERS TO DISK;

Azure Database for MySQL and
mysql_replication_hostgroups
Using the mysql_replication_hostgroups to conﬁgure the replication setup, ProxySQL will monitor the
read_only value of each server to decide which is the replica and which is the source.
insert into mysql_replication_hostgroups
(writer_hostgroup,reader_hostgroup) values (1,2);

Some note on the example on the setup above:
Note that if the hostgroup 2, which handles the read trafﬁc for some reason has no available DB servers,
your reads will fail. Thus you should consider either:
● Add the writer also to Hostgroup 2 but with weight column being very low as below:
a. INSERT INTO mysql_servers (hostgroup_id,hostname,weight) VALUES (1,'proxysql-test-db.mysql.database.azure.com',1);
b. INSERT INTO mysql_servers (hostgroup_id,hostname,weight) VALUES (2,'proxysql-test-db.mysql.database.azure.com',1);
c. INSERT INTO mysql_servers (hostgroup_id,hostname,weight) VALUES (2,'proxysql-test-db0.mysql.database.azure.com',10000);
● Enable variable mysql-monitor_writer_is_also_reader

Gotcha: Azure Database for MySQL using
mysql_replication_hostgroups
ProxySQL automatically decides which is the replica evaluating the read_only value.
Stopping replication in Azure Database for MySQL the replica server becomes a standalone server and can get writes (read_only=0).
Azure also throws a warning before stopping replication.
https://docs.microsoft.com/en-us/azure/mysql/single-server/how-to-read-replicas-portal#stop-replication-to-a-replica-server

Horizontal scaling CloudSQL MySQL

ProxySQL + CloudSQL for MySQL
ProxySQL helps to achieve:
● Connection pooling and multiplexing
● Read-Write split to scale reads across replicas
● Intelligent load balancing
● Failover detection, when a high availability writer server is used

in CloudSQL MySQL using ProxySQL.
Prerequisites:
a) A Source/replica in CloudSQL for MySQL
b) proxySQL running on a VM.
Step 1. Create proxysql’s monitor user in CloudSQL for MySQL
Step 3. Conﬁgure MySQL instances on proxySQL
INSERT INTO mysql_servers (hostgroup_id,hostname) VALUES (0,'<IP ADDRESS-SOURCE>’);
INSERT INTO mysql_servers (hostgroup_id,hostname,max_replication_lag) VALUES (1,'<IP ADDRESS-REPLICA>’,5);

in CloudSQL MySQL using ProxySQL.
Step 4 conﬁgure mysql_replication_hostgroups:
The server in the hostgroup 1 is the writer and the servers in the hostgroup 2 is the reader.
insert into mysql_replication_hostgroups (writer_hostgroup, reader_hostgroup) values (0, 1);
Step 5. Create query rules to split you read and write trafﬁc. (example not suitable for production)
insert into mysql_query_rules(rule_id,active,match_digest,destination_hostgroup,apply)values(1,1,'^SELECT.*FOR
UPDATE$',0,1);
insert into mysql_query_rules(rule_id,active,match_digest,destination_hostgroup,apply)values(2,1,'^SELECT',1,1);

in CloudSQL MySQL using proxySQL.
Step 5. Conﬁgure application user in proxySQL or you can use a different use for read trafﬁc.
INSERT INTO mysql_users (username,password,active,default_hostgroup) values ('sbtest','sbtestsbtest',1,0);
INSERT INTO mysql_users (username,password,active,default_hostgroup) values (‘read_only_user’,'read_only_pass',1,1);
Step 6. Apply the changes to runtime and save them to disk.
LOAD MYSQL QUERY RULES TO RUNTIME;
LOAD MYSQL VARIABLES TO RUNTIME;
—---------------------------------
SAVE MYSQL QUERY RULES TO DISK;
SAVE MYSQL VARIABLES TO DISK;

ProxySQL and failover in CloudSQL
When the writer is conﬁgured to be highly available (Multiple zones availability) a failover can happen.
ProxySQL will:
● detects failover
● Deletes the writer from the writers’ hostgroup
● Brings it back whenever it is available.
Tests show that the whole failover took about ~15sec of write unavailability through ProxySQL:
2022-05-22 01:37:54 MySQL_Monitor.cpp:1426:monitor_read_only_thread(): [ERROR] Got error: mmsd 0x7fd1c5a23c00 , MYSQL
0x7fd1c4e02800 , FD 41 : Lost connection to MySQL server during query
2022-05-22 01:38:00 MySQL_Monitor.cpp:1411:monitor_read_only_thread(): [ERROR] Server 35.192.13.44:3306 missed 3
read_only checks. Assuming read_only=1
2022-05-22 01:38:00 [INFO] read_only_action RO=1 phase 1 : Dumping mysql_servers for 35.192.13.44:3306
…
2022-05-22 01:38:09 [INFO] Changing status for server 1: 35.192.13.44:3306 (35.192.13.44:3306) from 1 (1) to 0
2022-05-22 01:38:09 [INFO] Creating new server in HG 0 : 35.192.13.44:3306 , gtid_port=0, weight=1, status=0

ProxySQL and failover in CloudSQL
What about reads?
Reads can happening during the failover.
This behavior can be controlled by the variable mysql-monitor_slave_lag_when_null.
During the failover the the slave is reproting Seconds_behind_master: Null.
1. IF Max_replication_lag in mysql_servers < mysql-monitor_slave_lag_when_null
Replica server for read will not be available for reads.
2. IF Max_replication_lag in mysql_servers > mysql-monitor_slave_lag_when_null
Replica server will be available for reads.

Disaster recovery in CloudSQL + ProxySQL
In case of total unavailability of the writer server (no HA) :
1. In ProxySQL : delete the former writer from mysql_servers (for safety in case it comes back)
2. From Google CloudSQL console/cli:
a. Disable replication
b. Promote the replica server
The replica is promoted as a standalone server and it is read_only=0
ProxySQL will automatically put it on the writer’ hostgroup.
No changes are required in the connection strings on application side about the new writer.

Amazon Aurora Features
Plug and play support for Amazon Aurora cluster

Aurora AWS Integration
● ProxySQL tracks metrics and status variables from REPLICA_HOST_STATUS
● Automatically detects writer/reader roles
● Automatically detects failovers
● Auto-discovery of new replicas
● AZ awareness
● Replication lag monitoring with millisecond granularity

Aurora AWS Integration
● New conﬁguration table
CREATE TABLE mysql_aws_aurora_hostgroups (
writer_hostgroup INT CHECK (writer_hostgroup>=0) NOT NULL PRIMARY KEY,
reader_hostgroup INT NOT NULL CHECK (reader_hostgroup<>writer_hostgroup AND reader_hostgroup>0),
active INT CHECK (active IN (0,1)) NOT NULL DEFAULT 1,
aurora_port INT NOT NUlL DEFAULT 3306,
domain_name VARCHAR NOT NULL CHECK (SUBSTR(domain_name,1,1) = '.'),
max_lag_ms INT NOT NULL CHECK (max_lag_ms>= 10 AND max_lag_ms <= 600000) DEFAULT 600000,
check_interval_ms INT NOT NULL CHECK (check_interval_ms >= 100 AND check_interval_ms <= 600000) DEFAULT 1000,
check_timeout_ms INT NOT NULL CHECK (check_timeout_ms >= 80 AND check_timeout_ms <= 3000) DEFAULT 800,
writer_is_also_reader INT CHECK (writer_is_also_reader IN (0,1)) NOT NULL DEFAULT 0,
new_reader_weight INT CHECK (new_reader_weight >= 0 AND new_reader_weight <=10000000) NOT NULL DEFAULT 1,
add_lag_ms INT NOT NULL CHECK (add_lag_ms >= 0 AND add_lag_ms <= 600000) DEFAULT 30,
min_lag_ms INT NOT NULL CHECK (min_lag_ms >= 0 AND min_lag_ms <= 600000) DEFAULT 30,
lag_num_checks INT NOT NULL CHECK (lag_num_checks >= 1 AND lag_num_checks <= 16) DEFAULT 1,
comment VARCHAR,
UNIQUE (reader_hostgroup))

mysql_aws_aurora_hostgroups
Defines replication hostgroups for use with Amazon AWS Aurora.
Similar to mysql_replication_hostgroups READER & WRITER hostgroups.
Key differences:
● ProxySQL will will retrieve cluster information from Aurora table
information_schema.replica_host_status , allowing auto-discovery
● ProxySQL determines which one is the writer/master based on SESSION_ID : if the value is
MASTER_SESSION_ID , the server specific in SERVER_ID is the writer
● A lot of tuning specific to the cluster

mysql_aws_aurora_hostgroups
Replication lag metrics are pulled at regular intervals.
Three variables perform some tweaks on the last value of replication lag to attenuate noise:
● if replication lag is less than min_lag_ms, use this value
● add add_lag_ms to all checks performed
● if lag_num_checks is greater than 1, the current replication lag is computed as the highest of the
last lag_num_checks reads

Aurora AWS Monitor
Monitor module has 3 tables with regards to AWS Aurora:
Admin> SHOW TABLES FROM monitor;
+--------------------------------------+
| tables |
+--------------------------------------+
| mysql_server_aws_aurora_check_status |
| mysql_server_aws_aurora_failovers |
| mysql_server_aws_aurora_log |

Table mysql_server_aws_aurora_log
Collects metrics from information_schema.replica_host_status :
CREATE TABLE mysql_server_aws_aurora_log (
error VARCHAR,
SERVER_ID VARCHAR NOT NULL DEFAULT '',
SESSION_ID VARCHAR,
LAST_UPDATE_TIMESTAMP VARCHAR,
replica_lag_in_milliseconds INT NOT NULL DEFAULT 0,
estimated_lag_ms INT NOT NULL DEFAULT 0,
CPU INT NOT NULL DEFAULT 0,
PRIMARY KEY (hostname, port, time_start_us, SERVER_ID))

Table mysql_server_aws_aurora_check_status
Provides some aggregate information :
CREATE TABLE mysql_server_aws_aurora_check_status (
writer_hostgroup INT NOT NULL,
last_checked_at VARCHAR,
checks_tot INT NOT NULL DEFAULT 0,
checks_ok INT NOT NULL DEFAULT 0,
last_error VARCHAR,
PRIMARY KEY (writer_hostgroup, hostname, port))

Table mysql_server_aws_aurora_failovers
Records all failovers :
CREATE TABLE mysql_server_aws_aurora_failovers (
writer_hostgroup INT NOT NULL,
inserted_at VARCHAR NOT NULL)

Routing and replication lag
ProxySQL is able
● to monitor the replication lag of replicas in milliseconds,
● to determine which replicas reads should be sent to, that is the replicas that are up to date within the configured thresholds.
Most of the configuration is performed in table mysql_aws_aurora_hostgroups, specifically on columns max_lag_ms,
add_lag_ms, min_lag_ms, and lag_num_checks .
The client is able to specify a more (or less) restrictive threshold for max_lag_ms sending a comment with setting max_lag_ms .
For example, the Aurora cluster could be configured with mysql_aws_aurora_hostgroups.max_lag_ms=1000 , but a client could send a
query like: SELECT /* max_lag_ms=20 */ …
In this case only replicas with replication lag less or equal than 20ms will be considered to process the query.

Writer in reader hostgroup
● If mysql_aws_aurora_hostgroups.writer_is_also_reader is enabled, then the writer is also
conﬁgured in the reader hostgroup.
● In order to avoid reads from writer when clients specify a very small value for max_lag_ms, variable
mysql-aurora_max_lag_ms_only_read_from_replicas should be tuned:
○ The variable deﬁnes the minimum number of replicas that need to be present in order to ignore the writer in
the reader hostgroup

Conﬁguring ProxySQL for Amazon Aurora for MySQL
Step 1. Create proxysql’s monitor user in Aurora cluster
Step 3. Conﬁgure aurora mysql instances on ProxySQL
INSERT INTO mysql_servers (hostgroup_id,hostname) VALUES (1,'test-proxysql-0.asdfasf.eu-west-1.rds.amazonaws.com');

Configuring ProxySQL for Amazon Aurora for MySQL
Step 4. Configure writer and reader hostgroups.
INSERT INTO mysql_aws_aurora_hostgroups(writer_hostgroup, reader_hostgroup, domain_name
writer_is_also_reader) VALUES (0,1, '.asdfasf.eu-west-1.rds.amazonaws.com',1);
Step 5. Configure user
INSERT INTO mysql_users (username,password,active,default_hostgroup, use_ssl) values ('sbtest','sbtestsbtest',1,0, 0);
Note that on step 3 we choose the option the writer to be also the reader, and also the domain name
includes the “dot” at the start.

ProxySQL + AWS Aurora
Use cases of ProxySQL + AWS Aurora for MySQL showed that ProxySQL
● protected databases from high trafﬁc spikes prevents databases from having high number of
connections due to the multiplexing feature
○ Very relevant because of hardcore limit in Aurora
● minimized the impact during planned/unexpected failovers or crashes of DBs.
(https://proxysql.com/blog/failover-comparison-in-aurora-mysql-2-10-0-using-proxysql-vs-auror
as-cluster-endpoint/)

ProxySQL + AWS Aurora : unplanned failover (1)

Thank you!
Visit: https://proxysql.com
Github: https://github.com/sysown/proxysql/
https://github.com/proxysql/
Mailing list: https://groups.google.com/g/proxysql
Twitter: @proxysql
Drop us an email: info@proxysql.com , rene.c@proxysql.com

ProxySQL in the Cloud

More Related Content

What's hot (20)

Similar to ProxySQL in the Cloud (20)

Recently uploaded (20)

ProxySQL in the Cloud