MySQL InnoDB Cluster and Group Replication in a Nutshell

MySQL InnoDB Cluster & Group Replication in a
Nutshell: Hands-On Tutorial

Percona Live Europe 2017 - Dublin
Frédéric Descamps - MySQL Community Manager - Oracle
Kenny Gryp - MySQL Practice Manager - Percona
3 / 152

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purpose only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied up in
making purchasing decisions. The development, release and timing of any features or
functionality described for Oracle´s product remains at the sole discretion of Oracle.
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
4 / 152

Who are we ?
5 / 152

6 / 152

Frédéric Descamps
@lefred
MySQL Evangelist
Managing MySQL since 3.23
devops believer
http://about.me/lefred

7 / 152

Kenny Gryp
@gryp
MySQL Practice Manager

8 / 152

get more at the conference
MySQL Group Replication
9 / 152

10 / 152

Agenda
Prepare your workstation
11 / 152

Agenda
What are MySQL InnoDB Cluster &Group Replication ?
12 / 152

Agenda
Migration fromMaster-Slave to GR
13 / 152

Agenda
Howto monitor ?
14 / 152

Agenda
Howto monitor ?
Application interaction
15 / 152

VirtualBox
Setup your workstation
16 / 152

Install VirtualBox 5
On the USB key, copy PLeu17_GR.ova on your laptop and doubleclick on it
Ensure you have vboxnet2 network interface
(VirtualBox Preferences -> Network -> Host-Only Networks -> +)
Start all virtual machines (mysql1, mysql2, mysql3 &mysql4)
Install putty if you are using Windows
17 / 152

Install VirtualBox 5
On the USB key, copy PLeu17_GR.ova on your laptop and doubleclick on it
Ensure you have vboxnet2 network interface
(VirtualBox Preferences -> Network -> Host-Only Networks -> +)
Start all virtual machines (mysql1, mysql2, mysql3 &mysql4)
Install putty if you are using Windows
Try to connect to all VM´s fromyour terminal or putty (rootpasswordisX):
ssh -p 8821 root@127.0.0.1 to mysql1
18 / 152

LAB1: Current situation
19 / 152

launch
run_app.sh
on mysql1 into
a screen
session
verify that
mysql2 is a
running slave
LAB1: Current situation
20 / 152

Summary

+--------+----------+--------------+-----------------+
| | ROLE | SSH PORT | INTERNAL IP |
+--------+----------+--------------+-----------------+
| | | | |
| mysql1 | master | 8821 | 192.168.56.11 |
| | | | |
| mysql2 | slave | 8822 | 192.168.56.12 |
| | | | |
| mysql3 | n/a | 8823 | 192.168.56.13 |
| | | | |
| mysql4 | n/a | 8824 | 192.168.56.14 |
| | | | |
+--------+----------+--------------+-----------------+
21 / 152

Easy High Availability
MySQL InnoDB Cluster
22 / 152

InnoDB
cluster
Ease-of-Use
Extreme Scale-Out
Out-of-Box Solution
Built-in HA
High Performance
Everything Integrated
23 / 152

InnoDB Cluster's Architecture
Application
MySQL Connector
MySQL Router
MySQL Shell
InnoDB
cluster
Application
MySQL Connector
MySQL Router
Mp
M
M
24 / 152

Group Replication: heart of MySQL InnoDB
Cluster
25 / 152

Group Replication: heart of MySQL InnoDB
Cluster
26 / 152

but what is it ?!?
27 / 152

but what is it ?!?
GR is a plugin for MySQL, made by MySQL and packaged with MySQL
28 / 152

but what is it ?!?
GR is an implementation of Replicated Database State Machine theory
29 / 152

but what is it ?!?
Paxos based protocol
30 / 152

but what is it ?!?
GR allows to write on all Group Members (cluster nodes) simultaneously while
retaining consistency
31 / 152

but what is it ?!?
GR implements conflict detection and resolution
32 / 152

but what is it ?!?
GR allows automatic distributed recovery
33 / 152

but what is it ?!?
GR allows automatic distributed recovery
Supported on all MySQL platforms !!
Linux, Windows, Solaris, OSX, FreeBSD
34 / 152

And for users ?
35 / 152

And for users ?
not longer necessary to handle server fail-over manually or with a complicated script
36 / 152

And for users ?
GR provides fault tolerance
37 / 152

And for users ?
GR enables update-everywhere setups
38 / 152

And for users ?
GR handles crashes, failures, re-connects automatically
39 / 152

And for users ?
GR handles crashes, failures, re-connects automatically
Allows an easy setup of a highly available MySQL service!
40 / 152

ready ?
Migration from Master-Slave to GR
41 / 152

The plan
42 / 152

1) We install and
setup MySQL InnoDB
Cluster on one of the
newservers
The plan
43 / 152

2) We restore a
backup
3) setup
asynchronous
replication on the new
server.
The plan
44 / 152

4) We add a new
instance to our group
The plan
45 / 152

5) We point the
application to one of
our newnodes.
6) We wait and check
that asynchronous
replication is caught
up
7) we stop those
asynchronous slaves
The plan
46 / 152

8) We attach the
mysql2 slave to the
group
The plan
47 / 152

9) Use MySQL Router
for directing traffic
The plan
48 / 152

Latest MySQL 8.0.3-RC is already installed
on mysql3.
Let´s take a backup on mysql1:
[mysql1 ~]# xtrabackup --backup
--target-dir=/tmp/backup
--user=root
--password=X --host=127.0.0.1
[mysql1 ~]# xtrabackup --prepare
--target-dir=/tmp/backup
LAB2: Prepare mysql3
Asynchronousslave
49 / 152

LAB2: Prepare mysql3 (2)
Asynchronousslave
Copy the backup frommysql1 to mysql3:
[mysql1 ~]# scp -r /tmp/backup mysql3:/tmp
And restore it:
[mysql3 ~]# systemctl stop mysqld
[mysql3 ~]# rm -rf /var/lib/mysql/*
[mysql3 ~]# xtrabackup --copy-back --target-dir=/tmp/backup
[mysql3 ~]# chown -R mysql. /var/lib/mysql
50 / 152

LAB3: mysql3 as asynchronous slave (2)
Asynchronousslave
Configure /etc/my.cnf with the minimal requirements:
[mysqld]
...
server_id=3
enforce_gtid_consistency = on
gtid_mode = on
#log_bin # new default
#log_slave_updates # new default
51 / 152

Asynchronousslave
Let´s start MySQL on mysql3:
[mysql3 ~]# systemctl start mysqld
52 / 152

Asynchronousslave
Let´s start MySQL on mysql3:
[mysql3 ~]# mysql_upgrade
53 / 152

find the GTIDs purged
change MASTER
set the purged GTIDs
start replication

54 / 152

Find the latest purged GTIDs:
[mysql3 ~]# cat /tmp/backup/xtrabackup_binlog_info
mysql-bin.000002 167646328 b346474c-8601-11e6-9b39-08002718d305:1-771
Connect to mysql3 and setup replication:
mysql> CHANGE MASTER TO MASTER_HOST="mysql1",
MASTER_USER="repl_async", MASTER_PASSWORD='Xslave',
MASTER_AUTO_POSITION=1;
mysql> RESET MASTER;
mysql> SET global gtid_purged="VALUE FOUND PREVIOUSLY";
mysql> START SLAVE;
Check that you receive the application´s traffic
55 / 152

Administration made easy and more...
MySQL-Shell
56 / 152

MySQL Shell
The MySQL Shell is an interactive Javascript, Python, or SQL interface supporting
development and administration for MySQL. MySQL Shell includes the AdminAPI--available
in JavaScript and Python--which enables you to set up and manage InnoDB clusters. It
provides a modern and fluent API which hides the complexity associated with configuring,
provisioning, and managing an InnoDB cluster, without sacrificing power, flexibility, or
security.
57 / 152

MySQL Shell (2)
As example. the same operations as before but using the Shell:
58 / 152

MySQL Shell (3)
59 / 152

LAB4: MySQL InnoDB Cluster
Createasingleinstancecluster
Time to use the newMySQL Shell !
[mysql3 ~]# mysqlsh
Let´s verify if our server is ready to become a member of a newcluster:
mysql-js> dba.checkInstanceCon guration('root@mysql3:3306')
60 / 152

LAB4: MySQL InnoDB Cluster
Time to use the newMySQL Shell !
[mysql3 ~]# mysqlsh
Let´s verify if our server is ready to become a member of a newcluster:
Change the configuration !
mysql-js> dba.con gureLocalInstance()
61 / 152

LAB4: MySQL InnoDB Cluster (2)
Restartmysqldtousethenewconfiguration:
[mysql3 ~]# systemctl restart mysqld
62 / 152

[mysql3 ~]# mysqlsh
63 / 152

[mysql3 ~]# mysqlsh
mysql-js> c root@mysql3:3306
mysql-js> cluster = dba.createCluster('perconalive')
64 / 152

LAB4: Cluster Status
mysql-js> cluster.status()
{
"clusterName": "perconalive",
"defaultReplicaSet": {
"name": "default",
"primary": "mysql3:3306",
"ssl": "DISABLED",
"status": "OK_NO_TOLERANCE",
"statusText": "Cluster is NOT tolerant to any failures.",
"topology": {
"mysql3:3306": {
"address": "mysql3:3306",
"mode": "R/W",
"readReplicas": {},
"role": "HA",
"status": "ONLINE"
}
}
}
}
65 / 152

Add mysql4 to the Group:
restore the backup
set the purged GTIDs
use MySQL Shell
LAB5: add mysql4 to the cluster (1)

66 / 152

LAB5: add mysql4 to the cluster (2)
Copy the backup frommysql1 to mysql4:
[mysql1 ~]# scp -r /tmp/backup mysql4:/tmp
And restore it:
[mysql4 ~] systemctl stop mysqld
[mysql4 ~] rm -rf /var/lib/mysql/*
[mysql4 ~]# xtrabackup --copy-back --target-dir=/tmp/backup
[mysql4 ~]# chown -R mysql. /var/lib/mysql
Start MySQL on mysql4:
67 / 152

LAB5: MySQL Shell to add an instance (3)
[mysql4 ~]# mysqlsh
Let´s verify the config:
68 / 152

[mysql4 ~]# mysqlsh
And change the configuration:
69 / 152

[mysql4 ~]# mysqlsh
And change the configuration:
Restart the service to enable the changes:
70 / 152

Groupof2instances
Find the latest purged GTIDs:
[mysql4 ~]# cat /tmp/backup/xtrabackup_binlog_info
mysql-bin.000002 167646328 b346474c-8601-11e6-9b39-08002718d305:1-77177
Connect to mysql4 and set GTID_PURGED
[mysql4 ~]# mysqlsh
mysql-js> sql
mysql-sql> RESET MASTER;
mysql-sql> SET global gtid_purged="VALUE FOUND PREVIOUSLY";
71 / 152

mysql-sql> js
mysql-js> cluster = dba.getCluster()
mysql-js> cluster.checkInstanceState('root@mysql4:3306')
mysql-js> cluster.addInstance("root@mysql4:3306")
72 / 152

Cluster Status
{
"name": "default",
"ssl": "DISABLED",
"status": "OK_NO_TOLERANCE",
"statusText": "Cluster is NOT tolerant to any failures. 1 member is not active"
"topology": {
"mysql3:3306": {
"mode": "R/W",
"readReplicas": {},
"role": "HA",
"status": "ONLINE"
},
"mysql4:3306": {
"mode": "R/O",
"readReplicas": {},
"role": "HA",
"status": "RECOVERING"
}
}
} Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
73 / 152

Recovering progress
On standard MySQL, monitor the group_replication_recovery channel to see
the progress:
mysql4> show slave status for channel 'group_replication_recovery'G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: mysql3
Master_User: mysql_innodb_cluster_rpl_user
...
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
...
Retrieved_Gtid_Set: 6e7d7848-860f-11e6-92e4-08002718d305:1-6,
7c1f0c2d-860d-11e6-9df7-08002718d305:1-15,
b346474c-8601-11e6-9b39-08002718d305:1964-77177,
e8c524df-860d-11e6-9df7-08002718d305:1-2
Executed_Gtid_Set: 7c1f0c2d-860d-11e6-9df7-08002718d305:1-7,
b346474c-8601-11e6-9b39-08002718d305:1-45408,
e8c524df-860d-11e6-9df7-08002718d305:1-2
...
74 / 152

point the application
to the cluster
Migrate the application
75 / 152

LAB6: Migrate the application
Make sure gtid_executed range on mysql2 is lower or equal than on mysql3
mysql[2-3]> show global variables like 'gtid_executed'G
76 / 152

Make sure gtid_executed range on mysql2 is lower or equal than on mysql3
mysql[2-3]> show global variables like 'gtid_executed'G
When they are OK, stop asynchronous replication on mysql2 and mysql3:
mysql2> stop slave;
mysql3> stop slave;
77 / 152

Nowwe need to point the application to mysql3, this is the only downtime !
...
[ 21257s] threads: 4, tps: 12.00, reads: 167.94, writes: 47.98, response time: 18
^C
[mysql1 ~]# run_app.sh mysql3
78 / 152

Nowwe need to point the application to mysql3, this is the only downtime !
...
^C
[mysql1 ~]# run_app.sh mysql3
Nowthey can forget about mysql1:
mysql[2-3]> reset slave all;
79 / 152

previous slave
(mysql2) can now
be part of the cluster
Add a third instance
80 / 152

LAB7: Add mysql2 to the group
We first upgrade to MySQL 8.0.3 :
81 / 152

[mysql2 ~]# rpm -Uvh /root/rpms/mysql*rpm
82 / 152

and then we validate the instance using MySQL Shell and we configure it:
[mysql2 ~]# mysqlsh
83 / 152

[mysql2 ~]# mysqlsh
84 / 152

[mysql2 ~]# mysqlsh
85 / 152

LAB7: Add mysql2 to the group (2)
Back in MySQL Shell we add the newinstance:
[mysql2 ~]# mysqlsh
86 / 152

Back in MySQL Shell we add the newinstance:
[mysql2 ~]# mysqlsh
mysql-js> cluster.addInstance("root@mysql2:3306")
87 / 152

{
"name": "default",
"status": "OK",
"statusText": "Cluster is ONLINE and can tolerate up to ONE failure.",
"topology": {
"mysql2:3306": {
"mode": "R/O",
"readReplicas": {},
"role": "HA",
"status": "ONLINE"
},
"mysql3:3306": {
"mode": "R/W",
"readReplicas": {},
"role": "HA",
"status": "ONLINE"
},
"mysql4:3306": {
"mode": "R/O",
"readReplicas": {},Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
88 / 152

writing to a single server
Single Primary Mode
89 / 152

Default = Single Primary Mode
By default, MySQL InnoDB Cluster enables Single Primary Mode.
90 / 152

mysql> show global variables like 'group_replication_single_primary_mode';
+---------------------------------------+-------+
| Variable_name | Value |
+---------------------------------------+-------+
| group_replication_single_primary_mode | ON |
+---------------------------------------+-------+
91 / 152

mysql> show global variables like 'group_replication_single_primary_mode';
+---------------------------------------+-------+
+---------------------------------------+-------+
| group_replication_single_primary_mode | ON |
+---------------------------------------+-------+
In Single Primary Mode, a single member acts as the writable master (PRIMARY) and the
rest of the members act as hot-standbys (SECONDARY).
The group itself coordinates and configures itself automatically to determine which
member will act as the PRIMARY, through a leader election mechanism.
92 / 152

Who´s the Primary Master ? old fashion style
As the Primary Master is elected, all nodes part of the group knows which one was
elected. This value is exposed in status variables:
93 / 152

mysql> show status like 'group_replication_primary_member';
+----------------------------------+--------------------------------------+
+----------------------------------+--------------------------------------+
| group_replication_primary_member | 28a4e51f-860e-11e6-bdc4-08002718d305 |
+----------------------------------+--------------------------------------+
94 / 152

mysql> show status like 'group_replication_primary_member';
+----------------------------------+--------------------------------------+
+----------------------------------+--------------------------------------+
| group_replication_primary_member | 28a4e51f-860e-11e6-bdc4-08002718d305 |
+----------------------------------+--------------------------------------+
mysql> select member_host as "primary master"
from performance_schema.global_status
join performance_schema.replication_group_members
where variable_name = 'group_replication_primary_member'
and member_id=variable_value;
+---------------+
| primary master|
+---------------+
| mysql3 |
+---------------+
95 / 152

Who´s the Primary Master ? new fashion style
mysql> select member_host
from performance_schema.replication_group_members
where member_role='PRIMARY';
+-------------+
| member_host |
+-------------+
| mysql3 |
+-------------+
96 / 152

Create a Multi-Primary Cluster:
It´s also possible to create a Multi-Primary Cluster using the Shell:
97 / 152

mysql-js> cluster=dba.createCluster('perconalive',{multiMaster: true})
98 / 152

A new InnoDB cluster will be created on instance 'root@mysql3:3306'.
The MySQL InnoDB cluster is going to be setup in advanced Multi-Master Mode.
Before continuing you have to con rm that you understand the requirements and
limitations of Multi-Master Mode. Please read the manual before proceeding.
I have read the MySQL InnoDB cluster manual and I understand the requirements
and limitations of advanced Multi-Master Mode.
Con rm [y|N]:
99 / 152

Con rm [y|N]:
Or you can force it to avoid interaction (for automation) :
100 / 152

Con rm [y|N]:
Or you can force it to avoid interaction (for automation) :
> cluster=dba.createCluster('perconalive',{multiMaster: true, force: true})
101 / 152

get more info
Monitoring
102 / 152

Performance Schema
Group Replication uses Performance_Schema to expose status
mysql3> SELECT * FROM performance_schema.replication_group_membersG
*************************** 1. row ***************************
CHANNEL_NAME: group_replication_applier
MEMBER_ID: ade14d5c-9e1e-11e7-b034-08002718d305
MEMBER_HOST: mysql4
MEMBER_PORT: 3306
MEMBER_STATE: ONLINE
MEMBER_ROLE: SECONDARY
MEMBER_VERSION: 8.0.3
*************************** 2. row ***************************
MEMBER_ID: b9d01593-9dfb-11e7-8ca6-08002718d305
MEMBER_HOST: mysql3
MEMBER_PORT: 3306
MEMBER_STATE: ONLINE
MEMBER_ROLE: PRIMARY
MEMBER_VERSION: 8.0.3
103 / 152

mysql3> SELECT * FROM performance_schema.replication_connection_statusG
*************************** 1. row ***************************
GROUP_NAME: 8fc848d7-9e1c-11e7-9407...
SOURCE_UUID: 8fc848d7-9e1c-11e7-9407...
THREAD_ID: NULL
SERVICE_STATE: ON
COUNT_RECEIVED_HEARTBEATS: 0
LAST_HEARTBEAT_TIMESTAMP: 0000-00-00 00:00:00
RECEIVED_TRANSACTION_SET: 8fc848d7-9e1c-11e7-9407...
b9d01593-9dfb-11e7-8ca6-08002718d305:1-21,
da2f0910-8767-11e6-b82d-08002718d305:1-164741
LAST_ERROR_NUMBER: 0
LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
LAST_QUEUED_TRANSACTION: 8fc848d7-9e1c-11e7-9407...
LAST_QUEUED_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP: 0000-00-00 00:00:00
LAST_QUEUED_TRANSACTION_IMMEDIATE_COMMIT_TIMESTAMP: 0000-00-00 00:00:00
LAST_QUEUED_TRANSACTION_START_QUEUE_TIMESTAMP: 2017-09-20 16:22:36.486...
LAST_QUEUED_TRANSACTION_END_QUEUE_TIMESTAMP: 2017-09-20 16:22:36.486...
QUEUEING_TRANSACTION:
QUEUEING_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP: 0000-00-00 00:00:00
QUEUEING_TRANSACTION_IMMEDIATE_COMMIT_TIMESTAMP: 0000-00-00 00:00:00
QUEUEING_TRANSACTION_START_QUEUE_TIMESTAMP: 0000-00-00 00:00:00
104 / 152

Member State
These are the different possible state for a node member:
ONLINE
OFFLINE
RECOVERING
ERROR: when a node is leaving but the plugin was not instructed to stop
UNREACHABLE
105 / 152

Status information & metrics
Members
mysql> SELECT member_host, member_state, member_role
FROM performance_schema.replication_group_members;
106 / 152

Status information & metrics - connections
mysql> SELECT * FROM performance_schema.replication_connection_statusG
108 / 152

Status information & metrics - connections
mysql> SELECT * FROM performance_schema.replication_connection_statusG
*************************** 1. row ***************************
GROUP_NAME: 8fc848d7-9e1c-11e7-9407-...
SOURCE_UUID: 8fc848d7-9e1c-11e7-9407-...
THREAD_ID: NULL
SERVICE_STATE: ON
COUNT_RECEIVED_HEARTBEATS: 0
LAST_HEARTBEAT_TIMESTAMP: 0000-00-00 00:00:00
RECEIVED_TRANSACTION_SET: 8fc848d7-9e1c-11e7-9407-...
b9d01593-9dfb-11e7-8ca6-08002718d305:1-21,
da2f0910-8767-11e6-b82d-08002718d305:1-164741
LAST_ERROR_NUMBER: 0
LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
LAST_QUEUED_TRANSACTION: 8fc848d7-9e1c-11e7-9407-...
LAST_QUEUED_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP: 0000-00-00 00:00:00
LAST_QUEUED_TRANSACTION_IMMEDIATE_COMMIT_TIMESTAMP: 0000-00-00 00:00:00
LAST_QUEUED_TRANSACTION_START_QUEUE_TIMESTAMP: 2017-09-20 16:22:36.4864...
LAST_QUEUED_TRANSACTION_END_QUEUE_TIMESTAMP: 2017-09-20 16:22:36.4865...
QUEUEING_TRANSACTION:
QUEUEING_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP: 0000-00-00 00:00:00
QUEUEING_TRANSACTION_IMMEDIATE_COMMIT_TIMESTAMP: 0000-00-00 00:00:00
QUEUEING_TRANSACTION_START_QUEUE_TIMESTAMP: 0000-00-00 00:00:00
109 / 152

Previously there was only local node statistics, now they are exposed all
over the Group
mysql> select * from performance_schema.replication_group_member_statsG
110 / 152

Previously there was only local node statistics, now they are exposed all
over the Group
mysql> select * from performance_schema.replication_group_member_statsG
************************** 1. row ***************************
VIEW_ID: 15059231192196925:2
MEMBER_ID: ade14d5c-9e1e-11e7-b034-08002...
COUNT_TRANSACTIONS_IN_QUEUE: 0
COUNT_TRANSACTIONS_CHECKED: 27992
COUNT_CONFLICTS_DETECTED: 0
COUNT_TRANSACTIONS_ROWS_VALIDATING: 0
TRANSACTIONS_COMMITTED_ALL_MEMBERS: 8fc848d7-9e1c-11e7-9407-08002...
b9d01593-9dfb-11e7-8ca6-08002718d305:1-21,
da2f0910-8767-11e6-b82d-08002718d305:1-164741
LAST_CONFLICT_FREE_TRANSACTION: 8fc848d7-9e1c-11e7-9407-08002...
COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE: 0
COUNT_TRANSACTIONS_REMOTE_APPLIED: 27992
COUNT_TRANSACTIONS_LOCAL_PROPOSED: 0
COUNT_TRANSACTIONS_LOCAL_ROLLBACK: 0
111 / 152

************************** 2. row ***************************
VIEW_ID: 15059231192196925:2
MEMBER_ID: b9d01593-9dfb-11e7-8ca6-08002...
COUNT_TRANSACTIONS_IN_QUEUE: 0
COUNT_TRANSACTIONS_CHECKED: 28000
COUNT_CONFLICTS_DETECTED: 0
COUNT_TRANSACTIONS_ROWS_VALIDATING: 0
TRANSACTIONS_COMMITTED_ALL_MEMBERS: 8fc848d7-9e1c-11e7-9407-08002...
b9d01593-9dfb-11e7-8ca6-08002718d305:1-21,
da2f0910-8767-11e6-b82d-08002718d305:1-164741
LAST_CONFLICT_FREE_TRANSACTION: 8fc848d7-9e1c-11e7-9407-08002...
COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE: 0
COUNT_TRANSACTIONS_REMOTE_APPLIED: 1
COUNT_TRANSACTIONS_LOCAL_PROPOSED: 28000
COUNT_TRANSACTIONS_LOCAL_ROLLBACK: 0
112 / 152

Performance_Schema
You can find GR information in the following Performance_Schema tables:
replication_applier_con guration
replication_applier_status
replication_applier_status_by_worker
replication_connection_con guration
replication_connection_status
replication_group_member_stats
replication_group_members
113 / 152

Status during recovery
mysql> SHOW SLAVE STATUS FOR CHANNEL 'group_replication_recovery'G
114 / 152

Status during recovery
mysql> SHOW SLAVE STATUS FOR CHANNEL 'group_replication_recovery'G
*************************** 1. row ***************************
Slave_IO_State:
Master_Host: <NULL>
Master_User: gr_repl
Master_Port: 0
...
Relay_Log_File: mysql4-relay-bin-group_replication_recovery.000001
...
Slave_IO_Running: No
Slave_SQL_Running: No
...
Executed_Gtid_Set: 5de4400b-3dd7-11e6-8a71-08002774c31b:1-814089,
afb80f36-2bff-11e6-84e0-0800277dd3bf:1-5718
...
Channel_Name: group_replication_recovery
115 / 152

Sys Schema
The easiest way to detect if a node is a member of the primary component (when there
are partitioning of your nodes due to network issues for example) and therefore a valid
candidate for routing queries to it, is to use the sys table.
Additional information for sys can be found at https://goo.gl/XFp3bt
On the primary node:
[mysql3 ~]# mysql < /root/addition_to_sys_mysql8.sql
116 / 152

Sys Schema
Is this node part of PRIMARY Partition:
mysql3> SELECT sys.gr_member_in_primary_partition();
+------------------------------------+
| sys.gr_node_in_primary_partition() |
+------------------------------------+
| YES |
+------------------------------------+
117 / 152

Sys Schema
Is this node part of PRIMARY Partition:
mysql3> SELECT sys.gr_member_in_primary_partition();
+------------------------------------+
| sys.gr_node_in_primary_partition() |
+------------------------------------+
| YES |
+------------------------------------+
To use as healthcheck:
mysql3> SELECT * FROM sys.gr_member_routing_candidate_status;
+------------------+-----------+---------------------+----------------------+
| viable_candidate | read_only | transactions_behind | transactions_to_cert |
+------------------+-----------+---------------------+----------------------+
| YES | YES | 0 | 0 |
+------------------+-----------+---------------------+----------------------+
118 / 152

LAB8: Sys Schema - Health Check
On one of the non Primary nodes, run the following command:
mysql-sql> ush tables with read lock;
119 / 152

Nowyou can verify what the healthcheck exposes to you:
mysql-sql> SELECT * FROM sys.gr_member_routing_candidate_status;
+------------------+-----------+---------------------+----------------------+
+------------------+-----------+---------------------+----------------------+
| YES | YES | 950 | 0 |
+------------------+-----------+---------------------+----------------------+
120 / 152

Nowyou can verify what the healthcheck exposes to you:
mysql-sql> SELECT * FROM sys.gr_member_routing_candidate_status;
+------------------+-----------+---------------------+----------------------+
+------------------+-----------+---------------------+----------------------+
| YES | YES | 950 | 0 |
+------------------+-----------+---------------------+----------------------+
mysql-sql> UNLOCK TABLES;
121 / 152

application interaction
MySQL Router
122 / 152

MySQL Router
MySQL Router is lightweight middleware that provides transparent routing between your
application and backend MySQL Servers. It can be used for a wide variety of use cases,
such as providing high availability and scalability by effectively routing database traffic to
appropriate backend MySQL Servers.
123 / 152

MySQL Router
MySQL Router is lightweight middleware that provides transparent routing between your
application and backend MySQL Servers. It can be used for a wide variety of use cases,
such as providing high availability and scalability by effectively routing database traffic to
appropriate backend MySQL Servers.
MySQL Router doesn´t require any specific configuration. It configures itself automatically
(bootstrap) using MySQL InnoDB Cluster´s metadata.
124 / 152

LAB9: MySQL Router
We will nowuse mysqlrouter between our application and the cluster.
125 / 152

LAB9: MySQL Router (2)
Configure MySQL Router that will run on the app server (mysql1). We bootstrap it using
the Primary-Master:
[root@mysql1 ~]# mysqlrouter --bootstrap mysql3:3306 --user mysqlrouter
Please enter MySQL password for root:
WARNING: The MySQL server does not have SSL ...
Bootstrapping system MySQL Router instance...
MySQL Router has now been con gured for the InnoDB cluster 'perconalive'.
The following connection information can be used to connect to the cluster.
Classic MySQL protocol connections to cluster 'perconalive':
- Read/Write Connections: localhost:6446
- Read/Only Connections: localhost:6447
X protocol connections to cluster 'perconalive':
- Read/Write Connections: localhost:64460
- Read/Only Connections: localhost:64470
[root@mysql1 ~]# chown -R mysqlrouter. /var/lib/mysqlrouter
126 / 152

Nowlet´s modify the configuration file to listen on port 3306:
127 / 152

in /etc/mysqlrouter/mysqlrouter.conf:
[routing:perconalive_default_rw]
-bind_port=6446
+bind_port=3306
128 / 152

in /etc/mysqlrouter/mysqlrouter.conf:
[routing:perconalive_default_rw]
-bind_port=6446
+bind_port=3306
We can stop mysqld on mysql1 and start mysqlrouter into a screen session:
[mysql1 ~]# systemctl start mysqlrouter
129 / 152

Before killing a member we will change systemd´s default behavior that restarts
mysqld immediately:
130 / 152

Before killing a member we will change systemd´s default behavior that restarts
mysqld immediately:
in /usr/lib/systemd/system/mysqld.service add the following under
[Service]
RestartSec=30
[mysql3 ~]# systemctl daemon-reload
131 / 152

Nowwe can point the application to the router (back to mysql1):
[mysql1 ~]# run_app.sh
132 / 152

Check app and kill mysqld on mysql3 (the Primary Master R/Wnode) !
[mysql3 ~]# kill -9 $(pidof mysqld)
133 / 152

Check app and kill mysqld on mysql3 (the Primary Master R/Wnode) !
mysql2> select member_host
from performance_schema.replication_group_members
where member_role='PRIMARY';
+-------------+
| member_host |
+-------------+
| mysql4 |
+-------------+
134 / 152

ProxySQL / HA Proxy / F5 / ...
3rd party router/proxy
135 / 152

MySQL InnoDB Cluster can also work with third party router / proxy.
136 / 152

If you need some specific features that are not yet available in MySQL Router, like
transparent R/Wsplitting, then you can use your software of choice.
137 / 152

The important part of such implementation is to use a good health check to verify if the
MySQL server you plan to route the traffic is in a valid state.
138 / 152

MySQL Router implements that natively, and it´s very easy to deploy.
139 / 152

ProxySQL also has native support for Group
Replication which makes it maybe the best
choice for advanced users.
MySQL Router implements that natively, and it´s very easy to deploy.
140 / 152

operational tasks
Recovering Node
141 / 152

Recovering Nodes/Members
The old master (mysql3) got killed.
142 / 152

MySQL got restarted automatically by systemd
143 / 152

MySQL got restarted automatically by systemd
Let´s add mysql3 back to the cluster
144 / 152

LAB10: Recovering Nodes/Members
[mysql3 ~]# mysqlsh
mysql-js> c root@mysql4:3306 # The current master
mysql-js> cluster.rejoinInstance("root@mysql3:3306")
Rejoining the instance to the InnoDB cluster. Depending on the original
problem that made the instance unavailable, the rejoin operation might not be
successful and further manual steps will be needed to x the underlying
problem.
Please monitor the output of the rejoin operation and take necessary action if
the instance cannot rejoin.
Please provide the password for 'root@mysql3:3306':
Rejoining instance to the cluster ...
The instance 'root@mysql3:3306' was successfully rejoined on the cluster.
The instance 'mysql3:3306' was successfully added to the MySQL Cluster.
145 / 152

{ "clusterName": "perconalive",
"name": "default",
"status": "OK",
"statusText": "Cluster is ONLINE and can tolerate up to ONE failure.",
"topology": {
"mysql2:3306": {
"mode": "R/O",
"readReplicas": {},
"role": "HA",
"status": "ONLINE" },
"mysql3:3306": {
"mode": "R/O",
"readReplicas": {},
"role": "HA",
"status": "ONLINE" },
"mysql4:3306": {
"mode": "R/W",
"readReplicas": {},
"role": "HA",
"status": "ONLINE" }
}
}
}
146 / 152

Recovering Nodes/Members (automatically)
This time before killing a member of the group, we will persist the configuration on disk in
my.cnf.
147 / 152

Recovering Nodes/Members (automatically)
This time before killing a member of the group, we will persist the configuration on disk in
my.cnf.
We will use again the same MySQL command as previously
dba.con gureLocalInstance() but this time when all nodes are already part
of the Group.
148 / 152

LAB10: Recovering Nodes/Members (2)
Verify that all nodes are ONLINE.
...
149 / 152

Verify that all nodes are ONLINE.
...
Then on all nodes run:
150 / 152

Kill one node again:
systemd will restart mysqld and verify if the node joined.
151 / 152

Thank you !
Any Questions ?
152 / 152

MySQL InnoDB Cluster and Group Replication in a Nutshell

More Related Content

What's hot (20)

Similar to MySQL InnoDB Cluster and Group Replication in a Nutshell (20)

More from Frederic Descamps (20)

Recently uploaded (20)

MySQL InnoDB Cluster and Group Replication in a Nutshell