Percona Live Europe 2018 MySQL Group Replication... the magic explained

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purpose only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied up in
making purchasing decisions. The development, release and timing of any features or
functionality described for Oracle´s product remains at the sole discretion of Oracle.
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
3 / 168

about.me/lefred
Who am I ?
4 / 168

Frédéric Descamps
@lefred
MySQL Evangelist
Managing MySQL since 3.23
devops believer
living in Belgium 🇧🇪
https://lefred.be

5 / 168

Group Replication: heart of MySQL InnoDB
Cluster
6 / 168

Group Replication: heart of MySQL InnoDB
Cluster
7 / 168

MySQL Group Replication
but what is it ?!?
8 / 168

but what is it ?!?
GR is a plugin for MySQL, made by MySQL and packaged with MySQL
9 / 168

but what is it ?!?
GR is an implementation of Replicated Database State Machine theory
10 / 168

but what is it ?!?
Paxos based protocol (our implementation is close to Mencius)
11 / 168

but what is it ?!?
GR allows to write on all Group Members (cluster nodes) simultaneously while
retaining consistency
12 / 168

but what is it ?!?
GR implements conflict detection and resolution
13 / 168

but what is it ?!?
GR allows automatic distributed recovery
14 / 168

but what is it ?!?
GR allows automatic distributed recovery
Supported on all MySQL platforms !!
Linux, Windows, Solaris, OSX, FreeBSD
15 / 168

terminology
Write vs Writeset
16 / 168

Let's illustrate a table:
17 / 168

Now let's make a change
18 / 168

and at commit time:
19 / 168

Writesets
Contain the hash for the rows PKs that are changed and in some cases the hashes of
foreign keys or others dependencies that need to be captured (e.g. non NULL UKs).
Writesets are gathered during transaction execution.
20 / 168

Writesets
Contain the hash for the rows PKs that are changed and in some cases the hashes of
foreign keys or others dependencies that need to be captured (e.g. non NULL UKs).
Writesets are gathered during transaction execution.
Writes
Called also write values, refers to the actual changes. Write values are also gathered
during transaction execution.
21 / 168

Writeset - examples
+-------+-----------+------+-----+---------+-------+
+-------+-----------+------+-----+---------+-------+
+-------+-----------+------+-----+---------+-------+
mysql> insert into t2 values (1,2);
23 / 168

Writeset - examples
+-------+-----------+------+-----+---------+-------+
+-------+-----------+------+-----+---------+-------+
+-------+-----------+------+-----+---------+-------+
pke: PRIMARY | test | t2 | 1 | 1 hash: 11853456929268668462
24 / 168

Writeset - examples
+-------+-----------+------+-----+---------+-------+
+-------+-----------+------+-----+---------+-------+
+-------+-----------+------+-----+---------+-------+
mysql> update t2 set name=3 where id=1;
25 / 168

Writeset - examples
+-------+-----------+------+-----+---------+-------+
+-------+-----------+------+-----+---------+-------+
+-------+-----------+------+-----+---------+-------+
26 / 168

+-------+-----------+------+-----+---------+-------+
+-------+-----------+------+-----+---------+-------+
+-------+-----------+------+-----+---------+-------+
mysql> insert into t3 values (1,2,3);
28 / 168

+-------+-----------+------+-----+---------+-------+
+-------+-----------+------+-----+---------+-------+
+-------+-----------+------+-----+---------+-------+
30 / 168

+-------+-----------+------+-----+---------+-------+
+-------+-----------+------+-----+---------+-------+
+-------+-----------+------+-----+---------+-------+
pke: name | test | t3 | 3 hash: 18082071075512932388
31 / 168

+-------+-----------+------+-----+---------+-------+
+-------+-----------+------+-----+---------+-------+
+-------+-----------+------+-----+---------+-------+
[after image]
[before image]
32 / 168

GR is nice, but how does it work ?
33 / 168

it´s just ...
34 / 168

it´s just ...
35 / 168

it´s just ...
... no, in fact the writesets replication is synchronous and then certification and apply of
the changes are local to each nodes and asynchronous.
36 / 168

it´s just ...
... no, in fact the writesets replication is synchronous and then certification and apply of
the changes are local to each nodes and asynchronous.
not that easy to understand... right ? As a picture is worth a 1000 words, let´s illustrate
this...
37 / 168

38 / 168

39 / 168

40 / 168

41 / 168

42 / 168

43 / 168

44 / 168

45 / 168

46 / 168

47 / 168

48 / 168

49 / 168

50 / 168

MySQL Group Replication (full transaction)
51 / 168

52 / 168

53 / 168

54 / 168

55 / 168

56 / 168

57 / 168

58 / 168

59 / 168

MySQL Group Communication System (GCS)
MySQL Xcom protocol
60 / 168

MySQL Xcom protocol
Replicated Database State Machine
61 / 168

MySQL Xcom protocol
Paxos based protocol
62 / 168

MySQL Xcom protocol
its task: deliver messages across the distributed system:
63 / 168

MySQL Xcom protocol
atomically
64 / 168

MySQL Xcom protocol
atomically
in Total Order
65 / 168

MySQL Xcom protocol
atomically
in Total Order
MySQL Group Replication receives the Ordered 'tickets' from this GCS subsystem.
66 / 168

Total Order
GTID generation
67 / 168

How does Group Replication handle GTIDs ?
There are two ways of generating GTIDs:
68 / 168

AUTOMATIC: the transaction is assigned with an automatically generated id during
commit. Where regular replication uses the source server UUID, on Group Replication,
the group name is used.
69 / 168

AUTOMATIC: the transaction is assigned with an automatically generated id during
commit. Where regular replication uses the source server UUID, on Group Replication,
the group name is used.
ASSIGNED: the user assigns manually a GTID through SET GTID_NEXT to the
transaction. This is common to any replication format and the id is assigned before
the transaction starts.
70 / 168

71 / 168

Group Replication : Total Order Delivery - GTID
72 / 168

73 / 168

74 / 168

75 / 168

76 / 168

77 / 168

78 / 168

79 / 168

80 / 168

81 / 168

82 / 168

83 / 168

84 / 168

85 / 168

86 / 168

87 / 168

88 / 168

89 / 168

90 / 168

Group Replication : GTID
The previous example was not totally in sync with reality. In fact, a writer allocates a
block of GTID and when we have multiple writes (multi-primary mode) all writers will use
GTID sequence numbers in their allocated block.
The size of the block is defined by
group_replication_gtid_assignment_block_size (default to 1M)
91 / 168

Example:
Executed_Gtid_Set: 0b5c746d-d552-11e8-bef0-08002718d305:1-355
92 / 168

Example:
New write on an other node:
Executed_Gtid_Set: 0b5c746d-d552-11e8-bef0-08002718d305:1-355,1000354
93 / 168

Example:
Let's write on the third node:
Executed_Gtid_Set: 0b5c746d-d552-11e8-bef0-08002718d305:1-355:1000354:2000354
94 / 168

Example:
Let's write on the third node:
And writing back on the first one:
95 / 168

done !
Return from Commit
96 / 168

Group Replication: return from commit
Asynchronous Replication:
97 / 168

Group Replication: return from commit (2)
Semi-Sync Replication:
98 / 168

Group Replication: return from commit (3)
Group Replication:
99 / 168

Does this mean we can have a distant node and
always let it ack later ?
100 / 168

NO!
101 / 168

NO!
Because the system has to wait for the noop (single skip message) from the “distant”
node where latency is higher
The size of the GCS consensus messages window can be get and set from UDF functions:
group_replication_get_write_concurrency(), group_replication_set_write_concurrency()
mysql> select group_replication_get_write_concurrency();
+-------------------------------------------+
| group_replication_get_write_concurrency() |
+-------------------------------------------+
| 10 |
+-------------------------------------------+
102 / 168

Event Horizon
GCS Write Consensus Concurrency
103 / 168

Event Horizon
group replication write concurrency
104 / 168

Event Horizon
105 / 168

Event Horizon
106 / 168

Event Horizon
107 / 168

Event Horizon
108 / 168

Event Horizon
109 / 168

Event Horizon
110 / 168

conflict
Optimistic Locking
111 / 168

Group Replication : Optimistic Locking
Group Replication uses optimistic locking
112 / 168

during a transaction, local (InnoDB) locking happens
113 / 168

optimistically assumes there will be no conflicts across nodes
(no communication between nodes necessary)
114 / 168

cluster-wide conflict resolution happens only at COMMIT, during certification
115 / 168

cluster-wide conflict resolution happens only at COMMIT, during certification
Let´s first have a look at the traditional locking to compare.
116 / 168

Traditional locking
117 / 168

Traditional locking
118 / 168

Traditional locking
119 / 168

Traditional locking
120 / 168

Traditional locking
121 / 168

Traditional locking
122 / 168

Optimistic Locking
123 / 168

Optimistic Locking
124 / 168

Optimistic Locking
125 / 168

Optimistic Locking
126 / 168

Optimistic Locking
127 / 168

Optimistic Locking
128 / 168

Optimistic Locking
The system returns error 149 as certification failed:
ERROR 1180 (HY000): Got error 149 during COMMIT
129 / 168

Such conflicts happen only when using multi-
primary group !

not totally true in MySQL < 8.0.13 when failover happens
130 / 168

Drawbacks of optimistic locking
having a first-committer-wins system means conflicts will more likely happen when
writing on multiple members with:
131 / 168

large transactions
132 / 168

large transactions
long running transactions
133 / 168

large transactions
long running transactions
hotspot records
134 / 168

can the transaction be committed ?
Certification
135 / 168

Certification
Certification is the process that only needs to answer the following unique question:
136 / 168

Certification
can the write (transaction) be committed ?
137 / 168

Certification
based on yet to be applied transactions
138 / 168

Certification
such conflicts must come for other members/nodes
139 / 168

Certification
happens on every member/node and is deterministic
140 / 168

Certification
results are not reported to the group (does not require a new communication step)
141 / 168

Certification
pass: commit/queue to appy
142 / 168

Certification
fail: rollback/drop the transaction
143 / 168

Certification
serialized by the total order in GCS/XCOM + GTID
144 / 168

Certification
serialized by the total order in GCS/XCOM + GTID
cost is based on trx size (# rows & # keys)
145 / 168

Certification
146 / 168

Houston we have a problem !
Flow Control
147 / 168

Flow Control
In Group Replication, every member send statistics about its queues (applier queue and
certification queue) to the other members. Then every node decide to slow down or not
if they realize that one node reached the threshold for one of the queue.
148 / 168

Flow Control
So when group_replication_ ow_control_mode is set to QUOTA on the
node seeing that one of the other members of the cluster is lagging behind (threshold
reached), it will throttle the write operations to the a quota that is calculated based on
the number of transactions applied in the last second, and then it is reduced below that
by subtracting the “over the quota” messages from the last period.
149 / 168

Flow Control
So when group_replication_ ow_control_mode is set to QUOTA on the
node seeing that one of the other members of the cluster is lagging behind (threshold
reached), it will throttle the write operations to the a quota that is calculated based on
the number of transactions applied in the last second, and then it is reduced below that
by subtracting the “over the quota” messages from the last period.
This mean that the threshold is NOT decided on the node being slow, but the node
writing a transaction checks its threshold flow control values and compare them to the
statistics from the other nodes to decide to throttle or not.
150 / 168

Flow Control - on writer
>quota
151 / 168

Flow Control - on all members
152 / 168

Flow Control - configuration variables
As in MySQL 8.0.13:
+-----------------------------------------------------+-------+
| Variable_name | Value |
+-----------------------------------------------------+-------+
| group_replication_ ow_control_applier_threshold | 25000 |
| group_replication_ ow_control_certi er_threshold | 25000 |
| group_replication_ ow_control_hold_percent | 10 |
| group_replication_ ow_control_max_quota | 0 |
| group_replication_ ow_control_member_quota_percent | 0 |
| group_replication_ ow_control_min_quota | 0 |
| group_replication_ ow_control_min_recovery_quota | 0 |
| group_replication_ ow_control_mode | QUOTA |
| group_replication_ ow_control_period | 1 |
| group_replication_ ow_control_release_percent | 50 |
+-----------------------------------------------------+-------+
153 / 168

154 / 168

transaction's lifecycle in Group Replication
Summary
155 / 168

begin;
156 / 168

begin;
update table1
set c = 999
where id =2;
157 / 168

begin;
update table1
set c = 999
where id =2;
update table1
set b = "eee"
where id = 3;
158 / 168

begin;
update table1
set c = 999
where id =2;
update table1
set b = "eee"
where id = 3;
commit;
clientblocksoncommit...
159 / 168

begin;
update table1
set c = 999
where id =2;
update table1
set b = "eee"
where id = 3;
commit;
writesets
+ gtid_event
+ write values
160 / 168

begin;
update table1
set c = 999
where id =2;
update table1
set b = "eee"
where id = 3;
commit;
writesets
+ gtid_event
+ write values
certify
161 / 168

begin;
update table1
set c = 999
where id =2;
update table1
set b = "eee"
where id = 3;
commit;
writesets
+ gtid_event
+ write values
certify
certify
162 / 168

begin;
update table1
set c = 999
where id =2;
update table1
set b = "eee"
where id = 3;
commit;
writesets
+ gtid_event
+ write values
certify
certify
certify
163 / 168

begin;
update table1
set c = 999
where id =2;
update table1
set b = "eee"
where id = 3;
commit;
commit finalized
writesets
+ gtid_event
+ write values
certify
certify
certify
+ GTID
bin
log
164 / 168

begin;
update table1
set c = 999
where id =2;
update table1
set b = "eee"
where id = 3;
commit;
commit finalized
writesets
+ gtid_event
+ write values
certify
certify
certify
+ GTID
bin
log
+ GTID
165 / 168

begin;
update table1
set c = 999
where id =2;
update table1
set b = "eee"
where id = 3;
commit;
commit finalized
writesets
+ gtid_event
+ write values
certify
certify
certify
+ GTID
bin
log
+ GTID
+ GTIDrelay
log
relay
log
166 / 168

begin;
update table1
set c = 999
where id =2;
update table1
set b = "eee"
where id = 3;
commit;
commit finalized
writesets
+ gtid_event
+ write values
certify
certify
certify
+ GTID
bin
log
+ GTID
+ GTIDrelay
log
relay
log
bin
log
bin
log
167 / 168

Thank you !
Any Questions ?
168 / 168

Percona Live Europe 2018 MySQL Group Replication... the magic explained

More Related Content

What's hot (20)

Similar to Percona Live Europe 2018 MySQL Group Replication... the magic explained (20)

More from Frederic Descamps (20)

Recently uploaded (20)

Percona Live Europe 2018 MySQL Group Replication... the magic explained