Webinar slides: Managing MySQL Replication for High Availability

Copyright 2015 Severalnines AB
Managing MySQL Replication for High Availability
February 2, 2016
Krzysztof Książek
Severalnines
krzysztof@severalnines.com
1

2
Your host & some logistics
I'm Jean-Jérôme from the Severalnines Team
and I'm your host for today's webinar!
Feel free to ask any questions in the
Questions section of this application or via
the Chat box.
You can also contact me directly via the chat
box or via email: jj@severalnines.com
during or after the webinar.

! MySQL replication and ClusterControl (Demo)
! Deployment
! Topology changes
! Important metrics
! How to deal with
! Schema changes
! Topology changes
! Potential issues in MySQL replication 
3
Agenda

! Available since 3.23
! Proven, robust, based on binary logs
! GTID introduced in MySQL 5.6
! Multi-threaded replication introduced in 5.6, vastly
improved in 5.7
! Asynchronous or semisynchronous
! Backbone of majority of large MySQL deployments
4
MySQL Replication

MySQL replication and ClusterControl
5

! Deployment process
! Add slave
! Promote slave
! Recover slave
! Monitoring of the replication environment
6

7
DEMO

Deployment scenarios for replication
8

! Single DC
! Master - slave
! Intermediate master
! Master - master
! Multi-DC
! Master - slave, over WAN
! Active - passive
! WAN replication with Galera or MySQL Cluster 
9

10
Master - slave
Intermediate master

11
Master - master, active - standby
Two synchronous clusters 
connected over WAN

How to deal with schema changes?
12

! Schema changes in MySQL
! Online DDL’s
! not-so-online DDL’s
! Rolling schema upgrades
! external tools like pt-online-schema-change
13

! With MySQL replication lag is a serious limiting factor in
DDL’s
! Online DDL will be online only on the master
! Replication will be locked for the duration of the DDL
! Even when you use Multi-Threaded Slave
! Direct alters are suitable only for small tables
! unless you can accept replication lag, of course
14

! Rolling Schema upgrades - run DDL from the bottom to the top
of replication chain
! SET SESSION sql_log_bin=0 to avoid errant transactions
! Once you reach the master, failover and then alter the old
master
! Schema changes have to be compatible, you’ll be running
DML’s on a mix of altered and not altered tables
! Schema changes have to be compatible, you’ll be reading
from a mix of altered and not altered tables 
15

! pt-online-schema-change - part of the Percona Toolkit
! Create a new table with desired schema
! Setup triggers to copy data from old to the new table
! Copy the data from old to new table in batches
! Rename table once the data has been copied
16

17

! Point-in-time change, you can use for any DDL
! Replication-aware, it can monitor lag and throttle itself
! Introduces additional load, slow down your system
! It has issues with foreign keys
! Can’t work with tables which have triggers
! Won’t work with tables without PK and unique key
18

! Otherwise pt-online-schema-change is a very reliable tool
! Schema change may take a long time but it’s throttling
itself and can be paused at any time
! One of the most important tools in the DBA arsenal
! Sometimes it’s the only way to run a schema change
without downtime
19

How to deal with topology changes?
20

! If you use GTID - it’s easy
! GTID allows for a flexible topology change
! CHANGE MASTER TO … MASTER_AUTO_POSITION=1;
! You can slave off every node in the replication chain
! Be aware of errant transactions, though
! http://www.severalnines.com/blog/mysql-replication-and-
gtid-based-failover-deep-dive-errant-transactions
21

! If you do not use GITD, things are more complex
! Slave knows only a binlog position of itself and it’s master
! You can use binary and relay logs to identify position of a
given transaction in the replication chain but it’s a
complex and error prone process
! To avoid it, you have to stop writes on a involved subset of
hosts
22

23

! First step requires to slave
DB3 and DB4 off DB2
! You need to enable log-
slave-updates on DB2
! Stop DB2, DB3 and DB4 at
the same position using
START SLAVE UNTIL …
! Start it until a beginning of
binary log two files later
24

! Locate your first position in binlog using mysqlbinlog tool
! START SLAVE UNTIL master_log_file='mysql-bin.000121',
master_log_pos=4;
! Run FLUSH LOGS on the master two times to rotate logs -
slaves should end up stopped at the same position
! Check the position on DB2 (SHOW MASTER STATUS)
! Use it to slave DB3 and DB4 off DB2
25

! We should finally reach the
desired topology
! Next step - setup master -
master replication between
DB1 and DB2
! Confirm that DB2 is not taking
writes other than the replication
! Use server_id in binary logs for
that - you should see only DB1
! If it does writes, you’re in
trouble - investigate and
eliminate the culprit
26

! Once confirmed writes are
not hitting DB2, execute
CHANGE MASTER TO … on
DB1 pointing it to the DB2 and
using any recent coordinates
! Monitor Exec_Master_Log_Pos
in the SHOW SLAVE STATUS on
DB1, it should be stable. If it
does increase, something is
still writing to DB2.
! Once all is set up, you are
ready for a failover
27

Potential issues in MySQL replication
28

! Lag is inevitable part of MySQL replication, async or
semisync, it doesn’t matter
! Lag can be caused by large number of writes and DDL’s
! Multi-threaded slave + GTID is a good choice to minimize
lag caused by writes (for DDL’s - use pt-online-schema-
change)
! Hardware has to follow too, especially I/O subsystem
29

! Data drift may happen even if you take all necessary
precautions
! Use ROW binary log format for the best data consistency
! Use read_only and super_read_only on slaves to minimize
possibilities of errant writes
! Use pt-table-checksum and pt-table-sync to keep an eye on
the data consistency
! When in doubt, rebuild a slave from the source of truth (master) 
30

! Errant transactions can ruin your day
! Those are transactions executed on a slave but not on a
master
! read_only and super_read_only can help to avoid them
! GTID_SUBSET(), GTID_SUBTRACT() will help you to detect them
! We’ve covered the topic here: 
http://www.severalnines.com/blog/mysql-replication-and-
gtid-based-failover-deep-dive-errant-transactions 
31

! More blogs featuring MySQL replication:
! http://severalnines.com/blog/become-mysql-dba-blog-series-
common-operations-schema-changes
! http://severalnines.com/blog/become-mysql-dba-blog-series-
common-operations-replication-topology-changes
! http://severalnines.com/blog/become-mysql-dba-blog-series-live-
migration-using-mysql-replication
! http://severalnines.com/tutorials/mysql-replication-high-availability-
tutorial
! Contact: krzysztof@severalnines.com 
32
Thank You!

Webinar slides: Managing MySQL Replication for High Availability

More Related Content

What's hot (20)

Similar to Webinar slides: Managing MySQL Replication for High Availability (20)

More from Severalnines (20)

Recently uploaded (20)

Webinar slides: Managing MySQL Replication for High Availability