SlideShare a Scribd company logo
Easy MySQL Replication
Setup and Troubleshooting
Bob Burgess
Salesforce ExactTarget Marketing Cloud (Radian6)
• Introduction
• Description
• Configuration Items
• Commands
• Setup
• Beyond the Basics
• Troubleshooting Faults
• Gotchas
• Closing
• Introduction
• Description
• Configuration Items
• Commands
• Setup
• Beyond the Basics
• Troubleshooting Faults
• Gotchas
• Closing
Slides
• I will post these slides on the conference site
and slideshare
• No need to copy them down
Who Am I?
• MySQL user since 2007
• Radian6: Social Media Monitoring
• Now part of
Salesforce ExactTarget Marketing Cloud
• Large Data Set
• Dozens of Databases
• Replication is CRUCIAL for us
Why Am I Doing This?
• Replication isn’t hard
• Show simplest replication case
• Share how we deal with common problems
Our First Replication
• Set up a separate database for running
reports, to separate back-end from user-facing
activity
• Alexander Rubin of MySQL Support in 2008
What is “Replication”?
Why Replication?
• Database Failure
• Share Read Traffic (Scale-Out)
• Reporting/Analytics Databases
• Network Load
• Backups
Replication Compared
• DRBD
– Block-level
– Replica is unusable until switched to
• Galera / Percona Cluster
– More complicated
– Not WAN-Friendly for High Transaction Rate
• Tungsten Replicator
• Others..?
• Introduction
• Description
• Configuration Items
• Commands
• Setup
• Beyond the Basics
• Troubleshooting Faults
• Gotchas
• Closing
Replication Described
Master Replica
Replication Described
Master
table1
table2
mysql
Update table1 set...
Insert into table2...
Binlogs:
• Events in commit order
• Contains timestamps and auto-increment IDs
client
Replica
table1
table2
IO Thread SQL Thread
Update table1 set...
Insert into table2...
mysql
• Introduction
• Description
• Configuration Items
• Commands
• Setup
• Beyond the Basics
• Troubleshooting Faults
• Gotchas
• Closing
Replication Described
log_bin=/data/log/mysql-bin
Config items
Master
table1
table2
mysql
Update table1 set...
Insert into table2...
client
Replica
table1
table2
IO Thread SQL Thread
Update table1 set...
Insert into table2...
mysql
Replication Described
server_id=1 server_id=2
Config items
(1) Master
table1
table2
mysql
Update table1 set...
Insert into table2...
client
Replica (2)
table1
table2
IO Thread SQL Thread
Update table1 set...
Insert into table2...
mysql
• Introduction
• Description
• Configuration Items
• Commands
• Setup
• Beyond the Basics
• Troubleshooting Faults
• Gotchas
• Closing
Replication Commands
• change master to
• start slave
• stop slave
• reset slave
• show slave statusG
• show master status
• mysqlbinlog
• grant replication slave
Replication Commands
change master to...
Which server? Port? Credentials? Log file & position?
(1) Master
table1
table2
mysql
client
Replica (2)
table1
table2
IO Thread SQL Thread
Update table1 set...
Insert into table2...
mysql
Update table1 set...
Insert into table2...
Replication Commands
• change master to...
Which server? Port? Credentials? Log file & position?
• change master to
master_host=‘Master’, master_port=3306,
master_user=‘repl’, master_password=‘pass’,
master_log_file=‘mysql_bin.000001’,
master_log_pos=4563744;
Replication Commands
(1) Master
table1
table2
mysql
Update table1 set...
Insert into table2...
client
Replica (2)
table1
table2
mysql
Replication Commands
change master to
master_host=‘Master’, master_port=3306,
master_user=‘repl’, master_password=‘pass’,
master_log_file=‘mysql_bin.000001’,
master_log_pos=4563744;
(1) Master
table1
table2
mysql
Update table1 set...
Insert into table2...
client
Replica (2)
table1
table2
IO Thread
Update table1 set...
Insert into table2...
mysql
Replication Commands
start slave;
(1) Master
table1
table2
mysql
Update table1 set...
Insert into table2...
client
Replica (2)
table1
table2
IO Thread SQL Thread
Update table1 set...
Insert into table2...
mysql
Replication Commands
stop slave;
(1) Master
table1
table2
mysql
Update table1 set...
Insert into table2...
client
Replica (2)
table1
table2
Update table1 set...
Insert into table2...
mysql
Replication Commands
reset slave;
(1) Master
table1
table2
mysql
Update table1 set...
Insert into table2...
client
Replica (2)
table1
table2
mysql
Replication Commands
• show slave statusG
• Manual troubleshooting
• Use in scripts for alarming
LAG=$(mysql -s -e’show slave statusG’|grep
‘Seconds_Behind_Master’|cut -f2 -d:| tr -d ‘ ‘)
IO_UP=$(mysql -s -e’show slave statusG’|grep
‘Slave_IO_Running’|cut -f2 -d:| tr -d ‘ ‘)
Slave_IO_State: Waiting for
master to send
event
Master_Host: Master
Master_User: repl
Master_Port: 3306
Connect_Retry: 3
Master_Log_File: mysql-bin.000001
Read_Master_Log_Pos: 28762843
Relay_Log_File: relay-bin.000002
Relay_Log_Pos: 548
Relay_Master_Log_File: mysql-bin.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 23456234
Relay_Log_Space: 52
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 8
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids: 0
Master_Server_Id: 1
Slave_IO_State: Waiting for
master to
send event
Master_Host: Master
Master_User: repl
Master_Port: 3306
Connect_Retry: 3
Master_Log_File: mysql-bin.000001
Read_Master_Log_Pos: 28762843
Relay_Log_File: relay-bin.000002
Relay_Log_Pos: 548
Relay_Master_Log_File: mysql-bin.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 23456234
Relay_Log_Space: 52
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 8
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids: 0
Master_Server_Id: 1
What it’s doing
Slave_IO_State: Waiting for
master to send
event
Master_Host: Master
Master_User: repl
Master_Port: 3306
Connect_Retry: 3
Master_Log_File: mysql-bin.000001
Read_Master_Log_Pos: 28762843
Relay_Log_File: relay-bin.000002
Relay_Log_Pos: 548
Relay_Master_Log_File: mysql-bin.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 23456234
Relay_Log_Space: 52
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 8
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids: 0
Master_Server_Id: 1
Where it’s connected
Slave_IO_State: Waiting for
master to send
event
Master_Host: Master
Master_User: repl
Master_Port: 3306
Connect_Retry: 3
Master_Log_File: mysql-bin.000001
Read_Master_Log_Pos: 28762843
Relay_Log_File: relay-bin.000002
Relay_Log_Pos: 548
Relay_Master_Log_File: mysql-bin.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 23456234
Relay_Log_Space: 52
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 8
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids: 0
Master_Server_Id: 1
Where the IO Thread has read up to
Master
table1
table2
mysql
Update table1 set...
Insert into table2...
client
Replica
table1
table2
IO Thread SQL Thread
Update table1 set...
Insert into table2...
mysql
Master_Log_File: mysql-bin.000001
Read_Master_Log_Pos: 28762843
Slave_IO_State: Waiting for
master to send
event
Master_Host: Master
Master_User: repl
Master_Port: 3306
Connect_Retry: 3
Master_Log_File: mysql-bin.000001
Read_Master_Log_Pos: 28762843
Relay_Log_File: relay-bin.000002
Relay_Log_Pos: 548
Relay_Master_Log_File: mysql-bin.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 23456234
Relay_Log_Space: 52
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 8
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids: 0
Master_Server_Id: 1
Where the SQL Thread has executed up to
(in relaylog)
Master
table1
table2
mysql
Update table1 set...
Insert into table2...
client
Replica
table1
table2
IO Thread SQL Thread
Update table1 set...
Insert into table2...
mysql
Relay_Log_File: relay-bin.000002
Relay_Log_Pos: 548
Slave_IO_State: Waiting for
master to send
event
Master_Host: Master
Master_User: repl
Master_Port: 3306
Connect_Retry: 3
Master_Log_File: mysql-bin.000001
Read_Master_Log_Pos: 28762843
Relay_Log_File: relay-bin.000002
Relay_Log_Pos: 548
Relay_Master_Log_File:
mysql-bin.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos:
23456234
Relay_Log_Space: 52
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 8
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids: 0
Master_Server_Id: 1
Where the SQL Thread has executed up to
(corresponding position in Master’s binlog)
Master
table1
table2
mysql
Update table1 set...
Insert into table2...
client
Replica
table1
table2
IO Thread SQL Thread
Update table1 set...
Insert into table2...
mysql
Relay_Master_Log_File: mysql-bin.000001
Exec_Master_Log_Pos: 23456234
Slave_IO_State: Waiting for
master to send
event
Master_Host: Master
Master_User: repl
Master_Port: 3306
Connect_Retry: 3
Master_Log_File: mysql-bin.000001
Read_Master_Log_Pos: 28762843
Relay_Log_File: relay-bin.000002
Relay_Log_Pos: 548
Relay_Master_Log_File: mysql-bin.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 23456234
Relay_Log_Space: 52
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 8
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids: 0
Master_Server_Id: 1
If the IO and SQL Threads are running
Slave_IO_State: Waiting for
master to send
event
Master_Host: Master
Master_User: repl
Master_Port: 3306
Connect_Retry: 3
Master_Log_File: mysql-bin.000001
Read_Master_Log_Pos: 28762843
Relay_Log_File: relay-bin.000002
Relay_Log_Pos: 548
Relay_Master_Log_File: mysql-bin.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 23456234
Relay_Log_Space: 52
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 8
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids: 0
Master_Server_Id: 1
Replication Filters in effect
Slave_IO_State: Waiting for
master to send
event
Master_Host: Master
Master_User: repl
Master_Port: 3306
Connect_Retry: 3
Master_Log_File: mysql-bin.000001
Read_Master_Log_Pos: 28762843
Relay_Log_File: relay-bin.000002
Relay_Log_Pos: 548
Relay_Master_Log_File: mysql-bin.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 23456234
Relay_Log_Space: 52
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 8
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids: 0
Master_Server_Id: 1
Replication Lag (how long ago was
timestamp of latest-processed query?)
Can be “NULL”
Slave_IO_State: Waiting for
master to send
event
Master_Host: Master
Master_User: repl
Master_Port: 3306
Connect_Retry: 3
Master_Log_File: mysql-bin.000001
Read_Master_Log_Pos: 28762843
Relay_Log_File: relay-bin.000002
Relay_Log_Pos: 548
Relay_Master_Log_File: mysql-bin.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 23456234
Relay_Log_Space: 52
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 8
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids: 0
Master_Server_Id: 1
Replication Error (SQL Thread)
Slave_IO_State: Waiting for
master to send
event
Master_Host: Master
Master_User: repl
Master_Port: 3306
Connect_Retry: 3
Master_Log_File: mysql-bin.000001
Read_Master_Log_Pos: 28762843
Relay_Log_File: relay-bin.000002
Relay_Log_Pos: 548
Relay_Master_Log_File: mysql-bin.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 23456234
Relay_Log_Space: 52
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 8
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids: 0
Master_Server_Id: 1
Replication Error (IO Thread)
• Introduction
• Description
• Configuration Items
• Commands
• Setup
• Beyond the Basics
• Troubleshooting Faults
• Gotchas
• Closing
Basic Setup Walk-Through
• Unique server_id values
my.cnf on master
[mysqld]
server_id=1
my.cnf on replica
[mysqld]
server_id=2
Basic Setup Walk-Through
• Replication User
On Master:
grant replication slave on *.* to ‘repl’
identified by ‘pass’;
Basic Setup Walk-Through
• Point to the correct binlog position
• On master:
show master status;
...once both databases are equal
• On Replica:
change master to ...
Basic Setup Walk-Through
• Start it up
• On replica:
start slave;
• Check it
• On replica:
show slave statusG
And that’s it.
• Introduction
• Description
• Configuration Items
• Commands
• Setup
• Beyond the Basics
• Troubleshooting Faults
• Gotchas
• Closing
Limiting Tables or Schemas
• If you don’t need all the tables
• In my.cnf:
replicate-do-table=schema.tableName
replicate-skip-table=schema.otherTable
• If you don’t need all the schemas
• In my.cnf
replication-do-db=schemaOne
replication_skip_db=schemaTwo
Limiting Tables
replicate-do-table=schema.tableName
(1) Master
table1
table2
mysql
Update table1 set...
Insert into table2...
client
Replica (2)
table1
table2
IO Thread SQL Thread
with filter
Update table1 set...
Insert into table2...
mysql
Limiting Columns or Rows
• If you don’t need all columns
• Subset of table for reporting or analytics
Limiting Columns or Rows
• Original table
create table x (a int primary key,
b int, c int) engine=InnoDB;
• Blackhole table on replica like master table
create table x (a int, b int, c int)
engine=blackhole;
• Target table on replica
create table y (a int primary key,
b int) engine=InnoDB;
Limiting Columns or Rows
• Before-insert trigger on replica to put data into
second “real” table
• Only columns a and b, and only for b=1
create trigger x_bi before insert on x
for each row
insert into y (a, b)
select new.a, new.b
from dual
where new.b=1;
Limiting Columns or Rows
(1) Master
table1
table2
mysql
Update table1 set...
Insert into table2...
client
Replica (2)
x y
IO Thread SQL Thread
Update table1 set...
Insert into table2...
mysql
TRG
Multi-Level
• Local and Remote DR Site
• Spreading load over many servers
M R M R
M R
R
R
R
R
R
R
R
...
Multi-Level
• Config
log_slave_updates=1
Multi-Level
Master Replica
table1
table2
table1
table2
mysql IO Thread SQL Thread
Update table1 set...
Insert into table2...
Update table1 set...
Insert into table2...
Replica
table1
table2
IO Thread SQL Thread
Update table1 set...
Insert into table2...
Update table1 set...
Insert into table2...
mysql
Types of Binlogs
• Statement-Based
– Contains the actual statements run
– Data drift is more likely (code modified by
triggers, etc)
• Row-based
– Replicates changes for each row
– Mostly unreadable by humans
– Unaffected by how the change is done (data drift less
likely)
– mysqlbinlog -v
• Introduction
• Description
• Configuration Items
• Commands
• Setup
• Beyond the Basics
• Troubleshooting Faults
• Gotchas
• Closing
Fault: Lag
• Long-running transactions
• High concurrency on master becomes single
session on replica
• show slave statusG
• Easy to alarm on
Fault: Stopping from Error
• Any error stops SQL Thread!
• Duplicate Key
• Unknown Function
• slave_exec_mode=idempotent
...suppresses duplicate-key and no-key-found
errors
• show slave statusG
Faults: Can’t Connect to Master
• Fault shows in slave status as soon as you
start slave.
• Did you create replication slave user?
• Correct user/password?
• Correct hostname/port?
• Master is running and reachable?
Fault: Can’t Find Binlog
• Clearly stated in the slave status
• Can happen if replica has been stopped (or is
lagged) and the binlogs are removed from the
master
• Execute from archive
OR
• Rebuild replica and restart from a known good
binlog position
Faults: Configuration
• replication-do-table
• replication-do-schema
• log_slave_updates
Faults: server_id
• Must be unique!
• If equal: “Seconds behind” jumps everywhere.
• Introduction
• Description
• Configuration Items
• Commands
• Setup
• Beyond the Basics
• Troubleshooting Faults
• Gotchas
• Closing
Gotcha: functions
• Functions will be run on the replica even if
only appearing in a select on the master
• Make sure all functions exist on replica
DB, even if only a dummy version
create function myFtn (a int, b int)
returns int deterministic return 0;
Gotcha: Reconnection
• When a replica is stopped and reset, until the
database is restarted, replication may
reconnect to a master after “reset slave”
change master to master_port=1234;
Gotcha: Master Crash
• Replication becomes unstable if master crashes
• Error: unknown instruction at position xxx
• Often need to use mysqlbinlog to find a
suitable position in the binlog to restart
replication from
stop slave; reset slave;
change master to ...;
start slave;
Gotcha: MySQL Versions
• Replication not completely compatible
between versions (especially later=>earlier)
• 5.5 master to 5.1 replica: character set error
“sometimes”
• Stops replication
Impossible
• One replica can have only one master
(until 5.7)
• Can’t change table or schema filters
dynamically
• Can’t put binlogs back on master server, for
recovery purposes
• Introduction
• Description
• Configuration Items
• Commands
• Setup
• Beyond the Basics
• Troubleshooting Faults
• Gotchas
• Closing
The Book
Alternatives
• Tungsten Replicator from Continuent
• Others?
Questions?
Thank you!
Bob Burgess
rburgess@salesforce.com
Don’t forget your session evaluation!

More Related Content

PDF
Percona Live 2012PPT: introduction-to-mysql-replication
PDF
Oss4b - pxc introduction
PDF
Why MySQL Replication Fails, and How to Get it Back
PPTX
MySQL Multi Master Replication
PPTX
Maria DB Galera Cluster for High Availability
PDF
Webinar slides: Introducing Galera 3.0 - Now supporting MySQL 5.6
PDF
Percona XtraDB Cluster vs Galera Cluster vs MySQL Group Replication
PPTX
MySQL Replication Overview -- PHPTek 2016
Percona Live 2012PPT: introduction-to-mysql-replication
Oss4b - pxc introduction
Why MySQL Replication Fails, and How to Get it Back
MySQL Multi Master Replication
Maria DB Galera Cluster for High Availability
Webinar slides: Introducing Galera 3.0 - Now supporting MySQL 5.6
Percona XtraDB Cluster vs Galera Cluster vs MySQL Group Replication
MySQL Replication Overview -- PHPTek 2016

What's hot (20)

PDF
9 DevOps Tips for Going in Production with Galera Cluster for MySQL - Slides
PPT
Codership's galera cluster installation and quickstart webinar march 2016
PPT
Galera webinar migration to galera cluster from my sql async replication
PDF
Galera explained 3
PDF
MariaDB Galera Cluster - Simple, Transparent, Highly Available
PDF
How to understand Galera Cluster - 2013
PDF
Webinar Slides: Migrating to Galera Cluster
PPT
Taking Full Advantage of Galera Multi Master Cluster
PPTX
Consistency between Engine and Binlog under Reduced Durability
PDF
Repair & Recovery for your MySQL, MariaDB & MongoDB / TokuMX Clusters - Webin...
PDF
Scaling with sync_replication using Galera and EC2
PDF
Best practices for MySQL High Availability
PDF
Advanced Percona XtraDB Cluster in a nutshell... la suite
PDF
Introduction to Galera
PDF
Galera 3.0 Webinar Slides: Galera Monitoring & Management
PDF
Galera Cluster 3.0 Features
PDF
Performance Tuning Best Practices
PDF
A26 MariaDB : The New&Implemented MySQL Branch by Colin Charles
PDF
MariaDB 5.5 and what comes next - Percona Live NYC 2012
9 DevOps Tips for Going in Production with Galera Cluster for MySQL - Slides
Codership's galera cluster installation and quickstart webinar march 2016
Galera webinar migration to galera cluster from my sql async replication
Galera explained 3
MariaDB Galera Cluster - Simple, Transparent, Highly Available
How to understand Galera Cluster - 2013
Webinar Slides: Migrating to Galera Cluster
Taking Full Advantage of Galera Multi Master Cluster
Consistency between Engine and Binlog under Reduced Durability
Repair & Recovery for your MySQL, MariaDB & MongoDB / TokuMX Clusters - Webin...
Scaling with sync_replication using Galera and EC2
Best practices for MySQL High Availability
Advanced Percona XtraDB Cluster in a nutshell... la suite
Introduction to Galera
Galera 3.0 Webinar Slides: Galera Monitoring & Management
Galera Cluster 3.0 Features
Performance Tuning Best Practices
A26 MariaDB : The New&Implemented MySQL Branch by Colin Charles
MariaDB 5.5 and what comes next - Percona Live NYC 2012
Ad

Similar to Easy MySQL Replication Setup and Troubleshooting (20)

ODP
MySQL 101 PHPTek 2017
PDF
MySQL Replication Basics -Ohio Linux Fest 2016
PDF
MySQL Replication Update -- Zendcon 2016
PPT
Basic Knowledge on MySql Replication
PPTX
ConFoo MySQL Replication Evolution : From Simple to Group Replication
PPTX
MySQL Replication Evolution -- Confoo Montreal 2017
PDF
MySQL Replication Troubleshooting for Oracle DBAs
PPT
Download presentation
PPT
Download presentation531
PPT
Mysql replication @ gnugroup
PPT
MySQL 5.1 Replication
PPT
Download presentation
PDF
MySQL database replication
PDF
2012 scale replication
PDF
MySQL Webinar 2/4 Performance tuning, hardware, optimisation
PPTX
mysql replication
DOCX
Mater,slave on mysql
PDF
2012 replication
PPTX
Migrate your EOL MySQL servers to HA Complaint GR Cluster / InnoDB Cluster Wi...
PPT
MySQL Replication Basics
MySQL 101 PHPTek 2017
MySQL Replication Basics -Ohio Linux Fest 2016
MySQL Replication Update -- Zendcon 2016
Basic Knowledge on MySql Replication
ConFoo MySQL Replication Evolution : From Simple to Group Replication
MySQL Replication Evolution -- Confoo Montreal 2017
MySQL Replication Troubleshooting for Oracle DBAs
Download presentation
Download presentation531
Mysql replication @ gnugroup
MySQL 5.1 Replication
Download presentation
MySQL database replication
2012 scale replication
MySQL Webinar 2/4 Performance tuning, hardware, optimisation
mysql replication
Mater,slave on mysql
2012 replication
Migrate your EOL MySQL servers to HA Complaint GR Cluster / InnoDB Cluster Wi...
MySQL Replication Basics
Ad

Recently uploaded (20)

PDF
TRAFFIC-MANAGEMENT-AND-ACCIDENT-INVESTIGATION-WITH-DRIVING-PDF-FILE.pdf
PPT
ISS -ESG Data flows What is ESG and HowHow
PPTX
Introduction to Knowledge Engineering Part 1
PDF
annual-report-2024-2025 original latest.
PPTX
advance b rammar.pptxfdgdfgdfsgdfgsdgfdfgdfgsdfgdfgdfg
PDF
22.Patil - Early prediction of Alzheimer’s disease using convolutional neural...
PPTX
climate analysis of Dhaka ,Banglades.pptx
PPTX
DISORDERS OF THE LIVER, GALLBLADDER AND PANCREASE (1).pptx
PPTX
STUDY DESIGN details- Lt Col Maksud (21).pptx
PDF
“Getting Started with Data Analytics Using R – Concepts, Tools & Case Studies”
PDF
Galatica Smart Energy Infrastructure Startup Pitch Deck
PPTX
Qualitative Qantitative and Mixed Methods.pptx
PDF
Business Analytics and business intelligence.pdf
PPTX
Introduction to machine learning and Linear Models
PPTX
iec ppt-1 pptx icmr ppt on rehabilitation.pptx
PPTX
AI Strategy room jwfjksfksfjsjsjsjsjfsjfsj
PPTX
Database Infoormation System (DBIS).pptx
PPT
Reliability_Chapter_ presentation 1221.5784
PPTX
Introduction to Basics of Ethical Hacking and Penetration Testing -Unit No. 1...
PPTX
Introduction to Firewall Analytics - Interfirewall and Transfirewall.pptx
TRAFFIC-MANAGEMENT-AND-ACCIDENT-INVESTIGATION-WITH-DRIVING-PDF-FILE.pdf
ISS -ESG Data flows What is ESG and HowHow
Introduction to Knowledge Engineering Part 1
annual-report-2024-2025 original latest.
advance b rammar.pptxfdgdfgdfsgdfgsdgfdfgdfgsdfgdfgdfg
22.Patil - Early prediction of Alzheimer’s disease using convolutional neural...
climate analysis of Dhaka ,Banglades.pptx
DISORDERS OF THE LIVER, GALLBLADDER AND PANCREASE (1).pptx
STUDY DESIGN details- Lt Col Maksud (21).pptx
“Getting Started with Data Analytics Using R – Concepts, Tools & Case Studies”
Galatica Smart Energy Infrastructure Startup Pitch Deck
Qualitative Qantitative and Mixed Methods.pptx
Business Analytics and business intelligence.pdf
Introduction to machine learning and Linear Models
iec ppt-1 pptx icmr ppt on rehabilitation.pptx
AI Strategy room jwfjksfksfjsjsjsjsjfsjfsj
Database Infoormation System (DBIS).pptx
Reliability_Chapter_ presentation 1221.5784
Introduction to Basics of Ethical Hacking and Penetration Testing -Unit No. 1...
Introduction to Firewall Analytics - Interfirewall and Transfirewall.pptx

Easy MySQL Replication Setup and Troubleshooting

  • 1. Easy MySQL Replication Setup and Troubleshooting Bob Burgess Salesforce ExactTarget Marketing Cloud (Radian6)
  • 2. • Introduction • Description • Configuration Items • Commands • Setup • Beyond the Basics • Troubleshooting Faults • Gotchas • Closing
  • 3. • Introduction • Description • Configuration Items • Commands • Setup • Beyond the Basics • Troubleshooting Faults • Gotchas • Closing
  • 4. Slides • I will post these slides on the conference site and slideshare • No need to copy them down
  • 5. Who Am I? • MySQL user since 2007 • Radian6: Social Media Monitoring • Now part of Salesforce ExactTarget Marketing Cloud • Large Data Set • Dozens of Databases • Replication is CRUCIAL for us
  • 6. Why Am I Doing This? • Replication isn’t hard • Show simplest replication case • Share how we deal with common problems
  • 7. Our First Replication • Set up a separate database for running reports, to separate back-end from user-facing activity • Alexander Rubin of MySQL Support in 2008
  • 9. Why Replication? • Database Failure • Share Read Traffic (Scale-Out) • Reporting/Analytics Databases • Network Load • Backups
  • 10. Replication Compared • DRBD – Block-level – Replica is unusable until switched to • Galera / Percona Cluster – More complicated – Not WAN-Friendly for High Transaction Rate • Tungsten Replicator • Others..?
  • 11. • Introduction • Description • Configuration Items • Commands • Setup • Beyond the Basics • Troubleshooting Faults • Gotchas • Closing
  • 13. Replication Described Master table1 table2 mysql Update table1 set... Insert into table2... Binlogs: • Events in commit order • Contains timestamps and auto-increment IDs client Replica table1 table2 IO Thread SQL Thread Update table1 set... Insert into table2... mysql
  • 14. • Introduction • Description • Configuration Items • Commands • Setup • Beyond the Basics • Troubleshooting Faults • Gotchas • Closing
  • 15. Replication Described log_bin=/data/log/mysql-bin Config items Master table1 table2 mysql Update table1 set... Insert into table2... client Replica table1 table2 IO Thread SQL Thread Update table1 set... Insert into table2... mysql
  • 16. Replication Described server_id=1 server_id=2 Config items (1) Master table1 table2 mysql Update table1 set... Insert into table2... client Replica (2) table1 table2 IO Thread SQL Thread Update table1 set... Insert into table2... mysql
  • 17. • Introduction • Description • Configuration Items • Commands • Setup • Beyond the Basics • Troubleshooting Faults • Gotchas • Closing
  • 18. Replication Commands • change master to • start slave • stop slave • reset slave • show slave statusG • show master status • mysqlbinlog • grant replication slave
  • 19. Replication Commands change master to... Which server? Port? Credentials? Log file & position? (1) Master table1 table2 mysql client Replica (2) table1 table2 IO Thread SQL Thread Update table1 set... Insert into table2... mysql Update table1 set... Insert into table2...
  • 20. Replication Commands • change master to... Which server? Port? Credentials? Log file & position? • change master to master_host=‘Master’, master_port=3306, master_user=‘repl’, master_password=‘pass’, master_log_file=‘mysql_bin.000001’, master_log_pos=4563744;
  • 21. Replication Commands (1) Master table1 table2 mysql Update table1 set... Insert into table2... client Replica (2) table1 table2 mysql
  • 22. Replication Commands change master to master_host=‘Master’, master_port=3306, master_user=‘repl’, master_password=‘pass’, master_log_file=‘mysql_bin.000001’, master_log_pos=4563744; (1) Master table1 table2 mysql Update table1 set... Insert into table2... client Replica (2) table1 table2 IO Thread Update table1 set... Insert into table2... mysql
  • 23. Replication Commands start slave; (1) Master table1 table2 mysql Update table1 set... Insert into table2... client Replica (2) table1 table2 IO Thread SQL Thread Update table1 set... Insert into table2... mysql
  • 24. Replication Commands stop slave; (1) Master table1 table2 mysql Update table1 set... Insert into table2... client Replica (2) table1 table2 Update table1 set... Insert into table2... mysql
  • 25. Replication Commands reset slave; (1) Master table1 table2 mysql Update table1 set... Insert into table2... client Replica (2) table1 table2 mysql
  • 26. Replication Commands • show slave statusG • Manual troubleshooting • Use in scripts for alarming LAG=$(mysql -s -e’show slave statusG’|grep ‘Seconds_Behind_Master’|cut -f2 -d:| tr -d ‘ ‘) IO_UP=$(mysql -s -e’show slave statusG’|grep ‘Slave_IO_Running’|cut -f2 -d:| tr -d ‘ ‘)
  • 27. Slave_IO_State: Waiting for master to send event Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1
  • 28. Slave_IO_State: Waiting for master to send event Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1 What it’s doing
  • 29. Slave_IO_State: Waiting for master to send event Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1 Where it’s connected
  • 30. Slave_IO_State: Waiting for master to send event Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1 Where the IO Thread has read up to
  • 31. Master table1 table2 mysql Update table1 set... Insert into table2... client Replica table1 table2 IO Thread SQL Thread Update table1 set... Insert into table2... mysql Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843
  • 32. Slave_IO_State: Waiting for master to send event Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1 Where the SQL Thread has executed up to (in relaylog)
  • 33. Master table1 table2 mysql Update table1 set... Insert into table2... client Replica table1 table2 IO Thread SQL Thread Update table1 set... Insert into table2... mysql Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548
  • 34. Slave_IO_State: Waiting for master to send event Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1 Where the SQL Thread has executed up to (corresponding position in Master’s binlog)
  • 35. Master table1 table2 mysql Update table1 set... Insert into table2... client Replica table1 table2 IO Thread SQL Thread Update table1 set... Insert into table2... mysql Relay_Master_Log_File: mysql-bin.000001 Exec_Master_Log_Pos: 23456234
  • 36. Slave_IO_State: Waiting for master to send event Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1 If the IO and SQL Threads are running
  • 37. Slave_IO_State: Waiting for master to send event Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1 Replication Filters in effect
  • 38. Slave_IO_State: Waiting for master to send event Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1 Replication Lag (how long ago was timestamp of latest-processed query?) Can be “NULL”
  • 39. Slave_IO_State: Waiting for master to send event Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1 Replication Error (SQL Thread)
  • 40. Slave_IO_State: Waiting for master to send event Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1 Replication Error (IO Thread)
  • 41. • Introduction • Description • Configuration Items • Commands • Setup • Beyond the Basics • Troubleshooting Faults • Gotchas • Closing
  • 42. Basic Setup Walk-Through • Unique server_id values my.cnf on master [mysqld] server_id=1 my.cnf on replica [mysqld] server_id=2
  • 43. Basic Setup Walk-Through • Replication User On Master: grant replication slave on *.* to ‘repl’ identified by ‘pass’;
  • 44. Basic Setup Walk-Through • Point to the correct binlog position • On master: show master status; ...once both databases are equal • On Replica: change master to ...
  • 45. Basic Setup Walk-Through • Start it up • On replica: start slave; • Check it • On replica: show slave statusG
  • 47. • Introduction • Description • Configuration Items • Commands • Setup • Beyond the Basics • Troubleshooting Faults • Gotchas • Closing
  • 48. Limiting Tables or Schemas • If you don’t need all the tables • In my.cnf: replicate-do-table=schema.tableName replicate-skip-table=schema.otherTable • If you don’t need all the schemas • In my.cnf replication-do-db=schemaOne replication_skip_db=schemaTwo
  • 49. Limiting Tables replicate-do-table=schema.tableName (1) Master table1 table2 mysql Update table1 set... Insert into table2... client Replica (2) table1 table2 IO Thread SQL Thread with filter Update table1 set... Insert into table2... mysql
  • 50. Limiting Columns or Rows • If you don’t need all columns • Subset of table for reporting or analytics
  • 51. Limiting Columns or Rows • Original table create table x (a int primary key, b int, c int) engine=InnoDB; • Blackhole table on replica like master table create table x (a int, b int, c int) engine=blackhole; • Target table on replica create table y (a int primary key, b int) engine=InnoDB;
  • 52. Limiting Columns or Rows • Before-insert trigger on replica to put data into second “real” table • Only columns a and b, and only for b=1 create trigger x_bi before insert on x for each row insert into y (a, b) select new.a, new.b from dual where new.b=1;
  • 53. Limiting Columns or Rows (1) Master table1 table2 mysql Update table1 set... Insert into table2... client Replica (2) x y IO Thread SQL Thread Update table1 set... Insert into table2... mysql TRG
  • 54. Multi-Level • Local and Remote DR Site • Spreading load over many servers M R M R M R R R R R R R R ...
  • 56. Multi-Level Master Replica table1 table2 table1 table2 mysql IO Thread SQL Thread Update table1 set... Insert into table2... Update table1 set... Insert into table2... Replica table1 table2 IO Thread SQL Thread Update table1 set... Insert into table2... Update table1 set... Insert into table2... mysql
  • 57. Types of Binlogs • Statement-Based – Contains the actual statements run – Data drift is more likely (code modified by triggers, etc) • Row-based – Replicates changes for each row – Mostly unreadable by humans – Unaffected by how the change is done (data drift less likely) – mysqlbinlog -v
  • 58. • Introduction • Description • Configuration Items • Commands • Setup • Beyond the Basics • Troubleshooting Faults • Gotchas • Closing
  • 59. Fault: Lag • Long-running transactions • High concurrency on master becomes single session on replica • show slave statusG • Easy to alarm on
  • 60. Fault: Stopping from Error • Any error stops SQL Thread! • Duplicate Key • Unknown Function • slave_exec_mode=idempotent ...suppresses duplicate-key and no-key-found errors • show slave statusG
  • 61. Faults: Can’t Connect to Master • Fault shows in slave status as soon as you start slave. • Did you create replication slave user? • Correct user/password? • Correct hostname/port? • Master is running and reachable?
  • 62. Fault: Can’t Find Binlog • Clearly stated in the slave status • Can happen if replica has been stopped (or is lagged) and the binlogs are removed from the master • Execute from archive OR • Rebuild replica and restart from a known good binlog position
  • 63. Faults: Configuration • replication-do-table • replication-do-schema • log_slave_updates
  • 64. Faults: server_id • Must be unique! • If equal: “Seconds behind” jumps everywhere.
  • 65. • Introduction • Description • Configuration Items • Commands • Setup • Beyond the Basics • Troubleshooting Faults • Gotchas • Closing
  • 66. Gotcha: functions • Functions will be run on the replica even if only appearing in a select on the master • Make sure all functions exist on replica DB, even if only a dummy version create function myFtn (a int, b int) returns int deterministic return 0;
  • 67. Gotcha: Reconnection • When a replica is stopped and reset, until the database is restarted, replication may reconnect to a master after “reset slave” change master to master_port=1234;
  • 68. Gotcha: Master Crash • Replication becomes unstable if master crashes • Error: unknown instruction at position xxx • Often need to use mysqlbinlog to find a suitable position in the binlog to restart replication from stop slave; reset slave; change master to ...; start slave;
  • 69. Gotcha: MySQL Versions • Replication not completely compatible between versions (especially later=>earlier) • 5.5 master to 5.1 replica: character set error “sometimes” • Stops replication
  • 70. Impossible • One replica can have only one master (until 5.7) • Can’t change table or schema filters dynamically • Can’t put binlogs back on master server, for recovery purposes
  • 71. • Introduction • Description • Configuration Items • Commands • Setup • Beyond the Basics • Troubleshooting Faults • Gotchas • Closing
  • 73. Alternatives • Tungsten Replicator from Continuent • Others?