SlideShare a Scribd company logo

MySQL User Camp: Multi-threaded Slaves

Shivji Kumar Jha, profile picture
Shivji Kumar Jha

The document discusses MySQL replication, detailing its purpose in enhancing scalability and redundancy by allowing data changes from a master server to be copied to slave servers. It introduces multi-threaded slave capabilities to improve performance and outlines processes, logs, and internal threads involved in replication. Additionally, it highlights advancements in parallel processing for more efficient data handling across multiple databases in newer MySQL versions.

TechnologyNews & Politics
1 of 66
Downloaded 74 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
MySQL Replication and Multi-threaded Slaves Shivji Kumar Jha, Software Developer, MySQL Replication Team
Safe Harbour Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. `2 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
Agenda Why Replication?  Replication Internals- A Simple Introduction  Why Multi-threaded Slaves (MTS)?  Different policies for Multi-threading Slaves  `3  Keeping track of replication threads Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
Replication: Copy Changes Master → Slave  MySQL Master Server – Changes data – Sends changes to slave  MySQL Slave Server – Receives changes from master – Applies received changes to database M `4 S Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
Why Replication? – Scalability  Read scale-out M write clients `5 S read clients Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
Why Replication? – Scalability  Read scale-out M write clients `6 S More reads? read clients Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
Why Replication? – Scalability  Read scale-out M write clients `7 S More reads? More slaves! read clients Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
Why Replication? – Scalability  Read scale-out S M S More reads? More slaves! M S S write clients write clients `8 read clients Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | read clients
Why Replication? – Redundancy  If master crashes, promote slave to master B A C `9 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
Why Replication? – Redundancy  If master crashes, promote slave to master B Crash A C `10 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
Why Replication? – Redundancy  If master crashes, promote slave to master B is the new master B A C `11 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
Why Replication? – Disaster recovery C A A B C B Image from www.ginkgomaps.com `12 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
But how do you copy changes to slave? `13 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
But how do you copy changes to slave? Are there LOGS floating around? `14 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
Introducing replication LOGS... `15 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
Replication Logs  MySQL Master Server M – Changes data (Writes) Binary log – Saves changes in Binary log  MySQL Slave Server – Copies changes to Relay log – Reads Relay log – Applies changes. `16 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | Relay log S
All Changes Written to Binary Log Client binary log A `17 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
All Changes Written to Binary Log Client create table t (a int); binary log A `18 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
All Changes Written to Binary Log Client create table t (a int); binary log create... A Table t `19 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. |
All Changes Written to Binary Log Client create table t (a int); insert into t values (1); binary log create... A Table t `20 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
All Changes Written to Binary Log Client create table t (a int); insert into t values (1); binary log A create... insert... Table t 1 `21 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
Slave Initiates Replication 1. Slave sends request to start replication to master Client relay log binary log B A 2. Master sends stream of replication data to slave `22 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
Slave Copies, Applies Same Changes Client create... binary log A `23 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | relay log B
Slave Copies, Applies Same Changes Client create... binary log relay log create... A Table t `24 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | B
Slave Copies, Applies Same Changes Client create... binary log relay log create... create... A Table t `25 B Table t Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
Slave Copies, Applies Same Changes Client create... insert... binary log A relay log create... insert... create... Table t 1 `26 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | B Table t
Slave Copies, Applies Same Changes Client create... insert... binary log A relay log create... insert... create... insert... Table t 1 `27 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | B Table t 1
OK makes sense ! `28 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
But I see clients applying changes at Master, who applies at Slave? `29 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
But I see clients applying changes at Master, who applies at Slave? Do you have some internal replication threads? `30 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
Introducing replication THREADS... `31 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
Replication Threads (4.0-5.5) Client create... insert... relay log binary log A create... insert... create... network insert... SQL THREAD IO THREAD Writes into relay log at slave `32 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | B Reads from relay log. Applies transactions at slave
Replication Threads (4.0-5.5) Create... insert... Client Client insert... More clients on master... Client relay log binary log A `33 create... insert... Insert... create... network insert... SQL THREAD IO THREAD Insert... Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | B
Replication Threads (4.0-5.5) Create... insert... Client Client insert... More clients on master... Client Slave starts to lag behind master relay log binary log A `34 create... insert... Insert... create... network insert... SQL THREAD IO THREAD Insert... Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | B
Replication Threads (4.0-5.5) Create... insert... Client Client insert... More clients on master... Client Slave starts to lag behind master relay log binary log A create... insert... Insert... create... network insert... SQL THREAD IO THREAD Insert... Table t 1 2 3 `35 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | B Table t 1 2
Lets Parallelize slave as well... `36 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
Lets Parallelize slave as well... Introducing parallelization by Database `37 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
Replication Threads (5.6+) create... Client insert... insert... Reads & assigns to worker(s) Client Client w create.. O insert... R K COORDINATOR E insert.. THREAD R S binary log A `38 create(db1) insert(db2) insert(db3) network IO THREAD Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | db1 B db2 db3 create(db1) insert(db2) insert(db3) Worker threads apply concurrently on diff dbs relay log
Replication Threads (5.6+) Setting up Parallelization by database mysql> STOP SLAVE; mysql> SET GLOBAL slave_parallel_workers=1; mysql> START SLAVE; `39 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
Replication Threads (5.6+) More databases? More workers! mysql> STOP SLAVE; mysql> SET GLOBAL slave_parallel_workers=2; mysql> START SLAVE; `40 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
What if I have a a loaded Database? Image credits: http://www.easetechnology.co.uk/ask-an-expert/ `41 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
Lets Parallelize slave somewhat similar to master `42 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
Lets Parallelize slave somewhat similar to master 5.7.2 Introducing parallelization by Master concurrency `43 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
Replication Threads (4.0-5.5) Create... insert... Client Client insert... More clients on master... Client Slave starts to lag behind master relay log binary log A Table t 1 2 3 `44 create... insert... Insert... create... network insert... SQL THREAD IO THREAD Insert... REVISITING THE OLD SLIDE !!! Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | B Table t 1 2
Replication Threads (5.7+) create... Client insert... insert... Reads & assigns to worker(s) Client Client COORDINATOR THREAD binary log A create... insert... Insert... network IO THREAD capture parallelization info on master `45 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | create... insert... Insert... relay log w O R K E R S B db1 db2 Worker threads apply concurrently
Replication Threads (5.7+) create... Client insert... insert... Reads & assigns to worker(s) Client Client COORDINATOR THREAD binary log A create... insert... Insert... network IO THREAD Save parallelization info in binary log `46 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | create... insert... Insert... relay log w O R K E R S B db1 db2 Worker threads apply concurrently
Replication Threads (5.7+) create... Client insert... insert... Reads & assigns to worker(s) Client Client COORDINATOR THREAD binary log A create... insert... Insert... network IO THREAD Send parallelization info to slave `47 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | create... insert... Insert... relay log w O R K E R S B db1 db2 Worker threads apply concurrently
Replication Threads (5.7+) create... Client insert... insert... Reads & assigns to worker(s) Client Client COORDINATOR THREAD binary log A create... insert... Insert... network IO THREAD Use parallelization info to assign `48 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | create... insert... Insert... relay log w O R K E R S B db1 db2 Worker threads apply concurrently
Replication Threads (5.7+)   `49 Leverage parallelization information obtained from the execution on the master: — Transactions that reach the prepare phase on the same data snapshot are non-contending; — Write to the binary log on which snapshot id each transaction prepared;  This identifies the commit parent of each transaction. — The commit parent is stepped every time transactions commit. Meanwhile, at the slave: — Transactions with the same commit parent can be executed in parallel; — Commit sequence at the slave may not be the same as that on the master, but it is still a correct execution history. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. |
Higher Slave Throughput: Intra-Schema MTS Concurrent Execution History on the Master T1 T2 T3 Time Execution Commit Prepare Commit `50 commit_parent++ Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Prepare
Higher Slave Throughput: Intra-Schema MTS Concurrent Execution History on the Master Parallel on the Slave. T1 T2 T3 Time Execution Commit Prepare Commit `51 commit_parent++ Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Prepare Not parallel on the slave.
Replication Threads (5.7+) Setting up Parallelization by master concurrency mysql> STOP SLAVE; mysql> SET GLOBAL slave_parallel_type=[ 'logical_clock' | 'database' ]; mysql> SET GLOBAL slave_parallel_workers=3; mysql> START SLAVE; database=>parallelize by database logical_clock=>parallelize by master concurrency `52 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
Too many threads here and there! How do I monitor? Knew that was coming! Check out replication Performance_Schema `53 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
Performance Schema (5.5+)  Inspect internal execution of the server at runtime  Exposed within performance_schema database  Records various run time statistics via in-built instrumentation points  Tracks latency for various events: File I/O, Mutexes, Read/Write Locks, Table I/O, Table Locks, Stages, Statements, Idle etc..  `54 Its a long growing list Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
Replication Performance Schema Tables (5.7+) replication 5.7.2 execution connection configuration status configuration Coordinator thread / SQL thread `55 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | status Worker thread(s)
The REPLICATION P_S Tables  We have six performance schema tables for replication: 5.7.2 (MySQL-5.7.2):  replication_connection_configuration  replication_connection_status (IO thread status)  replication_execute_configuration  replication_execute_status  replication_execute_status_by_coordinator  replication_execute_status_by_worker Tables for monitoring MTS `56 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
Monitoring The Coordinator Thread 5.7.2 mysql> select * from performance_schema.replication_execute_status_by_coordinatorG *************************** 1. row ***************************            THREAD_ID: 13        SERVICE_STATE: ON    LAST_ERROR_NUMBER: 0   LAST_ERROR_MESSAGE: LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00 `57 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | No error in thread
Monitoring The Coordinator Thread 5.7.2 mysql> select * from performance_schema.replication_execute_status_by_coordinatorG *************************** 1. row ***************************            THREAD_ID: 13        SERVICE_STATE: ON Oops! There was an error    LAST_ERROR_NUMBER: 1146   LAST_ERROR_MESSAGE:...'Table 'test.t' doesn't exist'... LAST_ERROR_TIMESTAMP: 2013-11-04 13:37:23 `58 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
Monitoring Worker Threads 5.7.2 mysql> SET GLOBAL slave_parallel_workers= 2; mysql> select * from performance_schema.replication_execute_status_by_workerG *************************** 1. row ***************************             WORKER_ID: 0             THREAD_ID: 16         SERVICE_STATE: ON LAST_SEEN_TRANSACTION: 2b3e03a6-453f-11e3-8668-0021ccb3570d:3     LAST_ERROR_NUMBER: 0    LAST_ERROR_MESSAGE: LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00 *************************** 2. row ***************************             WORKER_ID: 1             THREAD_ID: 17 one row per worker thread         SERVICE_STATE: ON ... `59 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
Monitoring Worker Threads 5.7.2 mysql> SET GLOBAL slave_parallel_workers= 2; mysql> select * from performance_schema.replication_execute_status_by_workerG *************************** 1. row ***************************             WORKER_ID: 0             THREAD_ID: 16         SERVICE_STATE: ON LAST_SEEN_TRANSACTION: 2b3e03a6-453f-11e3-8668-0021ccb3570d:3     LAST_ERROR_NUMBER: 0    LAST_ERROR_MESSAGE: LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00 *************************** 2. row ***************************             WORKER_ID: 1             THREAD_ID: 17 The newest transaction this worker is aware         SERVICE_STATE: ON ... `60 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | of
Monitoring Worker Threads 5.7.2 mysql> SET GLOBAL slave_parallel_workers= 2; mysql> select * from performance_schema.replication_execute_status_by_workerG *************************** 1. row ***************************             WORKER_ID: 0             THREAD_ID: 16         SERVICE_STATE: ON LAST_SEEN_TRANSACTION: 2b3e03a6-453f-11e3-8668-0021ccb3570d:3     LAST_ERROR_NUMBER: 0    LAST_ERROR_MESSAGE: No error in this worker LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00 *************************** 2. row ***************************             WORKER_ID: 1             THREAD_ID: 17         SERVICE_STATE: ON ... `61 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | thread
 A lot more to explore in MySQL-5.6, keep reading...  MySQL-5.7 is our current development branch. We want your valuable feedback.  Suggest Features, report bugs, contribute patches.  `62   We Want Your Feedback MySQL-5.6 is our latest GA release. Help make MySQL-5.7 even better! Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  Replication Logs  Replication Threads  Summary Introduction to MySQL Replication Multi-threaded Slave (MTS) – Need for Parallelization – Parallelize by database – Parallelize by master concurrency  Keeping track of replication threads – Performance Schema – Replication info. in Performance Schema `63 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
Read More About MySQL Replication `64 Find MySQL Official Documentation at http://dev.mysql.com/doc/refman/5.7/en/replication.html Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
 Read More About MultiThreaded Slaves `65 Parallelization by database - Luis's blog - Andrei's blog  Parallelization by master concurrency - Rohit's blog  Replication Performance Schema - Official MySQL documentation - Shiv's blog Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
Questions! Email: shivji.jha@oracle.com Blog: shivjijha.blogspot.in `66 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |

More Related Content

PDF
Pulsar summit asia 2021: Designing Pulsar for Isolation
Shivji Kumar Jha
 
PDF
Redo log improvements MYSQL 8.0
Mydbops
 
PDF
MySQL Parallel Replication by Booking.com
Jean-François Gagné
 
PDF
How pulsar stores data at Pulsar-na-summit-2021.pptx (1)
Shivji Kumar Jha
 
PDF
MySQL Group Replication
Kenny Gryp
 
PDF
Our answer to Uber
Alexander Korotkov
 
PDF
MariaDB MaxScale
MariaDB plc
 
PDF
Replication Troubleshooting in Classic VS GTID
Mydbops
 
Pulsar summit asia 2021: Designing Pulsar for Isolation
Shivji Kumar Jha
 
Redo log improvements MYSQL 8.0
Mydbops
 
MySQL Parallel Replication by Booking.com
Jean-François Gagné
 
How pulsar stores data at Pulsar-na-summit-2021.pptx (1)
Shivji Kumar Jha
 
MySQL Group Replication
Kenny Gryp
 
Our answer to Uber
Alexander Korotkov
 
MariaDB MaxScale
MariaDB plc
 
Replication Troubleshooting in Classic VS GTID
Mydbops
 

What's hot (20)

PDF
MySQL Ecosystem in 2023 - FOSSASIA'23 - Alkin.pptx.pdf
Alkin Tezuysal
 
PDF
MySQL InnoDB Cluster and NDB Cluster
Mario Beck
 
PDF
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)
Jean-François Gagné
 
PDF
ProxySQL High Avalability and Configuration Management Overview
René Cannaò
 
PDF
How to Manage Scale-Out Environments with MariaDB MaxScale
MariaDB plc
 
PDF
MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...
Jean-François Gagné
 
PDF
Percona Live 2022 - MySQL Architectures
Frederic Descamps
 
PDF
MySQL InnoDB Cluster - Group Replication
Frederic Descamps
 
PDF
MySQL GTID Concepts, Implementation and troubleshooting
Mydbops
 
PDF
Wars of MySQL Cluster ( InnoDB Cluster VS Galera )
Mydbops
 
PDF
Parallel Replication in MySQL and MariaDB
Mydbops
 
PDF
Patroni - HA PostgreSQL made easy
Alexander Kukushkin
 
PDF
MySQL InnoDB Cluster - A complete High Availability solution for MySQL
Olivier DASINI
 
PDF
MariaDB ColumnStore
MariaDB plc
 
PDF
Solving PostgreSQL wicked problems
Alexander Korotkov
 
PDF
Galera cluster for high availability
Mydbops
 
PDF
The Full MySQL and MariaDB Parallel Replication Tutorial
Jean-François Gagné
 
PDF
MySQL Group Replication: Handling Network Glitches - Best Practices
Frederic Descamps
 
PDF
MySQL InnoDB Cluster - New Features in 8.0 Releases - Best Practices
Kenny Gryp
 
PDF
MySQL Database Architectures - 2020-10
Kenny Gryp
 
MySQL Ecosystem in 2023 - FOSSASIA'23 - Alkin.pptx.pdf
Alkin Tezuysal
 
MySQL InnoDB Cluster and NDB Cluster
Mario Beck
 
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)
Jean-François Gagné
 
ProxySQL High Avalability and Configuration Management Overview
René Cannaò
 
How to Manage Scale-Out Environments with MariaDB MaxScale
MariaDB plc
 
MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...
Jean-François Gagné
 
Percona Live 2022 - MySQL Architectures
Frederic Descamps
 
MySQL InnoDB Cluster - Group Replication
Frederic Descamps
 
MySQL GTID Concepts, Implementation and troubleshooting
Mydbops
 
Wars of MySQL Cluster ( InnoDB Cluster VS Galera )
Mydbops
 
Parallel Replication in MySQL and MariaDB
Mydbops
 
Patroni - HA PostgreSQL made easy
Alexander Kukushkin
 
MySQL InnoDB Cluster - A complete High Availability solution for MySQL
Olivier DASINI
 
MariaDB ColumnStore
MariaDB plc
 
Solving PostgreSQL wicked problems
Alexander Korotkov
 
Galera cluster for high availability
Mydbops
 
The Full MySQL and MariaDB Parallel Replication Tutorial
Jean-François Gagné
 
MySQL Group Replication: Handling Network Glitches - Best Practices
Frederic Descamps
 
MySQL InnoDB Cluster - New Features in 8.0 Releases - Best Practices
Kenny Gryp
 
MySQL Database Architectures - 2020-10
Kenny Gryp
 
Ad

Similar to MySQL User Camp: Multi-threaded Slaves (20)

PDF
Drupal Camp Göteborg 2013: Skalbarhet och tillgänglighet med MySQL-replikering
Sven Sandberg
 
PDF
MySQL Developer Day conference: MySQL Replication and Scalability
Shivji Kumar Jha
 
PDF
2012 replication
sqlhjalp
 
PDF
2012 scale replication
sqlhjalp
 
PDF
MySQL Replication Update -- Zendcon 2016
Dave Stokes
 
PDF
MySQL Replication Basics -Ohio Linux Fest 2016
Dave Stokes
 
PDF
MySQL User Camp: GTIDs
Shivji Kumar Jha
 
PDF
MySQL 5.6 Replication Webinar
Mark Swarbrick
 
PPT
Download presentation
webhostingguy
 
PPTX
MySQL Replication Overview -- PHPTek 2016
Dave Stokes
 
PPT
Download presentation
Rachit Gaur
 
PPT
Download presentation531
Indra Pratap
 
PDF
2012 ohiolinuxfest replication
sqlhjalp
 
PDF
MySQL Cluster Asynchronous replication (2014)
Frazer Clement
 
PDF
Multi source replication pdf
Mysql User Camp
 
PDF
Keith Larson Replication
Dave Stokes
 
PDF
Evolution of MySQL Parallel Replication
Mydbops
 
PDF
MySQL Parallel Replication: inventory, use-case and limitations
Jean-François Gagné
 
PDF
MySQL Replication Troubleshooting for Oracle DBAs
Sveta Smirnova
 
PPT
Mysql replication @ gnugroup
Jayant Chutke
 
Drupal Camp Göteborg 2013: Skalbarhet och tillgänglighet med MySQL-replikering
Sven Sandberg
 
MySQL Developer Day conference: MySQL Replication and Scalability
Shivji Kumar Jha
 
2012 replication
sqlhjalp
 
2012 scale replication
sqlhjalp
 
MySQL Replication Update -- Zendcon 2016
Dave Stokes
 
MySQL Replication Basics -Ohio Linux Fest 2016
Dave Stokes
 
MySQL User Camp: GTIDs
Shivji Kumar Jha
 
MySQL 5.6 Replication Webinar
Mark Swarbrick
 
Download presentation
webhostingguy
 
MySQL Replication Overview -- PHPTek 2016
Dave Stokes
 
Download presentation
Rachit Gaur
 
Download presentation531
Indra Pratap
 
2012 ohiolinuxfest replication
sqlhjalp
 
MySQL Cluster Asynchronous replication (2014)
Frazer Clement
 
Multi source replication pdf
Mysql User Camp
 
Keith Larson Replication
Dave Stokes
 
Evolution of MySQL Parallel Replication
Mydbops
 
MySQL Parallel Replication: inventory, use-case and limitations
Jean-François Gagné
 
MySQL Replication Troubleshooting for Oracle DBAs
Sveta Smirnova
 
Mysql replication @ gnugroup
Jayant Chutke
 
Ad

More from Shivji Kumar Jha (17)

PPTX
Batch to near-realtime: inspired by a real production incident
Shivji Kumar Jha
 
PDF
Navigating Transactions: ACID Complexity in Modern Databases
Shivji Kumar Jha
 
PPTX
Druid Summit 2023 : Changing Druid Ingestion from 3 hours to 5 minutes
Shivji Kumar Jha
 
PPTX
osi-oss-dbs.pptx
Shivji Kumar Jha
 
PPTX
pulsar-platformatory-meetup-2.pptx
Shivji Kumar Jha
 
PDF
Pulsar Summit Asia 2022 - Streaming wars and How Apache Pulsar is acing the b...
Shivji Kumar Jha
 
PDF
Pulsar Summit Asia 2022 - Keeping on top of hybrid cloud usage with Pulsar
Shivji Kumar Jha
 
PPTX
Event sourcing Live 2021: Streaming App Changes to Event Store
Shivji Kumar Jha
 
PPTX
Apache Con 2021 Structured Data Streaming
Shivji Kumar Jha
 
PPTX
Apache Con 2021 : Apache Bookkeeper Key Value Store and use cases
Shivji Kumar Jha
 
PDF
Pulsar Summit Asia - Structured Data Stream with Apache Pulsar
Shivji Kumar Jha
 
PDF
Pulsar Summit Asia - Running a secure pulsar cluster
Shivji Kumar Jha
 
PDF
lessons from managing a pulsar cluster
Shivji Kumar Jha
 
PDF
FOSSASIA 2015: MySQL Group Replication
Shivji Kumar Jha
 
PDF
MySQL High Availability with Replication New Features
Shivji Kumar Jha
 
PDF
MySQL User Camp: MySQL Cluster
Shivji Kumar Jha
 
PDF
Open source India - MySQL Labs: Multi-Source Replication
Shivji Kumar Jha
 
Batch to near-realtime: inspired by a real production incident
Shivji Kumar Jha
 
Navigating Transactions: ACID Complexity in Modern Databases
Shivji Kumar Jha
 
Druid Summit 2023 : Changing Druid Ingestion from 3 hours to 5 minutes
Shivji Kumar Jha
 
osi-oss-dbs.pptx
Shivji Kumar Jha
 
pulsar-platformatory-meetup-2.pptx
Shivji Kumar Jha
 
Pulsar Summit Asia 2022 - Streaming wars and How Apache Pulsar is acing the b...
Shivji Kumar Jha
 
Pulsar Summit Asia 2022 - Keeping on top of hybrid cloud usage with Pulsar
Shivji Kumar Jha
 
Event sourcing Live 2021: Streaming App Changes to Event Store
Shivji Kumar Jha
 
Apache Con 2021 Structured Data Streaming
Shivji Kumar Jha
 
Apache Con 2021 : Apache Bookkeeper Key Value Store and use cases
Shivji Kumar Jha
 
Pulsar Summit Asia - Structured Data Stream with Apache Pulsar
Shivji Kumar Jha
 
Pulsar Summit Asia - Running a secure pulsar cluster
Shivji Kumar Jha
 
lessons from managing a pulsar cluster
Shivji Kumar Jha
 
FOSSASIA 2015: MySQL Group Replication
Shivji Kumar Jha
 
MySQL High Availability with Replication New Features
Shivji Kumar Jha
 
MySQL User Camp: MySQL Cluster
Shivji Kumar Jha
 
Open source India - MySQL Labs: Multi-Source Replication
Shivji Kumar Jha
 

Recently uploaded (20)

PDF
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Safe Software
 
PDF
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
SynapseIndia
 
PDF
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
PPTX
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
blackmambaettijean
 
PPTX
20250228 LYD VKU AI Blended-Learning.pptx
DatVoNgoc
 
PPT
Teaching material agriculture food technology
LiaRayya
 
PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
PDF
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
PPTX
Spectroscopy.pptx food analysis technology
nawahassan45
 
PDF
Building Integrated photovoltaic BIPV_UPV.pdf
SandeepSingh286037
 
PDF
Dropbox Q2 2025 Financial Results & Investor Presentation
Dropbox
 
PDF
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
PPTX
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
PDF
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
Julien SIMON
 
PDF
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
PDF
The Rise and Fall of 3GPP – Time for a Sabbatical?
3G4G
 
PDF
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
PDF
Machine learning based COVID-19 study performance prediction
IAESIJAI
 
PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Safe Software
 
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
SynapseIndia
 
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
blackmambaettijean
 
20250228 LYD VKU AI Blended-Learning.pptx
DatVoNgoc
 
Teaching material agriculture food technology
LiaRayya
 
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
Spectroscopy.pptx food analysis technology
nawahassan45
 
Building Integrated photovoltaic BIPV_UPV.pdf
SandeepSingh286037
 
Dropbox Q2 2025 Financial Results & Investor Presentation
Dropbox
 
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
Julien SIMON
 
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
The Rise and Fall of 3GPP – Time for a Sabbatical?
3G4G
 
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
Machine learning based COVID-19 study performance prediction
IAESIJAI
 
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 

MySQL User Camp: Multi-threaded Slaves

  • 1. MySQL Replication and Multi-threaded Slaves Shivji Kumar Jha, Software Developer, MySQL Replication Team
  • 2. Safe Harbour Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. `2 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 3. Agenda Why Replication?  Replication Internals- A Simple Introduction  Why Multi-threaded Slaves (MTS)?  Different policies for Multi-threading Slaves  `3  Keeping track of replication threads Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 4. Replication: Copy Changes Master → Slave  MySQL Master Server – Changes data – Sends changes to slave  MySQL Slave Server – Receives changes from master – Applies received changes to database M `4 S Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 5. Why Replication? – Scalability  Read scale-out M write clients `5 S read clients Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 6. Why Replication? – Scalability  Read scale-out M write clients `6 S More reads? read clients Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 7. Why Replication? – Scalability  Read scale-out M write clients `7 S More reads? More slaves! read clients Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 8. Why Replication? – Scalability  Read scale-out S M S More reads? More slaves! M S S write clients write clients `8 read clients Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | read clients
  • 9. Why Replication? – Redundancy  If master crashes, promote slave to master B A C `9 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 10. Why Replication? – Redundancy  If master crashes, promote slave to master B Crash A C `10 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 11. Why Replication? – Redundancy  If master crashes, promote slave to master B is the new master B A C `11 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 12. Why Replication? – Disaster recovery C A A B C B Image from www.ginkgomaps.com `12 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 13. But how do you copy changes to slave? `13 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 14. But how do you copy changes to slave? Are there LOGS floating around? `14 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 15. Introducing replication LOGS... `15 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 16. Replication Logs  MySQL Master Server M – Changes data (Writes) Binary log – Saves changes in Binary log  MySQL Slave Server – Copies changes to Relay log – Reads Relay log – Applies changes. `16 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | Relay log S
  • 17. All Changes Written to Binary Log Client binary log A `17 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 18. All Changes Written to Binary Log Client create table t (a int); binary log A `18 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 19. All Changes Written to Binary Log Client create table t (a int); binary log create... A Table t `19 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. |
  • 20. All Changes Written to Binary Log Client create table t (a int); insert into t values (1); binary log create... A Table t `20 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 21. All Changes Written to Binary Log Client create table t (a int); insert into t values (1); binary log A create... insert... Table t 1 `21 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 22. Slave Initiates Replication 1. Slave sends request to start replication to master Client relay log binary log B A 2. Master sends stream of replication data to slave `22 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 23. Slave Copies, Applies Same Changes Client create... binary log A `23 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | relay log B
  • 24. Slave Copies, Applies Same Changes Client create... binary log relay log create... A Table t `24 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | B
  • 25. Slave Copies, Applies Same Changes Client create... binary log relay log create... create... A Table t `25 B Table t Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 26. Slave Copies, Applies Same Changes Client create... insert... binary log A relay log create... insert... create... Table t 1 `26 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | B Table t
  • 27. Slave Copies, Applies Same Changes Client create... insert... binary log A relay log create... insert... create... insert... Table t 1 `27 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | B Table t 1
  • 28. OK makes sense ! `28 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 29. But I see clients applying changes at Master, who applies at Slave? `29 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 30. But I see clients applying changes at Master, who applies at Slave? Do you have some internal replication threads? `30 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 31. Introducing replication THREADS... `31 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 32. Replication Threads (4.0-5.5) Client create... insert... relay log binary log A create... insert... create... network insert... SQL THREAD IO THREAD Writes into relay log at slave `32 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | B Reads from relay log. Applies transactions at slave
  • 33. Replication Threads (4.0-5.5) Create... insert... Client Client insert... More clients on master... Client relay log binary log A `33 create... insert... Insert... create... network insert... SQL THREAD IO THREAD Insert... Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | B
  • 34. Replication Threads (4.0-5.5) Create... insert... Client Client insert... More clients on master... Client Slave starts to lag behind master relay log binary log A `34 create... insert... Insert... create... network insert... SQL THREAD IO THREAD Insert... Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | B
  • 35. Replication Threads (4.0-5.5) Create... insert... Client Client insert... More clients on master... Client Slave starts to lag behind master relay log binary log A create... insert... Insert... create... network insert... SQL THREAD IO THREAD Insert... Table t 1 2 3 `35 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | B Table t 1 2
  • 36. Lets Parallelize slave as well... `36 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 37. Lets Parallelize slave as well... Introducing parallelization by Database `37 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 38. Replication Threads (5.6+) create... Client insert... insert... Reads & assigns to worker(s) Client Client w create.. O insert... R K COORDINATOR E insert.. THREAD R S binary log A `38 create(db1) insert(db2) insert(db3) network IO THREAD Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | db1 B db2 db3 create(db1) insert(db2) insert(db3) Worker threads apply concurrently on diff dbs relay log
  • 39. Replication Threads (5.6+) Setting up Parallelization by database mysql> STOP SLAVE; mysql> SET GLOBAL slave_parallel_workers=1; mysql> START SLAVE; `39 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 40. Replication Threads (5.6+) More databases? More workers! mysql> STOP SLAVE; mysql> SET GLOBAL slave_parallel_workers=2; mysql> START SLAVE; `40 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 41. What if I have a a loaded Database? Image credits: http://www.easetechnology.co.uk/ask-an-expert/ `41 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 42. Lets Parallelize slave somewhat similar to master `42 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 43. Lets Parallelize slave somewhat similar to master 5.7.2 Introducing parallelization by Master concurrency `43 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 44. Replication Threads (4.0-5.5) Create... insert... Client Client insert... More clients on master... Client Slave starts to lag behind master relay log binary log A Table t 1 2 3 `44 create... insert... Insert... create... network insert... SQL THREAD IO THREAD Insert... REVISITING THE OLD SLIDE !!! Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | B Table t 1 2
  • 45. Replication Threads (5.7+) create... Client insert... insert... Reads & assigns to worker(s) Client Client COORDINATOR THREAD binary log A create... insert... Insert... network IO THREAD capture parallelization info on master `45 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | create... insert... Insert... relay log w O R K E R S B db1 db2 Worker threads apply concurrently
  • 46. Replication Threads (5.7+) create... Client insert... insert... Reads & assigns to worker(s) Client Client COORDINATOR THREAD binary log A create... insert... Insert... network IO THREAD Save parallelization info in binary log `46 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | create... insert... Insert... relay log w O R K E R S B db1 db2 Worker threads apply concurrently
  • 47. Replication Threads (5.7+) create... Client insert... insert... Reads & assigns to worker(s) Client Client COORDINATOR THREAD binary log A create... insert... Insert... network IO THREAD Send parallelization info to slave `47 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | create... insert... Insert... relay log w O R K E R S B db1 db2 Worker threads apply concurrently
  • 48. Replication Threads (5.7+) create... Client insert... insert... Reads & assigns to worker(s) Client Client COORDINATOR THREAD binary log A create... insert... Insert... network IO THREAD Use parallelization info to assign `48 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | create... insert... Insert... relay log w O R K E R S B db1 db2 Worker threads apply concurrently
  • 49. Replication Threads (5.7+)   `49 Leverage parallelization information obtained from the execution on the master: — Transactions that reach the prepare phase on the same data snapshot are non-contending; — Write to the binary log on which snapshot id each transaction prepared;  This identifies the commit parent of each transaction. — The commit parent is stepped every time transactions commit. Meanwhile, at the slave: — Transactions with the same commit parent can be executed in parallel; — Commit sequence at the slave may not be the same as that on the master, but it is still a correct execution history. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. |
  • 50. Higher Slave Throughput: Intra-Schema MTS Concurrent Execution History on the Master T1 T2 T3 Time Execution Commit Prepare Commit `50 commit_parent++ Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Prepare
  • 51. Higher Slave Throughput: Intra-Schema MTS Concurrent Execution History on the Master Parallel on the Slave. T1 T2 T3 Time Execution Commit Prepare Commit `51 commit_parent++ Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Prepare Not parallel on the slave.
  • 52. Replication Threads (5.7+) Setting up Parallelization by master concurrency mysql> STOP SLAVE; mysql> SET GLOBAL slave_parallel_type=[ 'logical_clock' | 'database' ]; mysql> SET GLOBAL slave_parallel_workers=3; mysql> START SLAVE; database=>parallelize by database logical_clock=>parallelize by master concurrency `52 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 53. Too many threads here and there! How do I monitor? Knew that was coming! Check out replication Performance_Schema `53 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 54. Performance Schema (5.5+)  Inspect internal execution of the server at runtime  Exposed within performance_schema database  Records various run time statistics via in-built instrumentation points  Tracks latency for various events: File I/O, Mutexes, Read/Write Locks, Table I/O, Table Locks, Stages, Statements, Idle etc..  `54 Its a long growing list Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 55. Replication Performance Schema Tables (5.7+) replication 5.7.2 execution connection configuration status configuration Coordinator thread / SQL thread `55 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | status Worker thread(s)
  • 56. The REPLICATION P_S Tables  We have six performance schema tables for replication: 5.7.2 (MySQL-5.7.2):  replication_connection_configuration  replication_connection_status (IO thread status)  replication_execute_configuration  replication_execute_status  replication_execute_status_by_coordinator  replication_execute_status_by_worker Tables for monitoring MTS `56 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 57. Monitoring The Coordinator Thread 5.7.2 mysql> select * from performance_schema.replication_execute_status_by_coordinatorG *************************** 1. row ***************************            THREAD_ID: 13        SERVICE_STATE: ON    LAST_ERROR_NUMBER: 0   LAST_ERROR_MESSAGE: LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00 `57 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | No error in thread
  • 58. Monitoring The Coordinator Thread 5.7.2 mysql> select * from performance_schema.replication_execute_status_by_coordinatorG *************************** 1. row ***************************            THREAD_ID: 13        SERVICE_STATE: ON Oops! There was an error    LAST_ERROR_NUMBER: 1146   LAST_ERROR_MESSAGE:...'Table 'test.t' doesn't exist'... LAST_ERROR_TIMESTAMP: 2013-11-04 13:37:23 `58 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 59. Monitoring Worker Threads 5.7.2 mysql> SET GLOBAL slave_parallel_workers= 2; mysql> select * from performance_schema.replication_execute_status_by_workerG *************************** 1. row ***************************             WORKER_ID: 0             THREAD_ID: 16         SERVICE_STATE: ON LAST_SEEN_TRANSACTION: 2b3e03a6-453f-11e3-8668-0021ccb3570d:3     LAST_ERROR_NUMBER: 0    LAST_ERROR_MESSAGE: LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00 *************************** 2. row ***************************             WORKER_ID: 1             THREAD_ID: 17 one row per worker thread         SERVICE_STATE: ON ... `59 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 60. Monitoring Worker Threads 5.7.2 mysql> SET GLOBAL slave_parallel_workers= 2; mysql> select * from performance_schema.replication_execute_status_by_workerG *************************** 1. row ***************************             WORKER_ID: 0             THREAD_ID: 16         SERVICE_STATE: ON LAST_SEEN_TRANSACTION: 2b3e03a6-453f-11e3-8668-0021ccb3570d:3     LAST_ERROR_NUMBER: 0    LAST_ERROR_MESSAGE: LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00 *************************** 2. row ***************************             WORKER_ID: 1             THREAD_ID: 17 The newest transaction this worker is aware         SERVICE_STATE: ON ... `60 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | of
  • 61. Monitoring Worker Threads 5.7.2 mysql> SET GLOBAL slave_parallel_workers= 2; mysql> select * from performance_schema.replication_execute_status_by_workerG *************************** 1. row ***************************             WORKER_ID: 0             THREAD_ID: 16         SERVICE_STATE: ON LAST_SEEN_TRANSACTION: 2b3e03a6-453f-11e3-8668-0021ccb3570d:3     LAST_ERROR_NUMBER: 0    LAST_ERROR_MESSAGE: No error in this worker LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00 *************************** 2. row ***************************             WORKER_ID: 1             THREAD_ID: 17         SERVICE_STATE: ON ... `61 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 | thread
  • 62.  A lot more to explore in MySQL-5.6, keep reading...  MySQL-5.7 is our current development branch. We want your valuable feedback.  Suggest Features, report bugs, contribute patches.  `62   We Want Your Feedback MySQL-5.6 is our latest GA release. Help make MySQL-5.7 even better! Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 63.   Replication Logs  Replication Threads  Summary Introduction to MySQL Replication Multi-threaded Slave (MTS) – Need for Parallelization – Parallelize by database – Parallelize by master concurrency  Keeping track of replication threads – Performance Schema – Replication info. in Performance Schema `63 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 64. Read More About MySQL Replication `64 Find MySQL Official Documentation at http://dev.mysql.com/doc/refman/5.7/en/replication.html Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 65.  Read More About MultiThreaded Slaves `65 Parallelization by database - Luis's blog - Andrei's blog  Parallelization by master concurrency - Rohit's blog  Replication Performance Schema - Official MySQL documentation - Shiv's blog Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |
  • 66. Questions! Email: shivji.jha@oracle.com Blog: shivjijha.blogspot.in `66 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp | Bangalore, India | 8 November 2013 |