Introduction to MySQL InnoDB Cluster

MySQL InnoDB Cluster
미리 보기
MySQL Power Group
2017.01.14

유승민 (Seungmin yu)
Loen Entertainment (2013 ~ )
MelOn IT Infra Team
Smyoo0316@gmail.com
발표자 소개

MySQL InnoDB Cluster
New HA solution (available on labs)
Read-out Scalability
Combination of three Technologies
MySQL Group Replication
MySQL Router
MySQL Shell

MySQL Group Replication
 The plugin that brings multi-master update everywhere (atomic broadcasting)
MySQL Router
 Lightweight middle ware between application and MySQL Server
 Read-write connection redirect (first-available)
 Read-only connection distribution (round-robin)
 Connection fail-over (redirect the failed connection to available MySQL Server)
MySQL Shell
 New command interface to MySQL Server.
 Support three type languages : javascript, python, sql
 Support admin command for InnoDB Cluster
 Must install python 2.7

MySQL (3306)
PRIMARY : R/W
[MySQL Group Replication]
Application
MySQL (3306)
SECONDARY : R/O
MySQL (3306)
SECONDARY : R/O
[MySQL Shell]
shell> mysqlsh
mysqlsh-js> c root@localhost:3306
mysqlsh-js> c = dba.createCluster(“Test”)
mysqlsh-js> c.addInstance(‘root@server2:3306)
mysqlsh-js> c.addInstance(‘root@server3:3306)
mysqlsh-js> c.describe()
mysqlsh-js> c.status()
[MySQL Router]
R/W : 6446
R/O : 6447
READ-WRITE
READ-ONLY

MySQL (3306)
PRIMARY : R/W
[MySQL Group Replication]
Application
MySQL (3306)
PRIMARY : R/W
MySQL (3306)
SECONDARY : R/O
[MySQL Shell]
mysql-js> c.status();
{
"clusterName": "TestCluster",
"defaultReplicaSet": {
"status": "Cluster is NOT tolerant to any failures.",
"topology": {
“server2:3306": {
"address": “server2:3306",
"status": "ONLINE",
"role": "HA",
"mode": "R/W",
"leaves": {
“server3:3306": {
"status": "ONLINE",
"role": "HA",
"mode": "R/O",
"leaves": {}
},
“server1:3306": {
"status": "OFFLINE",
"role": "HA",
"mode": "R/O",
"leaves": {}
}
}
}
}
}
}
[MySQL Router]
R/W : 6446
R/O : 6447
READ-WRITE
READ-ONLY

Introduction to MySQL InnoDB Cluster

PLUGIN
UPDATE EVERYWHERE
AUTO DISTRIBUTED RECOVERY
CLOUD-FRIENDLY
5.1.17 GA Released (2016.12.12)
MySQL MySQL MySQL MySQL MySQL
Group Replication Cluster

<이미지출처 : http://mysqlhighavailability.com/mysql-group-replication-hello-world/>
Atomic Broadcasting.
First writer wins rules.

SLAVESLAVE
MASTER
SERVER3SERVER2
SERVER1
Replication Group Replication

<이미지출처 : http://mysqlhighavailability.com/order-from-chaos-member-coordination-in-group-replication/>
기존 replication framework 에서 사용하는 기능
Binary log caching infrastructure
GTID framework
Row-based replication
완전 새로운 기능 (Xcom)
Messaging
Group membership
Auto distributed recovery mechanism
Atomic broadcasting
Transaction coordination
- Check whether the transaction should commit or not
- Propagate the changes
- Conflict resolution follows the first writer wins rule

MYSQL> SHOW CREATE TABLE T1G;
*************************** 1. row ***************************
Table: T1
Create Table: CREATE TABLE `t1` (
`col1` int(11) NOT NULL,
`col2` varchar(100) DEFAULT NULL,
PRIMARY KEY (`col1`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8
1 row in set (0.00 sec)
ERROR:
No query specified
MYSQL> INSERT INTO T1 VALUES (100, 'AAAA');
ERROR 3098 (HY000): The table does not comply with the requirements by an external plugin.
Only Available on InnoDB storage engine (That’s why they named InnoDB Cluster)

Single PRIMARY mode
Beginning with 5.7.15
group_replication_single_primary_mode=TRUE|FALSE
Which enables/disables the single primary mode
group_replication_enforce_update_everywhere_checks=FALSE|TRUE
Which enables/disables strict consistency checks for multi-master update everywhere
PRIMARY 외 다른 인스턴스들은 모두 SUPER_READ_ONLY 상태가 됨.

Please specify an administrative MASTER key for the cluster 'test':
Creating InnoDB cluster 'test' on 'root@localhost:3310'...
Traceback (most recent call last):
File "/usr/lib64/python2.6/runpy.py", line 122, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib64/python2.6/runpy.py", line 34, in _run_code
exec code in run_globals
File "/home/mysql/programs/mysql-innodb-cluster-labs201609-linux-glibc2.5-x86_64/mysql-shell-1.0.5-labs-linux-glibc2.12-x86-64bit/bin/mysqlprovision/__main__.py", line 32,
in <module>
File "/home/mysql/programs/mysql-innodb-cluster-labs201609-linux-glibc2.5-x86_64/mysql-shell-1.0.5-labs-linux-glibc2.12-x86-
64bit/bin/mysqlprovision/mysql_gadgets/common/options.py", line 32, in <module>
File "/home/mysql/programs/mysql-innodb-cluster-labs201609-linux-glibc2.5-x86_64/mysql-shell-1.0.5-labs-linux-glibc2.12-x86-
64bit/bin/mysqlprovision/mysql_gadgets/common/connection_parser.py", line 254
if hostportsock[0] in {'"', "'"}:
^
SyntaxError: invalid syntax
ArgumentError: Dba.createCluster: Dba.createCluster: Error while executing mysqlprovision (return 1) at (shell):1:14
in cluster = dba.createCluster('test');
^
mysql-js> cluster = dba.createCluster('test',);
SyntaxError: Unexpected token ) at (shell):1:35
in cluster = dba.createCluster('test',);
Python 2.7로 업그레이드 먼저 해야 함.
JS문법을 써도 python버전 에러가 남.
내부 구현체는 모두 python 인 듯.
파이썬 2.7 업그레이드 방법은 “여기”

ERROR: Error executing the 'start-replicaset' command: The operation could not continue due to the following requirements not being met:
Some active options on server 'localhost@3310' are incompatible with Group Replication.
Please restart the server 'localhost@3310' with the updated options file and try again.
Option name Required Value Current Value Result
------------------------------- --------------- --------------- -----
binlog_checksum NONE CRC32 FAIL
binlog_format ROW MIXED FAIL
enforce_gtid_consistency ON OFF FAIL
gtid_mode ON OFF FAIL
log_slave_updates ON 0 FAIL
master_info_repository TABLE FILE FAIL
relay_log_info_repository TABLE FILE FAIL
transaction_write_set_extraction XXHASH64 OFF FAIL
ArgumentError: Dba.createCluster: Dba.createCluster: The operation could not continue due to the following requirements not being met:
at (shell):1:14
in cluster = dba.createCluster('test');
^
필수 Parameter 가 맞지 않으면 유효성 검사에서 에러남. 아래와 같이 my.cnf 에 추가하고 DB 재시작
########################################
# InnoDB Cluster requirements
########################################
binlog_checksum = NONE
binlog_format = ROW
enforce_gtid_consistency = ON
gtid_mode = ON
log_slave_updates = ON
master_info_repository = TABLE
relay_log_info_repository = TABLE
transaction_write_set_extraction = XXHASH64

shell> mysqlsh // mysql shell 실행
mysql-js> c root@192.168.237.81:3310 // 클러스터를 생성할 DB 접속
mysql-js> c = dba.createCluster('TestCluster'); // Group Replication whitelist 관련 “ERROR 발생”. 하단 에러 정보 참고 (리모트에서만 발생)
mysql-js> sql // SQL 모드로 변경
mysql-sql> show global variables like '%whitelist%';
mysql-sql> set global group_replication_ip_whitelist = ‘192.168.237.81,192.168.237.82,192.168.237.83'; // whitelist 추가
mysql-sql> js // Java Script 모드로 변경
mysql-js> c = dba.createCluster('TestCluster'); // 다시 클러스터 생성. 이때 입력하는 “클러스터패스워드”를 잘 기억해야 함.
mysql-js> c.describe(); // 클러스터 멤버 정보 확인
mysql-js> c.status(); // 클러스터 인스턴스 상태 확인
< ERROR LOG >
2016-11-16T06:36:36.582375Z 0 [Warning] Plugin group_replication reported: '[GCS] Connection attempt from IP address 192.168.237.81 refused. Address is not in
the IP whitelist.'
2016-11-16T06:36:36.582528Z 0 [ERROR] Plugin group_replication reported: '[GCS] Error connecting to the local group communication engine instance.'
2016-11-16T06:36:36.582557Z 0 [Note] Plugin group_replication reported: 'state 4127 action xa_exit'
2016-11-16T06:36:36.582682Z 0 [Note] Plugin group_replication reported: 'Exiting xcom thread'
2016-11-16T06:36:37.593237Z 0 [ERROR] Plugin group_replication reported: '[GCS] The member was unable to join the group. Local port: 13310'

MySQL> install plugin group_replication soname 'group_replication.so';
MySQL> set global group_replication_ip_whitelist = '192.168.237.81,192.168.237.82,192.168.237.83'; // whitelist 미리 추가
MySQL> set global group_replication_allow_local_disjoint_gtids_join = TRUE; // 하단 설명 참조.
MySQL> exit
$mysqlsh
mysql-js> c c root@192.168.237.81:3310 // PRIMARY 인스턴스 접속
mysql-js> dba.validateInstance('root@192.168.237.83:3310'); // 필수 parameter 점검
mysql-js> c.addInstance('root@192.168.237.83:3310'); // 인스턴스 추가
mysql-js> c.status(); // RECOERVING 상태로 대기함. root 유저 생성한 sql문 충돌 때문 (리모트에서만 발생)
stop group_replication;
drop user 'root'@'192.168.237.%'; // error log 에서 충돌 원인 확인하여 조치
start group_replication;
mysql-sql> js // Java Script 모드로 변경
c.status(); // ONLINE 상태로 보이면 정상
group_replication_allow_local_disjoint_gtids_join
인스턴스 추가 할 때 기존 클러스터 멤버 보다 최신 데이터가 감지 될 시 복구나 런타임 에러를 방지하기 위해 REJECT함.
무시해도 된다고 확신 할 경우 TRUE로 하고 강제로 추가.

mysql-js> c.status();
{
"clusterName": "test",
"defaultReplicaSet": {
"status": "Cluster tolerant to up to ONE failure.",
"topology": {
"192.168.237.81:3310": {
"address": "192.168.237.81:3310",
"status": "ONLINE",
"role": "HA",
"mode": "R/W",
"leaves": {
"192.168.237.83:3310": {
"address": "192.168.237.83:3310",
"status": "ONLINE",
"role": "HA",
"mode": "R/O",
"leaves": {}
},
"192.168.237.82:3310": {
"address": "192.168.237.82:3310",
"status": "ONLINE",
"role": "HA",
"mode": "R/O",
"leaves": {}
}
}
}
}
}
}
두 번째와 동일한 방식으로 추가하고 status()에서 모두
ONLINE 상태이면 정상.
# mysqlsh 세션 종류 후 클러스터 인스턴스 컨트롤 하는 법
mysql-sh> c = dba.getCluster(); //클러스터패스워드 입력
mysql-sh> c.help();

show grants for 'mysql_innodb_cluster_admin'@'192.168.237.81';
+-----------------------------------------------------------------------------------------------------------------+
| Grants for mysql_innodb_cluster_admin@192.168.237.81 |
+-----------------------------------------------------------------------------------------------------------------+
| GRANT USAGE ON *.* TO 'mysql_innodb_cluster_admin'@'192.168.237.81' |
| GRANT ALL PRIVILEGES ON `mysql_innodb_cluster_metadata`.* TO 'mysql_innodb_cluster_admin'@'192.168.237.81' |
| GRANT SELECT ON `performance_schema`.`replication_group_members` TO 'mysql_innodb_cluster_admin'@'192.168.237.81' |
+-----------------------------------------------------------------------------------------------------------------+
3 rows in set (0.00 sec)
// MySQL Router 에서 접속할 유저 생성
GRANT USAGE ON *.* TO 'mysql_innodb_cluster_admin'@'192.168.237.105';
GRANT ALL PRIVILEGES ON `mysql_innodb_cluster_metadata`.* TO 'mysql_innodb_cluster_admin'@'192.168.237.105';
GRANT SELECT ON `performance_schema`.`replication_group_members` TO 'mysql_innodb_cluster_admin'@'192.168.237.105';
set password for 'mysql_innodb_cluster_admin'@'192.168.237.105' = '1'; // 클러스터패스워드와 동일해야 함.
// Failover 테스트를 위한 일반 DB유저 생성
create user 'smyoo'@'192.168.237.105' identified by 'smyoo';
grant select, insert, update, delete on *.* to 'smyoo'@'192.168.237.105';

$ ./mysqlrouter --bootstrap 192.168.237.81:3310 // DB로부터 클러스터 메타 정보를 읽어서 mysqlrouter.conf 파일을 생성
Please enter the administrative MASTER key for the MySQL InnoDB cluster: // 클러스터 패스워드 입력
MySQL Router has now been configured for the InnoDB cluster 'test'.
The following connection information can be used to connect to the cluster.
Classic MySQL protocol connections to cluster 'test':
- Read/Write Connections: localhost:6446
- Read/Only Connections: localhost:6447
$
$ nohup ./mysqlrouter & // 백그라운드로 기동
[1] 43986
$ nohup: ignoring input and appending output to `nohup.out'
$
$ ps -ef | grep mysqlrouter
root 43986 8680 0 15:04 pts/0 00:00:00 ./mysqlrouter
root 44050 8680 0 15:04 pts/0 00:00:00 grep mysqlrouter
$

// 자동으로 생성된 mysqlrouter.conf 파일 내용
[DEFAULT]
plugin_folder=/root/programs/mysql-innodb-cluster-labs201609-linux-glibc2.5-x86_64/mysql-router-2.1.0-labs-linux-glibc2.12-x86-
64bit/lib/mysqlrouter
# logging_folder=/root/programs/mysql-innodb-cluster-labs201609-linux-glibc2.5-x86_64/mysql-router-2.1.0-labs-linux-glibc2.12-x86-
64bit/bin/../
[logger]
level = INFO
[metadata_cache]
bootstrap_server_addresses=mysql://192.168.237:3310,mysql://192.168.237.82:3310,mysql://192.168.237.83:3310
user=mysql_innodb_cluster_reader
password=,wY,JbZ9Saxf%}(S
metadata_cluster=test
ttl=300
metadata_replicaset=default
[routing:default_rw]
bind_port=6446
destinations=metadata-cache:///default?role=PRIMARY
mode=read-write
[routing:default_ro]
bind_port=6447
destinations=metadata-cache:///default?role=SECONDARY
mode=read-only

$ mysqlsh --uri smyoo@localhost:6446 // MySQL Router 로 PRIMARY 인스턴스로 접속
Switching to SQL mode... Commands end with ;
mysql-sql>
mysql-sql> select @@hostname; // PRIMARY 인스턴스 접속 확인
+------------+
| @@hostname |
+------------+
| mytest1 |
+------------+
mysql-sql>
mysql-sql> select @@hostname;
ERROR: 2013 (HY000): Lost connection to MySQL server during query // PRIMARY 인스턴스 DOWN 감지 (약 1초 정도 딜레이)
The global session got disconnected. // 접속 실패한 세션은 그냥 사라짐. 자동으로 재 접속 하지 않는다.
Attempting to reconnect to 'smyoo@localhost:6446'...
The global session was successfully reconnected.
mysql-sql>
mysql-sql> select @@hostname; // Failover 된 것 확인
+------------+
| @@hostname |
+------------+
| mytest2 |
+------------+

MYSQL> SELECT * FROM PERFORMANCE_SCHEMA.REPLICATION_GROUP_MEMBERS ORDER BY MEMBER_HOST;
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| group_replication_applier | dae27541-b2e5-11e6-a22c-ac162d76e1c4 | mytest1 | 3310 | ONLINE |
| group_replication_applier | deeb5d42-b2e5-11e6-9ef5-002655e4a549 | mytest2 | 3310 | ONLINE |
| group_replication_applier | e2449c17-b2e5-11e6-b72a-002655e4a51d | mytest3 | 3310 | ONLINE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
3 rows in set (0.00 sec)
MYSQL> SELECT * FROM PERFORMANCE_SCHEMA.REPLICATION_GROUP_MEMBER_STATSG;
*************************** 1. row ***************************
CHANNEL_NAME: group_replication_applier
VIEW_ID: 14800615308848506:7
MEMBER_ID: e2449c17-b2e5-11e6-b72a-002655e4a51d
COUNT_TRANSACTIONS_IN_QUEUE: 0
COUNT_TRANSACTIONS_CHECKED: 1217439
COUNT_CONFLICTS_DETECTED: 0
COUNT_TRANSACTIONS_ROWS_VALIDATING: 0
TRANSACTIONS_COMMITTED_ALL_MEMBERS: dcd7ae4b-b2e6-11e6-b4a5-002655e4a51d:1-1217444,
e2449c17-b2e5-11e6-b72a-002655e4a51d:1-22
LAST_CONFLICT_FREE_TRANSACTION: dcd7ae4b-b2e6-11e6-b4a5-002655e4a51d:1217444
ERROR:
No query specified

1번 노드가 빠진 상태에서 다시 추가하기 전 큰 테
이블(1천만건) 미리 생성해 두고 sysbench트래픽이
유입되는 상태에서 노드 추가.
노드 추가 시 발생하는 자동 분산 복구 시 “아직은”
제대로 동작 안함.
DML 트래픽을 처리 못하고 트랜잭션 큐 지연 발생.
RECOVERING 상태에서 더 이상 진행이 안됨.

 요약
완전히 다른 동기화 메커니즘
구축 용이, 편리한 운영, 유연한 확장
 현재상황
기본적인 기능은 무난하게 동작함. (동기화성능, Failover)
Remote 구성이 매끄럽지 않음.
mysqlsh은 조금 익숙해지면 편리함.
온라인 자동 복구는 제대로 동작 안 함.
 제약사항
InnoDB만 된다.

Introduction to MySQL InnoDB Cluster

More Related Content

What's hot (20)

Similar to Introduction to MySQL InnoDB Cluster (20)

More from I Goo Lee (20)

Recently uploaded (20)

Introduction to MySQL InnoDB Cluster