MySQL 5.7 innodb_enhance_partii_20160527

MySQL5.7
InnoDBenhancements – Part II
LEE SAE WOONG
2016.05.27

4
MySQL 5.7: InnoDB enhancements - Part II
InnoDB enhancements
• Part I : InnoDB Performance
• Part I : InnoDB Buffer Pool Flushing
• Part I : InnoDB internal Transaction General
• Part I : InnoDB Improved adaptive flushing
• Part II : InnoDB Online DDL
• Part II : Tablespace management
• Part II : InnoDB Bulk Load for Create Index
• Part II : InnoDB Temporary Tables
• Part II : InnoDB Full-Text CJK Support
• Part II : Support Syslog on Linux / Unix OS
• Part II : Performance_schema
• Part II : Useful tips

5InnoDB enhancements
root@localhost:mglee 12:32:24>show global variables like 'innodb_buffer_pool_size';
+-------------------------+-------------+
| Variable_name | Value |
+-------------------------+-------------+
| innodb_buffer_pool_size | 21474836480 |
+-------------------------+-------------+
1 row in set (0.00 sec)
root@localhost:mglee 12:34:31>set global innodb_buffer_pool_size = 5368709120;
Query OK, 0 rows affected (0.01 sec)
root@localhost:mglee 12:35:12>show global variables like 'innodb_buffer_pool_size';
+-------------------------+-------------+
+-------------------------+-------------+
+-------------------------+-------------+
Buffer Pool
• Resize the InnoDB Buffer Pool online 20G to 5G
Syntax > set global innodb_buffer_pool_size = 5368709120;

Buffer Pool cont.
• Resize the InnoDB Buffer Pool online 20G to 5G

Buffer Pool cont.
#Dummy DATA JOB
select * from innodb_buffer_stats_by_schema;
+---------------+------------+------------+--------+--------------+-----------+-------------+
| object_schema | allocated | data | pages | pages_hashed | pages_old | rows_cached |
+---------------+------------+------------+--------+--------------+-----------+-------------+
| sysbench1 | 11.04 GiB | 9.89 GiB | 723808 | 723808 | 723808 | 48361888 |
| sysbench | 642.19 MiB | 595.11 MiB | 41100 | 41100 | 41100 | 3001433 |
| InnoDB System | 117.02 MiB | 107.61 MiB | 7489 | 7489 | 7489 | 96132 |
| mysql | 48.00 KiB | 13.34 KiB | 3 | 3 | 3 | 159 |
+---------------+------------+------------+--------+--------------+-----------+-------------+
4 rows in set (22.41 sec)
sysbench --test='/usr/share/doc/sysbench/tests/db/oltp.lua' --oltp-table-size=3000000 --num-threads=16
--max-requests=100000 --max-time=0 --mysql-host=localhost --mysql-user=root --mysql-password=test
--mysql-db=sysbench1 --mysql-table-engine=innodb --mysql-create-options=
--mysql-socket=/mysql/MyHome/tmp/mysql.sock --mysql-port=20306 --oltp-test-mode=complex
--oltp-read-only=off --db-driver=mysql
--db-ps-mode=disable
run
…………
…………
…………
Threads fairness:
events (avg/stddev): 6259.4375/38.17
execution time (avg/stddev): 45.7831/0.00
• Resize the InnoDB Buffer Pool online on LOAD using sysbench
<- no load time

Buffer Pool cont.
+---------------+------------+------------+--------+--------------+-----------+-------------+
+---------------+------------+------------+--------+--------------+-----------+-------------+
| sysbench1 | 11.04 GiB | 9.89 GiB | 723823 | 723823 | 723823 | 48379076 |
| sysbench | 642.19 MiB | 595.11 MiB | 41100 | 41100 | 41100 | 3001433 |
| mysql | 48.00 KiB | 13.34 KiB | 3 | 3 | 3 | 159 |
+---------------+------------+------------+--------+--------------+-----------+-------------+
sysbench --test='/usr/share/doc/sysbench/tests/db/oltp.lua' --oltp-table-size=3000000 --num-threads=16
--max-requests=100000 --max-time=0 --mysql-host=localhost --mysql-user=root --mysql-password=test
--mysql-db=sysbench1 --mysql-table-engine=innodb --mysql-create-options=
--mysql-socket=/mysql/MyHome/tmp/mysql.sock --mysql-port=20306 --oltp-test-mode=complex
--oltp-read-only=off --db-driver=mysql --db-ps-mode=disable
run
set global innodb_buffer_pool_size = 5368709120;
…………
…………
…………
Threads fairness:
events (avg/stddev): 6258.7500/44.18
execution time (avg/stddev): 58.2635/0.00
<- load time

Buffer Pool cont.
+---------------+------------+------------+--------+--------------+-----------+-------------+
+---------------+------------+------------+--------+--------------+-----------+-------------+
| sysbench1 | 2.69 GiB | 2.30 GiB | 176323 | 176323 | 176323 | 16874586 |
| mysql | 32.00 KiB | 13.34 KiB | 2 | 2 | 2 | 159 |
+---------------+------------+------------+--------+--------------+-----------+-------------+
root@localhost:sys 15:14:48>show global variables like 'innodb_buffer_pool_size';
+-------------------------+------------+
+-------------------------+------------+
+-------------------------+------------+
Sysbench : removed
Sysbench1 : reduced

Buffer Pool cont.

• General Tablespaces
– General TABLESPACE support / ORACLE DB also support?
– Store multiple tables in user defined shared tablespaces
- SQL syntax for explicit tablespace management
- Replaces legacy –innodb-file-per-table usage
CREATE TABLESPACE Logs ADD DATAFILE 'log01.ibd';
CREATE TABLE http_req(c1 varchar) TABLESPACE=Logs ;
ALTER TABLE some_table TABLESPACE=Logs;
DROP TABLESPACE Logs; - must be empty
Tablespace management

Tablespace management cont.
CREATE TABLESPACE Logs ADD DATAFILE 'log01.ibd';
[root@LCSYSTS1509:/mysql/MyHome/data]# ll -h
total 26G
drwxr-xr-x 10 mysql dba 4.0K May 16 15:37 .
drwxr-xr-x 12 7161 wheel 4.0K Mar 9 18:53 ..
-rw-r----- 1 mysql dba 56 Jan 12 11:47 auto.cnf
-rw-r----- 1 mysql dba 1.1K Mar 9 18:50 ib_buffer_pool
-rw-r----- 1 mysql dba 512M May 16 15:39 ibdata1
-rw-r----- 1 mysql dba 8.0G May 16 15:39 ib_logfile0
-rw-r----- 1 mysql dba 8.0G May 16 15:39 ib_logfile1
-rw-r----- 1 mysql dba 9.4G May 16 15:40 ibtmp1
-rw-r----- 1 mysql dba 96K May 16 15:39 log01.ibd
-rw-r----- 1 mysql dba 48 Mar 9 18:53 master.info
drwxr-x--- 2 mysql dba 4.0K Jan 12 17:15 mysql
-rw-rw---- 1 root root 6 Mar 9 18:53 mysqld_safe.pid
drwxr-x--- 2 mysql dba 4.0K Jan 12 11:47 performance_schema
………
………
root@localhost:rex 15:48:08>insert into http_req values('federer');
Query OK, 1 row affected (0.00 sec)
root@localhost:rex 15:48:21>select * from http_req;
+---------+
| c1 |
+---------+
| federer |
+---------+

CREATE TABLE rex.http_req(c1 varchar(100)) TABLESPACE=Logs ;
show create table rex.http_req;
+----------+-----------------------------------------------------------------------------------------------------------
| Table | Create Table |
+----------+-----------------------------------------------------------------------------------------------------------
| http_req | CREATE TABLE `http_req` (
`c1` varchar(100) DEFAULT NULL
) /*!50100 TABLESPACE `Logs` */ ENGINE=InnoDB DEFAULT CHARSET=utf8 |
+----------+-----------------------------------------------------------------------------------------------------------
show tables;
+---------------+
| Tables_in_rex |
+---------------+
| employees |
| http_req |
+---------------+
show create table employees;
+-----------+-------------------------------------------------------------------------------+
| Table | Create Table |
+-----------+-------------------------------------------------------------------------------+
| employees | CREATE TABLE `employees` (
`data` text
) ENGINE=InnoDB DEFAULT CHARSET=utf8 |
+-----------+-------------------------------------------------------------------------------+

select * from employees;
+---------------------------+
| data |
+---------------------------+
| {"id": 1, "name": "Jane"} |
| {"id": 2, "name": "Joe"} |
+---------------------------+
root@localhost:rex 15:40:45>ALTER TABLE employees TABLESPACE=Logs;
Records: 0 Duplicates: 0 Warnings: 0
Q. If Add tablespace? Is exist Create table.ibd ? Or not?
A. table.ibd는 생성되지 않고 table.frm만 생성이 된다. Inndb_file_per_table = 1로 되어 있어도 그렇다. 할당된 tablespace 해제도 안된다.
[root@LCSYSTS1509:/mysql/MyHome/data/rex]# ll
total 36
drwxr-x--- 2 mysql dba 4096 May 16 15:41 .
drwxr-xr-x 10 mysql dba 4096 May 16 15:37 ..
-rw-r----- 1 mysql dba 61 Feb 29 14:32 db.opt
-rw-r----- 1 mysql dba 8560 May 16 15:41 employees.frm
-rw-r----- 1 mysql dba 8556 May 16 15:39 http_req.frm
root@localhost:rex 19:05:23>alter table employees discard tablespace;
ERROR 1148 (42000): InnoDB: Cannot discard table `rex/employees` because it is in a general tablespace. It must be file-per-table.
Error (Code 1148): InnoDB: Cannot discard table `rex/employees` because it is in a general tablespace. It must be file-per-table.
Error (Code 1148): The used command is not allowed with this MySQL version

show global variables like 'innodb_file_per_table';
+-----------------------+-------+
+-----------------------+-------+
| innodb_file_per_table | ON |
+-----------------------+-------+
DROP TABLESPACE Logs;
ERROR 1529 (HY000): Failed to drop TABLESPACE Logs
Error (Code 1529): Failed to drop TABLESPACE Logs
Error (Code 3120): Tablespace `Logs` is not empty.
drop table http_req;
drop table employees;
DROP TABLESPACE Logs;
<- must be empty

• UNDO Log Space Management
Requires separate UNDO tablespaces to work
• Mysql5.6
innodb_undo_directory = /mysql/MyHome/data/undo
innodb_undo_tablespaces = 5 – default 0
innodb_undo_logs = 128
-- Defines the number of undo logs
• Mysql5.7.5
Innodb_purge_rseg_truncate_frequency – default 128 – advanced
innodb_max_undo_log_size = 1G – default
innodb_undo_log_truncate = on | off – default off
UNDO Truncate
http://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_undo_log_truncate
When you enable innodb_undo_log_truncate, undo tablespaces that exceed the threshold value defined by innodb_
max_undo_log_size are marked for truncation.
Only undo logs that reside in undo tablespaces can be truncated. Truncation of undo logs that reside in the syste
m tablespace is not supported. For truncation to occur, there must be at least two undo tablespaces and two redo
-enabled undo logs configured to use the undo tablespaces. This means that innodb_undo_tablespaces must be s
et to a value equal to or greater than 2, and innodb_undo_logs must set to a value equal to or greater than 35.

• my.cnf
innodb_undo_tablespaces = 5
innodb_undo_directory = /mysql/MyHome/data/undo
Service mysql restart
2016-05-18T01:59:39.023426Z 0 [ERROR] InnoDB: Expected to open 5 undo tablespaces but was a
ble to find only 0 undo tablespaces. Set the innodb_undo_tablespaces parameter to the correct value
and retry. Suggested value is 0
The number of innodb_undo_tablespaces must be set prior to initializing InnoDB
UNDO Truncate cont.
http://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_undo_log_truncate

• Support for 32K and 64K Page Sizes
Larger BLOBs can be stored “on-page”
Better compression with the new transparent page compression
Larger Page Sizes
8M
16M

Additional Online ALTER TABLE support
- VARCHAR size up : varchar(100) -> varchar(255)
ALTER TABLE T_ONLINE_DDL ALGORITHM=INPLACE, CHANGE COLUMN text text VARCHAR(25
5);
- Rename Index
ALTER TABLE T_ONLINE_DDL RENAME INDEX idx_text TO index_text;
show global variables like 'innodb_online_alter_log_max_size';
+----------------------------------+-----------+
+----------------------------------+-----------+
| innodb_online_alter_log_max_size | 134217728 |
+----------------------------------+-----------+
* innodb_online_alter_log_max_size
- Specifies an upper limit on the size of the temporary log files used during online DDL operations for InnoDB tables.
Default 134217728
Min Value 65536
Max Value 2**64-1
- Scenario : Start Alter table => DML size > innodb_online_alter_log_max_size => error alter table
ERROR 1799 (HY000) : Creating index 'PRIMARY'required more than 'innodb_online_alter_log_max_size'bytes of modification log.
Please try again.
http://dev.mysql.com/doc/refman/5.6/en/innodb-create-index-implementation.html

Mysql5.6.17
Mysql5.7.10
Alter table change column

Mysql5.6.17
Mysql5.7.10
Alter table rename index

23
Bulk Load for Create Index
InnoDB enhancements
• Much faster INDEX creation and bulk loads
• Sorted index builds, done from the bottom-up
– Improves speed by increasing locality and decreasing node splitting
• Pages are compressed only when full
• New “innodb_fill_factor” controls free space left in each page
• Performance results show
– 2~3x performance improvement for ADD/CREATE INDEX operations
– 2~5% improvement for standard INSERT operations
As of MySQL 5.7.5, InnoDB performs a bulk load when creating or rebuilding indexes. T
his method of index creation is known as a “sorted index build”. innodb_fill_factor defin
es the percentage of space on each B-tree page that is filled during a sorted index buil
d, with the remaining space reserved for future index growth. For example, setting innod
b_fill_factor to 80 reserves 20 percent of the space on each B-tree page for future index
growth.
As of MySQL 5.7.5, InnoDB performs a bulk load when creating or rebuilding indexes. Prior to the introduction of bulk load inde
x creation, index entries were inserted into the B-tree one record at a time using insert APIs.
https://dev.mysql.com/doc/refman/5.7/en/create-index-bulk-load.html

24
Bulk Load for Create Index cont.
InnoDB enhancements
• Performance results show
– 2~3x performance improvement for ADD/CREATE INDEX operations ???
– 2~5% improvement for standard INSERT operations
drop table if exists xxx;
create table xxx
as
SELECT no
, lpad(no, 10, 0) as uid
, ceil(rand()*10000000) as point
, now() as regdate
FROM
(
SELECT @RNUM := @RNUM + 1 AS no
FROM ( SELECT @RNUM:=0 ) a
, information_schema.columns b
, information_schema.columns c
, information_schema.columns d
, information_schema.columns e
, information_schema.columns f
LIMIT 1,20000000
) A;
alter table xxx modify column regdate datetime null default null;
create unique index idx_uid on xxx (no,uid);
alter table xxx add primary key (no);
Create index idx_point on xxx(point);

25
Temporary Tables
InnoDB enhancements
• InnoDB as default storage engine for disk based temp tables
– Optimizer switched from MyISAM to InnoDB(faster) for internal temp tables
This variable was added in MySQL 5.7.5 with a default of MYISAM. In MySQL 5.7.6, the default value was changed to INNODB.
With this change, the optimizer uses the InnoDB storage engine by default for on-disk internal temporary tables.
> Mysql5.6 > Mysql5.7

26
Temporary Tables cont.
InnoDB enhancements
> Mysql5.6 > Mysql5.7
select a.*
from xxx a, xxx b
order by a.point, a.regdate, a.uid, b.no desc
limit 100;
select a.*
from xxx a, xxx b
order by a.point, a.regdate, a.uid, b.no desc
limit 100;
[root@LCSYSTS1509:/mysql/MyHome/tmp]# ll -h
total 445G
drwxr-xr-x 16 root dba 4.0K Apr 12 12:24 ..
srwxrwxrwx 1 mysql dba 0 May 30 08:31 mysql.sock
-rw-rw---- 1 mysql dba 445G May 30 10:03 #sql_e1a7_0.MYD
-rw-rw---- 1 mysql dba 1.0K May 30 08:56 #sql_e1a7_0.MYI
[root@LCSYSTS1509:/mysql/MyHome/data]# ll -h /mysql/MyHome/t
mp/
total 12K
drwxr-xr-x 12 7161 wheel 4.0K Mar 15 18:34 ..
-rw------- 1 mysql dba 6 May 30 10:19 mysql.sock.lock
[root@LCSYSTS1509:/mysql/MyHome/data]# ll -h ibtmp1
-rw-r----- 1 mysql dba 45G May 30 10:59 ibtmp1
root@localhost:information_schema 11:04:43>select * from INNODB
_TEMP_TABLE_INFO;
Empty set (0.00 sec)

27
InnoDB enhancements
• New separate tablespace for temporary tables
– Improved CREATE/DROP performance
– DDL changes are transient, which eliminates some disk IO
• Optimize DML operations
– No REDO logging, no change buffering, less locking
• New intrinsic temporary tables
– Specialized temporary tables with tailored ACID/MVCC semantics
– Light weight and ultra-fast,great for intermediate query execution operations

28
InnoDB enhancements
• mysql5.6 vs mysql5.7 using temporary query
[root@LCSYSTS1509:/mysql/MyHome/tmp]# ll -h
total 445G
drwxr-xr-x 16 root dba 4.0K Apr 12 12:24 ..
-rw-rw---- 1 mysql dba 445G May 30 10:03 #sql_e1a7_0.MYD
-rw-rw---- 1 mysql dba 1.0K May 30 08:56 #sql_e1a7_0.MYI

29
Full-Text CJK Support
InnoDB enhancements
• Two new Full-Text Parser plugins
• N-gram parser supports Chinese, Japanese, & Korean
– Supports all ideographic languages that do not use word delimiters
• MeCab parser supports Japanese
– Native Japanese focused language support
• Easily customized
– Token sizes, stop words, …
• Supports advanced searches
– BOOLEAN MODE, NATURAL LANGUAGE MODE, with Ranking
http://dev.mysql.com/doc/refman/5.7/en/fulltext-stopwords.html
Defines the n-gram token size
for the n-gram full-text parser as mysql5.7.6
In computing, stop words are words which are
filtered out before or after processing of natura
l language data (text).[

30
Full-Text CJK Support cont.
InnoDB enhancements
• N-gram parser
supports for Chinese ,Korean and Japanese
CREATE TABLE `fts_ngram` (
`id` BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT,
`title` VARCHAR(100) DEFAULT NULL,
PRIMARY KEY (`id`),
FULLTEXT KEY `ngram_idx` (`title`) WITH PARSER `ngram`
) ENGINE=INNODB;
Craete mysql5.6 fulltext
without ‘with parser’
ALTER TABLE fts_ngramADD FULLTEXT INDEX ngram_idx(title) WITH PARSER ngram;
CREATE FULLTEXT INDEX ngram_idx ON fts_ngram(title) WITH PARSER ngram;
--ngram-token-size := 1 .. 10 (default 2)

31
InnoDB enhancements
• MeCab parser
supports for Japanese morpheme(형태소) open source
INSTALL PLUGIN mecab SONAME 'libpluginmecab.so';
SHOW STATUS LIKE 'mecab_charset';
Show plugins;
mysql> CREATE TABLE articles(
FTS_DOC_ID BIGINT UNSIGNED AUTO_INCREMENT NOT NULL PRIMARY KEY,
title VARCHAR(100),
FULLTEXT INDEX mecab_idx (title) WITH PARSER mecab
) ENGINE=InnoDB CHARACTER SET utf8mb4;

32
InnoDB enhancements
• N-gram parser
supports for Chinese ,Korean and Japanese
PRIMARY KEY (`id`),
) ENGINE=INNODB;
INSERT INTO fts_ngram (title) VALUES('프랑스오픈페더러');
INSERT INTO fts_ngram (title) VALUES('프랑스오픈조코');
INSERT INTO fts_ngram (title) VALUES('프랑스오픈머레이');
INSERT INTO fts_ngram (title) VALUES('프랑스오픈바브링카');
INSERT INTO fts_ngram (title) VALUES('프랑스오픈니시코리');
INSERT INTO fts_ngram (title) VALUES('프랑스오픈숑가');
INSERT INTO fts_ngram (title) VALUES('프랑스오픈나달');
INSERT INTO fts_ngram (title) VALUES('프랑스오픈휴이트');
INSERT INTO fts_ngram (title) VALUES('프랑스오픈몽피스');
INSERT INTO fts_ngram (title) VALUES('프랑스오픈페더러짱');
INSERT INTO fts_ngram (title) VALUES('ありがとうごじゃいます');
INSERT INTO fts_ngram (title) VALUES('しゃしんおとります');
INSERT INTO fts_ngram (title) VALUES('テニスをします');
INSERT INTO fts_ngram (title) VALUES('れんしゅうをします');
INSERT INTO fts_ngram (title) VALUES('りょこうをします');
INSERT INTO fts_ngram (title) VALUES('せんたくをします');
Craete mysql5.6 fulltext
without ‘with parser’

33
InnoDB enhancements
• N-gram parser - supports for Chinese and Korean
SELECT * FROM fts_ngram
WHERE MATCH (title) AGAINST ('더러짱' IN NATURAL LANGUAGE MODE);
WHERE MATCH (title) AGAINST ('더러짱' IN BOOLEAN MODE);
WHERE MATCH (title) AGAINST ('します' IN NATURAL LANGUAGE MODE);
WHERE MATCH (title) AGAINST ('ます' IN NATURAL LANGUAGE MODE);
WHERE MATCH (title) AGAINST ('します' IN BOOLEAN MODE );
show global variables like '%ngram%';
+------------------+-------+
+------------------+-------+
| ngram_token_size | 2 |
+------------------+-------+

34
InnoDB enhancements
• N-gram parser : mysql5.6 vs mysql5.7
INSERT INTO fts_ngram (title) VALUES('작년프랑스오픈우승자는조코비치,페더러,나달,바브린카중에서과연누구일까요?');
INSERT INTO fts_ngram (title) VALUES('작년 프랑스오픈 우승자는 조코비치,페더러,나달,바브린카 중에서 과연 누구일까요?');
INSERT INTO fts_ngram (title) VALUES('작년 프랑스오픈 우승자는 조코비치,페 더러,나달,바브린카 중에서 과연 누구일까요?');
INSERT INTO fts_ngram (title) VALUES('작년 프랑스오픈 우승자는 조코비치,페더 러,나달,바브린카 중에서 과연 누구일까요?');
INSERT INTO fts_ngram (title) VALUES('작년 프랑스 오픈 우승자는 조코비치,페더러,나달,바브린카 중에서 과연 누구일까요?');
INSERT INTO fts_ngram (title) VALUES('작년 프랑스 오픈 우승 자는 조코비치,페더러,나달,바브린카 중에서 과연 누구일까요?');
WHERE MATCH (title) AGAINST (‘우승' IN NATURAL LANGUAGE MODE);
SHOW GLOBAL VARIABLES LIKE '%innodb_ft_min_token_size%';
+--------------------------+-------+
+--------------------------+-------+
| innodb_ft_min_token_size | 1 |
+--------------------------+-------+
+------------------+-------+
+------------------+-------+
+------------------+-------+
PRIMARY KEY (`id`),
FULLTEXT KEY `ngram_idx` (`title`)
) ENGINE=INNODB;
PRIMARY KEY (`id`),
) ENGINE=INNODB;
Mysql 5.6 Mysql 5.7

35
InnoDB enhancements
• N-gram parser : boolean mode
INSERT INTO fts_ngram (title) VALUES('2005년 프랑스오픈 우승자는 나달이다.');
INSERT INTO fts_ngram (title) VALUES('2009년 프랑스오픈 우승자는 페더러이다.');
INSERT INTO fts_ngram (title) VALUES('2015년 프랑스오픈 우승자는 바브링카이다.');
WHERE MATCH (title) AGAINST ('우승자' IN BOOLEAN MODE);
+------------------+-------+
+------------------+-------+
+------------------+-------+
PRIMARY KEY (`id`),
) ENGINE=INNODB;
WHERE MATCH (title) AGAINST ('+우승자 -나달' IN BOOLEAN MODE);

Truncate bug
# mysql5.6.17
drop table tennis;
create table tennis (id int, nm varchar(100));
insert into tennis values(1,'Djokovic'),(2,'Murray'),(3,'Federer');
select * from tennis;
rm -rf tennis.ibd
ll -h
-rw-r----- 1 mysql dba 8.4K May 19 16:20 tennis.frm
truncate table tennis;
ll -h
-rw-r----- 1 mysql dba 96K May 19 16:21 tennis.ibd
insert into tennis values(4,'Wavrinca');
| 4 | Wavrinca |
ll -h
-rw-rw---- 1 mysql mysql 8.4K May 19 16:30 tennis.frm
-rw-rw---- 1 mysql mysql 96K May 19 16:31 tennis.ibd
root@localhost:rex 11:11:35>select * from tennis;
| 4 | Wavrinca |
# mysql5.7.10
drop table tennis;
create table tennis (id int, nm varchar(100));
insert into tennis values(1,'Djokovic'),(2,'Murray'),(3,'Federer');
rm -rf tennis.ibd
ll -h
truncate table tennis;
ll -h
insert into tennis values(4,'Wavrinca');
| 4 | Wavrinca |
ll -h
root@localhost:rex 11:10:59>select * from tennis;
ERROR 1812 (HY000): Tablespace is missing for table `rex`.`tennis`.
Warning (Code 1812): InnoDB: Tablespace is missing for table rex/tennis.
Error (Code 1812): Tablespace is missing for table `rex`.`tennis`.

Support Syslog on Linux / Unix OS
• Basic support with syslog
• Log output on syslog simply
• Dynamic option at running system
- log_syslog (ON/OFF, default OFF)

Support Syslog on Linux / Unix OS cont.
# tail -f /mysql/MyHome/logs/mysqld.err
2016-05-30T02:24:10.745378Z 0 [Warning] /mysql/MyHome/bin/mysqld: Forcing close of thread 4 user: 'agent'
2016-05-30T02:24:12.511463Z mysqld_safe mysqld from pid file /mysql/MyHome/logs/mysqld.pid ended
2016-05-30T02:24:13.446316Z mysqld_safe Starting mysqld daemon with databases from /mysql/MyHome/data
2016-05-30T02:24:13.685765Z 0 [Warning] The syntax '--log_warnings/-W' is deprecated and will be removed in a future release. Please use '--log_erro
r_verbosity' instead.
2016-05-30T02:24:13.685936Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server o
ption (see documentation for more details).
2016-05-30T02:24:13.685987Z 0 [Warning] Insecure configuration for --secure-file-priv: Current value does not restrict location of generated files. Consi
der setting it to a valid, non-empty path.
2016-05-30T02:24:13.696091Z 0 [ERROR] Plugin keyring_file reported: 'keyring_file initialization failure. Please check if the keyring_file_data points to read
able keyring file or keyring file can be created in the specified location. The keyring_file will stay unusable until correct path to the keyring file gets provided'
2016-05-30T02:24:13.697013Z 0 [Warning] InnoDB: Using innodb_file_format is deprecated and the parameter may be removed in future releases. See htt
p://dev.mysql.com/doc/refman/5.7/en/innodb-file-format.html
2016-05-30T02:24:16.611164Z 0 [ERROR] InnoDB: Operating system error number 2 in a file operation.
2016-05-30T02:24:16.611257Z 0 [ERROR] InnoDB: The error means the system cannot find the path specified.
2016-05-30T02:24:16.611294Z 0 [ERROR] InnoDB: If you are installing InnoDB, remember that you must create directories yourself, InnoDB does not creat
e them.
2016-05-30T02:24:16.611331Z 0 [ERROR] InnoDB: Cannot open datafile for read-only: './rex/tennis.ibd' OS error: 71
2016-05-30T02:24:16.611376Z 0 [ERROR] InnoDB: Operating system error number 2 in a file operation.
2016-05-30T02:24:16.611417Z 0 [ERROR] InnoDB: The error means the system cannot find the path specified.
2016-05-30T02:24:16.611456Z 0 [ERROR] InnoDB: If you are installing InnoDB, remember that you must create directories yourself, InnoDB does not creat
e them.
2016-05-30T02:24:16.611499Z 0 [ERROR] InnoDB: Could not find a valid tablespace file for `rex/tennis`. Please refer to http://dev.mysql.com/doc/refman
/5.7/en/innodb-troubleshooting-datadict.html for how to resolve the issue.
2016-05-30T02:24:16.611547Z 0 [Warning] InnoDB: Ignoring tablespace `rex/tennis` because it could not be opened.
2016-05-30T02:24:19.009747Z 0 [Warning] CA certificate ca.pem is self signed.
2016-05-30T02:24:19.012880Z 0 [Warning] 'user' entry 'root@localhost' ignored in --skip-name-resolve mode.
2016-05-30T02:24:19.012963Z 0 [Warning] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.
2016-05-30T02:24:19.013020Z 0 [Warning] 'user' entry 'agent@localhost' ignored in --skip-name-resolve mode.
2016-05-30T02:24:19.013109Z 0 [Warning] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.
2016-05-30T02:24:19.013141Z 0 [Warning] 'db' entry 'mysql agent@localhost' ignored in --skip-name-resolve mode.
2016-05-30T02:24:19.013200Z 0 [Warning] 'proxies_priv' entry '@ root@localhost' ignored in --skip-name-resolve mode.
2016-05-30T02:24:19.026155Z 0 [Warning] 'tables_priv' entry 'inventory agent@localhost' ignored in --skip-name-resolve mode.
2016-05-30T02:24:19.026203Z 0 [Warning] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.

Support Syslog on Linux / Unix OS cont.
# tail -f /var/log/messages
May 30 11:24:13 LCSYSTS1509 mysqld-mysql57_LINE[72438]: Plugin keyring_file reported: 'keyring_file initialization failure. Please check if the keyring_file
_data points to readable keyring file or keyring file can be created in the specified location. The keyring_file will stay unusable until correct path to the keyrin
g file gets provided'
May 30 11:24:13 LCSYSTS1509 mysqld-mysql57_LINE[72438]: InnoDB: Using innodb_file_format is deprecated and the parameter may be removed in futur
e releases. See http://dev.mysql.com/doc/refman/5.7/en/innodb-file-format.html
May 30 11:24:16 LCSYSTS1509 mysqld-mysql57_LINE[72438]: InnoDB: Operating system error number 2 in a file operation.
May 30 11:24:16 LCSYSTS1509 mysqld-mysql57_LINE[72438]: InnoDB: The error means the system cannot find the path specified.
May 30 11:24:16 LCSYSTS1509 mysqld-mysql57_LINE[72438]: InnoDB: If you are installing InnoDB, remember that you must create directories yourself, In
noDB does not create them.
May 30 11:24:16 LCSYSTS1509 mysqld-mysql57_LINE[72438]: InnoDB: Cannot open datafile for read-only: './rex/tennis.ibd' OS error: 71
May 30 11:24:16 LCSYSTS1509 mysqld-mysql57_LINE[72438]: InnoDB: Operating system error number 2 in a file operation.
May 30 11:24:16 LCSYSTS1509 mysqld-mysql57_LINE[72438]: InnoDB: The error means the system cannot find the path specified.
May 30 11:24:16 LCSYSTS1509 mysqld-mysql57_LINE[72438]: InnoDB: If you are installing InnoDB, remember that you must create directories yourself, In
noDB does not create them.
May 30 11:24:16 LCSYSTS1509 mysqld-mysql57_LINE[72438]: InnoDB: Could not find a valid tablespace file for `rex/tennis`. Please refer to http://dev.mys
ql.com/doc/refman/5.7/en/innodb-troubleshooting-datadict.html for how to resolve the issue.
May 30 11:24:16 LCSYSTS1509 mysqld-mysql57_LINE[72438]: InnoDB: Ignoring tablespace `rex/tennis` because it could not be opened.
May 30 11:24:19 LCSYSTS1509 mysqld-mysql57_LINE[72438]: CA certificate ca.pem is self signed.
May 30 11:24:19 LCSYSTS1509 mysqld-mysql57_LINE[72438]: 'user' entry 'root@localhost' ignored in --skip-name-resolve mode.
May 30 11:24:19 LCSYSTS1509 mysqld-mysql57_LINE[72438]: 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.
May 30 11:24:19 LCSYSTS1509 mysqld-mysql57_LINE[72438]: 'user' entry 'agent@localhost' ignored in --skip-name-resolve mode.
May 30 11:24:19 LCSYSTS1509 mysqld-mysql57_LINE[72438]: 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.
May 30 11:24:19 LCSYSTS1509 mysqld-mysql57_LINE[72438]: 'db' entry 'mysql agent@localhost' ignored in --skip-name-resolve mode.
May 30 11:24:19 LCSYSTS1509 mysqld-mysql57_LINE[72438]: 'proxies_priv' entry '@ root@localhost' ignored in --skip-name-resolve mode.
May 30 11:24:19 LCSYSTS1509 mysqld-mysql57_LINE[72438]: 'tables_priv' entry 'inventory agent@localhost' ignored in --skip-name-resolve mode.
May 30 11:24:19 LCSYSTS1509 mysqld-mysql57_LINE[72438]: 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.
• Conclusion
- When log_syslog option value’s true stored both logs of them.

Performance_schema
• Which queries do not use indexes?

Performance_schema cont.
• Where did my memory go?
• select host, sum(current_allocated)
from x$memory_by_host_by_current_bytes group by 1;
• select user, sum(current_allocated) m_size
from memory_by_thread_by_current_bytes group by 1 order by m_size desc limit 10;
• select user, sum(current_allocated)
from x$memory_by_user_by_current_bytes group by 1;
• select event_name, current_alloc
from memory_global_by_current_bytes limit 10;
• select *
from memory_global_total;

MySQL 5.7 innodb_enhance_partii_20160527

43
ETC – System Monitoring(Linux)
Tips

44
ETC – Linux Profiler “perf”
• Install :
• # yum –y install perf
• Use cases :
• # perf top –z –stdio <== live monitoring
• # perf record –a –g –F 99 sleep 20 <== record 20sec of data
• # perf report | more <== report from collected data
• # perf report --sort comm,dso <== report from collected data
• # perf annotate <== jump to source code
• links :
• https://perf.wiki.kernel.org <== main resource
• http://www.brendangregg.com/perf.html <== the most fun stuff !!!
Tips

45
Profiling example: # perf top –z --stdio
Tips

49
Profiling example (5) – CPU & I/O
http://www.brendangregg.com/perf.html
### CPU
# git clone https://github.com/brendangregg/FlameGraph # or download it from github
# cd FlameGraph
# perf record -F 99 -ag -- sleep 60
# perf script | ./stackcollapse-perf.pl > out.perf-folded
# cat out.perf-folded | ./flamegraph.pl > perf-kernel.svg
# perf script | ./stackcollapse-perf.pl | ./flamegraph.pl > perf-kernel.svg
### I/O
# perf record -e block:block_rq_issue -e block:block_rq_complete -a sleep 120
# perf script | awk '{ gsub(/:/, "") } $5 ~ /issue/ { ts[$6, $10] = $4 } $5 ~ /complete/ { if (l = ts[$6, $9]) { printf "%.
f %.fn", $4 * 1000000, ($4 - l) * 1000000; ts[$6, $10] = 0 } }' > out.lat_us
# ./trace2heatmap.pl --unitstime=us --unitslat=us --maxlat=50000 out.lat_us > out.svg
Tips

51
htop – interactive process view
https://www.linux.co.kr/home2/board/subbs/board.php?bo_table=lecture&wr_id=1791
http://www.tecmint.com/install-htop-linux-process-monitoring-for-rhel-centos-fedora/
1. Install htop
case1)
Yum –y install htop
Case2)
wget -N http://mondb.linecorp.com/download/system/htop-1.0.3-1.el6.x86_64.rpm
rpm –Uvh htop-1.0.3-1.el6.x86_64.rpm
2. Features
- 프로세스 선택을 커서나, 마우스를 사용할 수 있다.
- pstree와 top을 함께 보는 것 같은 기능이 있다.
- 선택된 프로세스를 죽이거나, 우선순위를 변경할 수 있다.
- 특정 프로세스 이름으로 검색, 필터링 하여 현황을 볼 수 있다.
- 선택된 프로세스의 'lsof' 명령 결과를 볼 수 있다.
- 선택된 프로세스의 시스템콜을 확인하는 strace 를 할 수 있다.
- 마우스를 사용할 수 있다.
3. Convenient and Useful tool
- top, pstree, ps + lsof, strace, renice = htop
- ps –ef | grep mysql + strace –p xxxx = htop
Tips

53
vtop - A graphical activity monitor for the command line. Written in node.js.
https://github.com/MrRio/vtop
Tips

54Tips
> select * from information_schema.GLOBAL_VARIABLES where variable_name like 'innodb_%';
ERROR 3167 (HY000): The 'INFORMATION_SCHEMA.GLOBAL_VARIABLES' feature is disabled; see th
e documentation for 'show_compatibility_56‘
> Show global variables like ‘show_compatibility_56’;
> set global show_compatibility_56 = 1;
Warning (Code 1287): 'INFORMATION_SCHEMA.GLOBAL_VARIABLES' is deprecated and will be remo
ved in a future release. Please use performance_schema.global_variables instead
Mysql5.7 information_schema.GLOBAL_VARIABLES

55Tips
>select @@innodb_buffer_pool_size;
+---------------------------+
| @@innodb_buffer_pool_size |
+---------------------------+
| 10737418240 |
+---------------------------+
>select 10737418240/1024/1024/1024;
+----------------------------+
| 10737418240/1024/1024/1024 |
+----------------------------+
| 10.000000000000 |
+----------------------------+
Mysql useful Function
>select 10737418240/power(1024,3);
+---------------------------+
| 10737418240/power(1024,3) |
+---------------------------+
| 10 |
+---------------------------+
>select 10737418240/pow(1024,3);
+---------------------------+
| 10737418240/pow(1024,3) |
+---------------------------+
| 10 |
+---------------------------+
http://dev.mysql.com/doc/refman/5.7/en/mathematical-functions.html

Reference 56
# MySQL Performance Architect
http://dimitrik.free.fr/blog/index.html
# Linux Profiler
http://www.pixelbeat.org/programming/profiling/
# Linux Profiling at Netflix
http://www.slideshare.net/brendangregg/scale2015-linux-perfprofiling?ref=http://www.brendangregg.com/perf.html
# Perf profiling – Brendan GREGG
http://www.brendangregg.com/perf.html
# innodb architecture and internals
http://www.slideshare.net/mysqlops/pldc2012-innodb-architecture-and-internals?qid=083bec48-1ef1-4e20-808d-69b2bef6ed67&v=&b=&from_search=7
# double write buffer
https://www.percona.com/blog/2014/05/23/improve-innodb-performance-write-bound-loads/
# MySQL 5.6 Performance – dirty pages
http://www.slideshare.net/myxplain/my-sql-56performance
# Redo loging in InnoDB (redo log, mtr)
https://blogs.oracle.com/mysqlinnodb/entry/redo_logging_in_innodb
# MySQL buffer pool usage + high load
http://dba.stackexchange.com/questions/73343/mysql-buffer-pool-usage-high-load

MySQL 5.7 innodb_enhance_partii_20160527

More Related Content

What's hot (20)

Similar to MySQL 5.7 innodb_enhance_partii_20160527 (20)

Recently uploaded (20)

MySQL 5.7 innodb_enhance_partii_20160527