Performance Schema for MySQL troubleshooting

<Insert Picture Here>

Performance Schema for MySQL troubleshooting
Sveta Smirnova
Senior Principal Technical MySQL Support Engineer

Content
•
•
•
•
•
•
•
•
•

History of Performance Schema
Tables for DBA
Tables for developers
Other tables
Tools
Performance and tests
Options
Information sources
Conclusion

History of Performance Schema
•
•
•
•

First version: in MySQL 5.5
17 tables
Useful mostly for developers of MySQL code
Tools for
– Mutexes
– Locks

• Required good knowledge of MySQL code

Kinds of tables
• Settings
– _setup
– _instances

• Events
– events_waits_

• Digests
• History
• Other

Version 5.6 turned its face to DBA
• More features
• 52 tables
• New tables, very useful
for DBA
• Knowledge of MySQL
source code is not a
requirement anymore

*That's me talking at Devconf 2012 about how I am,
as MySQL Support engineer,
is happy with new features in Performance Schema

*

Tables for DBA
• events_statements_*
• events_stages_*
• Connection

events_statements_*
• Statements
– statement/sql
• statement/sql/delete
• statement/sql/select

• Commands
– COM_PING, COM_QUIT, ...
– statement/com
• statement/com/Ping
• statement/com/Quit

• Errors
– statement/sql/error
– statement/com/Error

events_statements_*:
which queries finished with an error
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●

mysql> select THREAD_ID, substr(SQL_TEXT, 1, 20),
MYSQL_ERRNO from  events_statements_history_long where
MYSQL_ERRNO != 0;
++++
| THREAD_ID | substr(SQL_TEXT, 1, 20) | MYSQL_ERRNO |
++++
|        18 | select from * event_    |        1064 |
|        18 | select * from  event    |        1146 |
|        18 | select * from  event    |        1146 |
|        18 | select THREAD_ID, SQ    |        1146 |
++++
4 rows in set (0.00 sec)

events_statements_*:
queries which need to be optimized
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●

mysql> select THREAD_ID as TID, substr(SQL_TEXT, 1, 20)
as SQL_TEXT, ROWS_SENT as RS, ROWS_EXAMINED as RE from
events_statements_history_long where ROWS_EXAMINED >
ROWS_SENT * 10 limit 5;
+++++
| TID | SQL_TEXT             | RS | RE  |
+++++
|  18 | select THREAD_ID, SQ |  4 | 147 |
|  18 | select THREAD_ID, su |  4 | 148 |
+++++

events_statements_*: what also is worth attention
•
•
•
•
•
•
•
•
•

CREATED_TMP_DISK_TABLES
CREATED_TMP_TABLES
SELECT_FULL_JOIN
SELECT_RANGE_CHECK
SELECT_SCAN
SORT_MERGE_PASSES
SORT_SCAN
NO_INDEX_USED
NO_GOOD_INDEX_USED

events_statements_*: ps_helper view
•
•
•
•
•
•
•

http://www.markleith.co.uk/ps_helper/
View: statement_analysis
View: statements_with_runtimes_in_95th_percentile
View: statements_with_temp_tables
View: statements_with_sorting
View: statements_with_full_table_scans
View: statements_with_errors_or_warnings

event_stages_*
• Same information which you see in table
INFORMATION_SCHEMA.PROCESSLIST or SHOW
PROCESSLIST output
–
–
–
–

init
executing
Opening tables
...

• Replacement of SHOW PROFILE
• Only server-level
• No information from storage engine in this table!

event_stages_*:
«Sending data» for more than 10 seconds
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●

mysql> select events_stages_history_long.event_name,
sql_text,
events_stages_history_long.timer_wait/1000000000000
wait_s from events_stages_history_long join
events_statements_history_long on
(events_stages_history_long.nesting_event_id =
events_statements_history_long.event_id) where
events_stages_history_long.EVENT_NAME like '%Sending
data' and rows_sent < 10000000 and
events_stages_history_long.timer_wait > 10*1000000000000
order by events_stages_history_long.timer_wait descG
************************ 1. row ************************
event_name: stage/sql/Sending data
  sql_text: insert into test.t2 select * from test.t2
    wait_s: 243.5235

event_stages_*:
other operations which can run slow
• Everything, related to temporary tables
– EVENT_NAME LIKE 'stage/sql/%tmp%'

• Everything, related to locks
– EVENT_NAME LIKE 'stage/sql/%lock%'

• Everything in state «Waiting for»
– EVENT_NAME LIKE 'stage/%/Waiting for%'

• Frequently met issues (from my Support experience)
–
–
–
–
–

EVENT_NAME='stage/sql/end'
EVENT_NAME='stage/sql/freeing items'
EVENT_NAME='stage/sql/Sending data'
EVENT_NAME='stage/sql/cleaning up'
EVENT_NAME='stage/sql/closing tables'

event_stages_*: longest queries
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●

mysql> select eshl.event_name, sql_text,
eshl.timer_wait/1000000000000 wait_s from
events_stages_history_long eshl join
events_statements_history_long esthl on
(eshl.nesting_event_id = esthl.event_id) where
eshl.timer_wait > 10*1000000000000G
************************ 1. row ************************
event_name: stage/sql/copy to tmp table
  sql_text: alter table t2 engine=innodb
    wait_s: 186.8122
************************ 2. row ************************
event_name: stage/sql/Waiting for table metadata lock
  sql_text: insert into t2 select * from t2 LIMIT 10
    wait_s: 46.6250

event_stages_*: joins
• NESTING_EVENT_ID
– Statement
– Wait
– Stage

• EVENT_ID

events_statements
EVENT_ID
events_stages
NESTING_EVENT_ID
events_stages
NESTING_EVENT_ID
events_stages
NESTING_EVENT_ID

Connection Tables: accounts
●
●
●
●
●
●
●
●
●
●
●
●

mysql> select user, host, current_connections as cur,
total_connections as total from accounts;
+++++
| user | host      | cur | total |
+++++
| foo  | localhost |   0 |     3 |
| root | localhost |   1 |     3 |
| NULL | NULL      |  14 |    17 |
+++++

Connection Tables: users, hosts
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●

mysql> select user, current_connections as cur,
total_connections as total from users;
++++
| user | cur | total |
++++
| root |   1 |     3 |
| NULL |  14 |    17 |
| foo  |   0 |     3 |
++++
mysql> select host, current_connections as cur,
total_connections as total from hosts;
++++
| host      | cur | total |
++++
| NULL      |  14 |    17 |
| localhost |   1 |     6 |
++++

Connection Attribute Tables
●
●
●

mysql_init(&mysql);
mysql_options(&mysql,MYSQL_OPT_CONNECT_ATTR_RESET, 0);

●
●
●
●
●
●
●
●

mysql_options4(&mysql,MYSQL_OPT_CONNECT_ATTR_ADD,
"program", "Devconf2013");
"author", "Sveta Smirnova");
"session", "MySQL Performance Schema");

●
●
●
●

mysql_real_connect(&mysql, "127.0.0.1", "root", "",
"test", 13000, NULL, 0);

Connection Attribute Tables: foreigners prohibited!
●
●
●
●
●
●
●
●
●
●
●

mysql> select PROCESSLIST_ID as PID, ATTR_NAME,
ATTR_VALUE from session_account_connect_attrs where
attr_name='program_name';
++++
| PID | ATTR_NAME    | ATTR_VALUE  |
++++
|   9 | program_name | mysql       |
|  13 | program_name | Devconf2013 |
++++

Connection Attribute Tables: foreigners prohibited!
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●

mysql> select PROCESSLIST_ID as PID, ATTR_NAME,
ATTR_VALUE from session_account_connect_attrs where
attr_name='program_name' union select PROCESSLIST_ID as
PID, 'program_name' as ATTR_NAME,
sum(if(attr_name='program_name', 1, 0)) as ATTR_VALUE
from session_account_connect_attrs group by
processlist_id having(ATTR_VALUE=0);
++++
| PID | ATTR_NAME    | ATTR_VALUE  |
++++
|   9 | program_name | mysql       |
|  13 | program_name | Devconf2013 |
|  21 | program_name | 0           |
++++

host_cache
• Content of DNS cache
• Errors from
–
–
–
–

Name server
Connection
Authentication
max_connect_errors, max_user_errors, etc.

• Your first assistant in case of connection issue

threads
• Two kinds of THREADS
– Background
– Foreground

• Fields
– THREAD_ID
• Internal thread id
– PROCESSLIST_ID
• id, observable in the SHOW PROCESSLIST output
– NAME
• Instrument
– PARENT_THREAD_ID
• Internal id of the parent thread
– PROCESSLIST_*
• Only for для FOREGROUND threads

threads
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●

mysql> select * from threads where type = 'foreground'G
************************ 1. row ************************
          THREAD_ID: 16
               NAME: thread/sql/one_connection
               TYPE: FOREGROUND
     PROCESSLIST_ID: 1
   PROCESSLIST_USER: root
   PROCESSLIST_HOST: localhost
     PROCESSLIST_DB: performance_schema
PROCESSLIST_COMMAND: Query
   PROCESSLIST_TIME: 0
  PROCESSLIST_STATE: Sending data
   PROCESSLIST_INFO: select * from threads where type =
'foreground'
   PARENT_THREAD_ID: 1
               ROLE: NULL
       INSTRUMENTED: YES
1 row in set (0.00 sec)

events_waits_*
• EVENT_NAME
– wait/synch/rwlock/innodb/dict_operation_lock

• SOURCE
– Line of the source code

• OPERATION
– Kind of operation: read, lock, write

event_waits_*
●
●
●
●
●
●
●
●
●
●
●
●
●

mysql> select e.EVENT_NAME, e.SOURCE, e.OPERATION,
t.PROCESSLIST_INFO from events_waits_current e join
threads t using(thread_id) where type='foreground' and
processlist_id != 1G
************************ 1. row ************************
      EVENT_NAME:
wait/synch/cond/sql/Item_func_sleep::cond
          SOURCE: item_func.cc:4212
       OPERATION: timed_wait
PROCESSLIST_INFO: select sleep(100) from t1
1 row in set (0.01 sec)

●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●

$ cat n  sql/item_func.cc | head n 4220 | tail n 35
4186
4187 /**
4188   Wait for a given condition to be signaled.
4189
4190   @param cond   The condition variable to wait on.
4191   @param mutex  The associated mutex.
4192
4193   @remark The absolute timeout is preserved across
calls.
4194
4195   @retval return value from mysql_cond_timedwait
4196 */
4197

●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●

4198 int Interruptible_wait::wait(mysql_cond_t
*cond, mysql_mutex_t *mutex)
4199 {
4200   int error;
4201   struct timespec timeout;
4202
4203   while (1)
4204   {
4205     /* Wait for a fixed interval. */
4206     set_timespec_nsec(timeout,
m_interrupt_interval);
4207
4208     /* But only if not past the absolute
timeout. */
4209     if (cmp_timespec(timeout, m_abs_timeout) > 0)
4210       timeout= m_abs_timeout;

●
●
●
●
●
●
●
●
●
●
●
●
●
●
●


4212     error= mysql_cond_timedwait(cond, mutex,
&timeout);
4213     if (error == ETIMEDOUT || error == ETIME)
4214     {
4215       /* Return error if timed out or connection
is broken. */
4216       if (!cmp_timespec(timeout, m_abs_timeout) ||
!m_thd>is_connected())
4217         break;
4218     }
4219     /* Otherwise, propagate status to the caller.
*/
4220     else

Query statistics
●
●
●
●
●

mysql> UPDATE performance_schema.threads SET
instrumented = 'NO';
Query OK, 15 rows affected (0.04 sec)
Rows matched: 15  Changed: 15  Warnings: 0

●
●

Open new connection

●
●

mysql> truncate events_waits_history_long;



●
●

In new connection

●
●
●
●
●

mysql2> create temporary table norepl_t1 engine=myisam
select amount, price, money, id_product from test;
Records: 262144  Duplicates: 0  Warnings: 0

Query events_waits_history_long
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●

mysql> select e.EVENT_NAME, e.SOURCE, e.OPERATION,
count(*) as cnt from events_waits_history_long e join
threads t using(thread_id) where type='foreground' and
processlist_id not in (1, @@pseudo_thread_id) group by
e.EVENT_NAME, e.SOURCE, e.OPERATION order by cnt descG
************************ 1. row ************************
EVENT_NAME: wait/synch/mutex/innodb/lock_mutex
    SOURCE: lock0lock.cc:5529
OPERATION: lock
       cnt: 1428
************************ 2. row ************************
OPERATION: lock
       cnt: 1428

●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●

************************ 3. row ************************
EVENT_NAME: wait/synch/mutex/innodb/trx_sys_mutex
OPERATION: lock
       cnt: 1428
************************ 4. row ************************
EVENT_NAME: wait/synch/mutex/innodb/trx_mutex
OPERATION: lock
       cnt: 1423
************************ 5. row ************************
EVENT_NAME: wait/io/table/sql/handler
    SOURCE: handler.cc:2627
OPERATION: fetch
       cnt: 1423

●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●

************************ 6. row ************************
OPERATION: lock
       cnt: 1421
************************ 7. row ************************
EVENT_NAME: wait/synch/mutex/innodb/trx_sys_mutex
    SOURCE: trx0sys.ic:431
OPERATION: lock
       cnt: 1421
************************ 8. row ************************
EVENT_NAME: wait/synch/mutex/innodb/buf_pool_mutex
    SOURCE: buf0buf.ic:887
OPERATION: lock
       Cnt: 6
...

Which kind of events can we examine?
• setup_instruments.NAME
– wait/io/file
• Operations with files
– wait/io/socket
– wait/io/table/sql/handler
– wait/lock/table/sql/handler
– wait/synch/cond
• InnoDB, MyISAM, sql
– wait/synch/mutex
• sql, mysys, storage engines
– wait/synch/rwlock/
• sql, InnoDB, MyISAM

ps_helper
• All VIEWs work for MySQL 5.5
–
–
–
–
–
–
–

latest_file_io
top_io_by_file
top_io_by_thread
top_global_consumers_by_avg_latency
top_global_consumers_by_total_latency
top_global_io_consumers_by_latency
top_global_io_consumers_by_bytes_usage

• There are few views for 5.6 which use digest tables

*_instances tables
• file_instances
– Opened files

• socket_instances
– Connections

• cond_instances
• rwlock_instances
– select * from rwlock_instances where
READ_LOCKED_BY_COUNT > 0;
– select * from rwlock_instances where
WRITE_LOCKED_BY_THREAD_ID > 0;

• mutex_instances
– LOCKED_BY_THREAD_ID

Digests
•
•
•
•
•
•
•
•

events_stages_*
events_statements_*
events_waits_*
file_*
objects_*
socket_*
table_io_waits_*
table_lock_waits_*

Digests: events_stages_summary_*
• events_stages_summary_by_account_by_event_name
– Helps to find an account which performs problematic queries

• events_stages_summary_by_host_by_event_name
• events_stages_summary_by_user_by_event_name
– Same, but sorted by host and user name

• events_stages_summary_by_thread_by_event_name
– Easy to find out what makes troubles on your server right now
– Since statistics is saved for some time you can find it and after the
problem stopped to show up

• events_stages_summary_by_global_by_event_name
– Global stats by event name
– Does not indicate user, host, account and thread

•
•
•
•
•

Digests: events_statements_summary_*
events_statements_summary_by_account_by_event_name
events_statements_summary_by_host_by_event_name
events_statements_summary_by_user_by_event_name
events_statements_summary_by_thread_by_event_name
events_statements_summary_global_by_event_name
– Same as stages, but stats are taken from tables events_statements_*

• events_statements_summary_by_digest
– Stats by digest field:
• 42b93d481e96b9c9b4049b9407900194
• Query written as SELECT fname FROM tname WHERE fname = ?
– For example, you can find all statements which create temporary tables by
querying this table

Digests: events_waits_summary_*
•
•
•
•
•

events_waits_summary_by_account_by_event_name
events_waits_summary_by_host_by_event_name
events_waits_summary_by_thread_by_event_name
events_waits_summary_by_user_by_event_name
events_waits_summary_global_by_event_name
– Similar to events_stages_* digests

• events_waits_summary_by_instance
– By OBJECT_INSTANCE_BEGIN field

Other digests
• file_summary_by_event_name
– Does not show file name!

•
•
•
•
•

file_summary_by_instance
objects_summary_global_by_type
socket_summary_by_event_name
socket_summary_by_event_name
socket_summary_by_instance
– By OBJECT_INSTANCE_BEGIN field

• table_io_waits_summary_by_index_usage
• table_io_waits_summary_by_table
• table_lock_waits_summary_by_table

Digests
• WHERE COUNT_STAR > 0
• Sort or query by an operation you are interested in
• Sort by COUNT_STAR

Performance: version 5.5
• Performance Schema is OFF by default
• Noticeable performance issues
– Up to 7% in case of RO load
– Up to 20% in case of RW load
– Numbers based on tests by Dimitri Kravtchuk
(http://dimitrik.free.fr/blog/archives/2010/05/mysql-performance-using-performance-schema.html )

• No performance loss if turned off

Performance: version 5.6
• Performance Schema is ON by default
• Performance loss can happen, but not big
– Not more than 5% for most setups, likely near 0
– Maximum up to 10% in case if all instrumentations are turned ON
– Numbers based on tests by Dimitri Kravtchuk
(http://dimitrik.free.fr/blog/archives/2012/06/mysql-performance-pfs-overhead-in-56.html)

• global_instrumentation
– Minimal overhead

• Detailed instrumentation
– Noticeable overhead

• History tables
– minimal overhead

What, where and when to setup
• At compile time
• At the server startup
– Options in my.cnf
– All options are static

• Runtime
– setup_* tables

• What can you tune?
– Look for tables documentation

Configuration options
• performance_schema = ON|OFF
– Is it On or Off?

• performance_schema_%_size
– Size of history tables
– Size of instrumented objects

• performance_schema_max_%_classes
– Maximum number of cond|fle|io|% instruments

• performance_schema_max_%_instances
– Maximum number of cond|fle|io|% objects

Configuration options
• performance_schema_consumer_TABLE_NAME
– performance_schema_consumer_events_stages_current
– performance_schema_consumer_events_waits_current
– ...

• Turns instrumentations On of Off
– OFF, FALSE, 0
– ON, TRUE, 1

• setup_consumers table
– update setup_consumers set enabled='no'
where name='events_stages_current';

Tables setup_actors and setup_objects
• setup_actors
–
–
–
–

Which user threads to monitor
DELETE , then INSERT
UPDATE not allowed
insert into setup_actors values('%', 'sveta', '%');
• Only for user sveta

• setup_objects
– Which objects to monitor
– update setup_objects set enabled='no'
where object_schema='%';
– insert into setup_objects values
('TABLE', 'test', 't1', 'YES', 'YES');

•
•
•
•

setup_instruments table
Detailed setup of instruments
549 instruments in the standard distribution*
update setup_instruments set enabled='no';
update setup_instruments set enabled='yes'
where name like 'statement%';

*Written at June, 2013. Subject to change.

Timers
●

●

Values for your machine

●
●
●
●
●
●
●
●
●
●
●
●

mysql> select * from performance_timers;
+++++
| TIMER_NAME  | TIMER_FREQUENCY | TIMER_RESOLUTION | TIMER_OVERHEAD |
+++++
| CYCLE       |      2592796019 |                1 |             18 |
| NANOSECOND  |      1000000000 |                1 |             45 |
| MICROSECOND |         1000000 |                1 |             48 |
| MILLISECOND |            1037 |                1 |             54 |
| TICK        |             103 |                1 |            547 |
+++++

●

●
●
●

How to tune:
mysql> update setup_timers set timer_name='tick'
where name = 'stage';

•
•
•
•
•

What happens inside Performance Schema?
SHOW ENGINE PERFORMANCE_SCHEMA STATUS;
Contains information about memory usage
Table_name.attribute
(Internal_buffer).attribute
*.size, *.row_size
– Not-configurable, for example, size of a table row

• *.count, *.row_count
– Configurable with help of options

• *.memory
– size * count
– events_waits_history_long.memory
– performance_schema.memory

Where to find information?
• http://www.markleith.co.uk/ps_helper/
• http://www.drdobbs.com/database/detailed-profiling-of-sql-activity-in-my/240154959

• http://marcalff.blogspot.ru
• http://dimitrik.free.fr/blog/
• http://dev.mysql.com/doc/refman/5.6/en/performance-schema.html

Conclusion
• Performance schema — wonderful tool for a DBA
when she needs to troubleshoot performance issue
• You can configure it online: without server restart
• Allows very detailed setup
• Always tune it for your own needs!
• Don't instrument everything: use it for operations you
are interested in only

The preceding is intended to outline our general
product direction. It is intended for information
purposes only, and may not be incorporated into any
contract. It is not a commitment to deliver any
material, code, or functionality, and should not be
relied upon in making purchasing decisions.
The development, release, and timing of any
features or functionality described for Oracle’s
products remains at the sole discretion of Oracle.

Performance Schema for MySQL troubleshooting

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Performance Schema for MySQL troubleshooting (20)

More from Sveta Smirnova (20)

Recently uploaded (20)

Performance Schema for MySQL troubleshooting