MySQL Performance Metrics that Matter
7 likes3,512 views
This document discusses various MySQL performance metrics that are important to measure from within the database, operating system, and application. It outlines key InnoDB internal structures like the buffer pool and log system. Specific metrics that provide insight into buffer pool usage, page churn, and log writes are highlighted. Optimizing the working set size and ensuring sufficient free space in the log files are important factors for performance.
![Memory
Usage
[root@ip-10-183-136-169 mysql]# top -bn1
top - 18:24:52 up 1:27, 1 user, load average: 0.73, 0.47, 0.22
Tasks: 92 total, 1 running, 91 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.1%sy, 0.0%ni, 98.0%id, 0.6%wa, 0.0%hi, 0.0%si, 1.2%st
Mem: 70201328k total, 7684128k used, 62517200k free, 16356k buffers
Swap: 0k total, 0k used, 0k free, 4233840k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
..
18842 mysql 20 0 43.2g 2.9g 5832 S 0.0 4.3 0:09.86 mysqld
..
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
• Understand
difference
between
virtual
and
resident
size:
38](https://image.slidesharecdn.com/performance-metrics-that-matter-141029114202-conversion-gate02/85/MySQL-Performance-Metrics-that-Matter-38-320.jpg)
![Memory
Usage
(cont.)
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
• After
40G
buffer
pool
fills:
39
[root@ip-10-183-136-169 ~]# top -bn1 | grep 'PID|mysqld'
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
18842 mysql 20 0 43.3g 42g 6336 S 65.4 64.1 20:53.31 mysqld
18611 root 20 0 105m 1464 1172 S 0.0 0.0 0:00.03 mysqld_safe](https://image.slidesharecdn.com/performance-metrics-that-matter-141029114202-conversion-gate02/85/MySQL-Performance-Metrics-that-Matter-39-320.jpg)
MySQL Performance Metrics that Matter
- 2. MySQL Performance Metrics That Matter Morgan Tocker MySQL Community Manager October, 2014 Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
- 3. Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 3
- 4. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 4 Today’s Agenda Introduction From the Database From the Operating System From the Application 1 2 3 4
- 5. Today’s Question Start With Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • Will my database server be able to scale another 2x? • How about 10x? • Is it under load? • When I run this benchmark on my macbook… 5
- 6. These are actually hard questions Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • Number of active user sessions by 10x? • Database size by 10x? • Working set by 10x? • Some query patterns will scale better than others. • Unfortunate answer: it depends 6
- 7. What questions can we answer? Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • What can we measure that is useful to gauge performance? • Let’s include scope as: • Inside MySQL • From Operating System • From your application 7
- 8. Performance Phrasebook Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • Response -‐ “seconds per query”. • Throughput -‐ “queries per second”. • Scalability -‐ how big can I make throughput? Most benchmarks measure throughput. Users care about response time. 8
- 9. Performance Consistency Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • Response Time should be measured in a percentile-‐ based score. • i.e. 99th percentile. • Measuring throughput only does not show sudden stalls that users may experience. • e.g. an expensive item in cache expiring. 9
- 10. Today’s Agenda Introduction From the Database From the Operating System From the Application Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 10 1 2 3 4
- 11. iblogfile0 iblogfile1 iblogfile2 Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 11 Storage Caching Transaction System SYS_TABLES ibdata1 space 0 Page Cache A.ibd B.ibd C.ibd IBUF_HEADER IBUF_TREE TRX_SYS FIRST_RSEG DICT_HDR Data Dict. SYS_COLUMNS SYS_INDEXES SYS_FIELDS Block 1 (64 pages) Block 2 (64 pages) Tables with Doublewrite Buffer file_per_table Buffer Pool Data Dictionary Cache Adaptive Hash Indexes Buffer Pool LRU Additional Mem Pool Log Buffer Log Group Buffer Pool Flush List In Memory https://github.com/jeremycole/innodb_diagrams On Disk
- 12. iblogfile0 iblogfile1 iblogfile2 Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 12 InnoDB Storage Caching Transaction System SYS_TABLES ibdata1 space 0 Page Cache A.ibd B.ibd C.ibd IBUF_HEADER IBUF_TREE TRX_SYS FIRST_RSEG DICT_HDR Data Dict. SYS_COLUMNS SYS_INDEXES SYS_FIELDS Block 1 (64 pages) Block 2 (64 pages) Tables with Doublewrite Buffer file_per_table Buffer Pool Data Dictionary Cache Adaptive Hash Indexes Buffer Pool LRU Additional Mem Pool Log Buffer Log Group Buffer Pool Flush List SELECT * FROM a WHERE id = 10; mysqld Not Found Query -‐ Non Cached
- 13. iblogfile0 iblogfile1 iblogfile2 Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 13 InnoDB Storage Caching Transaction System SYS_TABLES ibdata1 space 0 Page Cache A.ibd B.ibd C.ibd IBUF_HEADER IBUF_TREE TRX_SYS FIRST_RSEG DICT_HDR Data Dict. SYS_COLUMNS SYS_INDEXES SYS_FIELDS Block 1 (64 pages) Block 2 (64 pages) Tables with Doublewrite Buffer file_per_table Buffer Pool Data Dictionary Cache Adaptive Hash Indexes Buffer Pool LRU Additional Mem Pool Log Buffer Log Group Buffer Pool Flush List SELECT * FROM a WHERE id = 10; mysqld Query -‐ Cached
- 14. iblogfile0 iblogfile1 iblogfile2 Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Update Query in a Transaction (simplified) 14 InnoDB Storage Caching Transaction System SYS_TABLES ibdata1 space 0 Page Cache A.ibd B.ibd C.ibd IBUF_HEADER IBUF_TREE TRX_SYS FIRST_RSEG DICT_HDR Data Dict. SYS_COLUMNS SYS_INDEXES SYS_FIELDS Block 1 (64 pages) Block 2 (64 pages) Tables with Doublewrite Buffer file_per_table Buffer Pool Data Dictionary Cache Adaptive Hash Indexes Buffer Pool LRU Additional Mem Pool Log Buffer Log Group Buffer Pool Flush List UPDATE a SET col1 = ‘new’ WHERE id = 10; mysqld commit;
- 15. Metrics to Measure Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • Size of the buffer pool: innodb_buffer_pool_size • Amount of the buffer pool currently being used: 15 ---------------------- BUFFER POOL AND MEMORY ---------------------- Total large memory allocated 137494528 Dictionary memory allocated 3129906 Buffer pool size 8192 Free buffers 1037 Database pages 7153 .. 5.59 reads/s, 341.58 creates/s, 378.21 writes/s Quick Math: 8192 as 16K pages = 128M (default) 16MB free.
- 16. Metrics to Measure Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • If there are free buffers, and still reads per second, the caches are probably still warming up: 16 ---------------------- BUFFER POOL AND MEMORY ---------------------- Total large memory allocated 137494528 Dictionary memory allocated 3129906 Buffer pool size 8192 Free buffers 1037 Database pages 7153 .. 5.59 reads/s, 341.58 creates/s, 378.21 writes/s
- 17. Metrics to Measure Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • If there are few free buffers, and a large number of reads, your working set is not fitting in memory: 17 ---------------------- BUFFER POOL AND MEMORY ---------------------- Total large memory allocated 137494528 Dictionary memory allocated 3129906 Buffer pool size 8192 Free buffers 1037 Database pages 7153 .. 5.59 reads/s, 341.58 creates/s, 378.21 writes/s
- 18. Metrics to Measure Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • Modified (dirty) pages shows work that has been delayed to be completed later. innodb_max_dirty_pages_pct (default: 75) helps ensures this number does not get too high: 18 ---------------------- BUFFER POOL AND MEMORY ---------------------- Total large memory allocated 137494528 Dictionary memory allocated 3129906 Buffer pool size 8192 .. Modified db pages 3217 5.59 reads/s, 341.58 creates/s, 378.21 writes/s
- 19. InnoDB Page Churn Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • 5.1 and below -‐ LRU. • 5.5 and above -‐ midpoint insertion. • 5.6 and above -‐ midpoint insertion with default 1s wait before hotlist promotion. 19
- 20. Reducing Buffer Pool Churn Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • Optimize Queries • Add Indexes / Remove Indexes • Increase buffer pool size / add more RAM • Transparent Page Compression • Table Partitioning • Normalize Table Schema • Denormalize Table Schema • Change Page Size • Optimize Data Types • Optimize Tables 20 .. By Improving Working Set Efficiency Danger: Not all of these techniques work all of the time!
- 21. How much churn is too much? Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • Rule of thumb is that if you need to access a page, you should be able to keep it in memory for 5 minutes. • Thus, having to churn out pages any more frequently means too little memory. 21
- 22. You can get close to this number from MySQL… mysql> call test.estimate_working_set(10, 30); .. +----------------------+ | pages_in_working_set | +----------------------+ | 100679 | +----------------------+ 1 row in set (5 min 55.61 sec) Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • It is possible to track distinct pages accessed over 5 minutes. 22 http://www.tocker.ca/2013/05/31/estimating-‐mysqls-‐working-‐set-‐with-‐information_schema.html
- 23. Twice as Much Ram != Twice as Fast Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • Production • 10G of RAM • 100 IOPS Storage Device (single hard disk) • Workload is 100% reads • 10K queries/second • QA • 9G of RAM • 100 IOPS Storage Device (single hard disk) • Same workload (100% reads) • Can only do 1K queries/ second! 23
- 24. InnoDB Log System Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • Like a tank tread. • Concatenated log files cyclic write. • Initial memory buffer, once per second + once per commit flush. 24
- 25. Log Metrics that Matter Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • Need to retain sufficient space • Head can not overwrite tail. 25 --- LOG --- Log sequence number 478969738 Log flushed up to 478969738 Pages flushed up to 418299472 Last checkpoint at 416542759 0 pending log flushes, 0 pending chkp writes 5784 log i/o's done, 5.95 log i/o's/second
- 26. Log Metrics that Matter Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • These are byte counts: • 57.8MB bytes log space used here • Compare to innodb_log_file_size*innodb_log_files_in_group (default:2) and innodb_adaptive_flushing_lwm (default: 10) 26 Log sequence number 478969738 Log flushed up to 478969738 Pages flushed up to 418299472 ..
- 27. Increasing Background Work Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • Specifying innodb_io_capacity, innodb_lru_scan_depth (mysql 5.6). • Rule of thumb: • 7200 RPM Hard Drive = 100 IOPS • Faster Hard drive = 200 IOPS • SSD = 3000+ IOPS (consult manufacturer details) 27
- 28. InnoDB Throughput Number Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • Pages read/second -‐ my favourite number to read, since it is most “raw” cost. • Queries/second doesn’t show how many rows/pages need to be touched. 28 -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Process ID=16185, Main thread ID=5137293312, state: sleeping Number of rows inserted 2093751, updated 0, deleted 0, read 2088735 47462.57 inserts/s, 0.00 updates/s, 0.00 deletes/s, 47462.57 reads/s
- 29. Additional Useful Metrics to Measure Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • I like to look at working set size as percentage of data. • Table growth rate: • http://www.percona.com/blog/2010/12/08/getting-‐ history-‐of-‐table-‐sizes-‐in-‐mysql/ 29
- 30. How slow is a slow query? Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • When optimizing for response: • My first preference is to find slow queries from the application. • Accounting for network for latency, any query can be slow (in aggregate). 30
- 31. How slow is a slow query? (cont.) Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • When optimizing for throughput: • Performance Schema (5.6) SYS statement digest can show query performance for right now. • Alternative: • The slow query log w/a zero second long query time • Capturing “a typical workload” (i.e. 20 minutes) • Aggregate and analyze. 31
- 32. Surrogate Measures Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • Temp tables on disk • Sort merge passes • etc. 32
- 33. Today’s Agenda Introduction From the Database From the Operating System From the Application Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 33 1 2 3 4
- 34. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | CPU Usage • Is 50% CPU usage a good thing? How about 100%? • New versions use more CPUs. • It can be a good thing! 34
- 35. Reason is Mutexes Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • “Internal Locking” that every multi-‐threaded program requires to protect access to resources. • Many mutexes refactored and impact reduced. 35
- 36. Query Cache -‐ An Example Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • Uses a global lock to protect cache • Only one person can insert into the cache at a time. • Now disabled by default. • Enabling it may reduce CPU usage, but look at throughput numbers • May perform worse. 36
- 37. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Mpstat 37 mpstat -P ALL Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011 _x86_64_ (4 CPU) 10:28:04 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 10:28:04 PM all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.99 10:28:04 PM 0 0.01 0.00 0.01 0.01 0.00 0.00 0.00 0.00 99.98 10:28:04 PM 1 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 99.98 10:28:04 PM 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 10:28:04 PM 3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
- 38. Memory Usage [root@ip-10-183-136-169 mysql]# top -bn1 top - 18:24:52 up 1:27, 1 user, load average: 0.73, 0.47, 0.22 Tasks: 92 total, 1 running, 91 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.1%sy, 0.0%ni, 98.0%id, 0.6%wa, 0.0%hi, 0.0%si, 1.2%st Mem: 70201328k total, 7684128k used, 62517200k free, 16356k buffers Swap: 0k total, 0k used, 0k free, 4233840k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND .. 18842 mysql 20 0 43.2g 2.9g 5832 S 0.0 4.3 0:09.86 mysqld .. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • Understand difference between virtual and resident size: 38
- 39. Memory Usage (cont.) Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • After 40G buffer pool fills: 39 [root@ip-10-183-136-169 ~]# top -bn1 | grep 'PID|mysqld' PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 18842 mysql 20 0 43.3g 42g 6336 S 65.4 64.1 20:53.31 mysqld 18611 root 20 0 105m 1464 1172 S 0.0 0.0 0:00.03 mysqld_safe
- 40. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | IO Usage • iostat is the canonical tool. • Not easy to read -‐ 100% utilized may be very healthy. 40 $ iostat -x Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011 avg-cpu: %user %nice %system %iowait %steal %idle 5.68 0.00 0.52 2.03 0.00 91.76 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 27.86 63.53 61.77 132.91 1096.46 1598.40 13.84 0.21 1.06 2.28 44.45 sda1 0.69 33.22 48.54 129.63 773.30 1328.84 11.80 1.39 7.82 2.28 40.57 sda2 27.16 30.32 13.23 3.28 323.13 269.56 35.90 0.55 32.96 3.44 5.68 sdb 39.15 215.16 202.20 169.04 945.80 1073.13 5.44 1.05 2.78 1.64 60.91 sdb1 39.15 215.16 202.20 169.04 945.77 1073.13 5.44 1.05 2.78 1.64 60.91 sdc 8.90 3.63 356.56 51.40 207.01 972.24 2.89 1.04 2.56 1.55 63.30 sdc1 8.90 3.63 356.55 51.40 206.99 972.24 2.89 1.04 2.56 1.55 63.30
- 41. Today’s Agenda Introduction From the Database From the Operating System From the Application Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 41 1 2 3 4
- 42. My favourite statistics Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • Goal for the page. • Something you set. i.e. 50ms. • Number of queries • Total time spend in MySQL • Break down of time per query with query text. 42
- 43. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Why? • Much easier to optimize around a task that is too slow, than a complete system.i.e. • “logins are slow”: Look at the breakdown of time, fix the part that takes the most time for the least value. • “the database is slow”: Look at slow queries, and potentially optimize some queries part of background tasks. 43
- 44. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Rule of Thumb • Per Query Network Latency between 0.1ms and 1ms. • Higher (maybe 3ms) if using multiple AZs etc. 44
- 45. Query Math On Example Table Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • Retrieving all 239 rows in one query: 1.14 ms • With network latency: 2.14ms • Retrieving each row at a time: 0.239ms • With network latency: 296ms 45
- 46. Back to Today’s Question It started with “it depends” Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 46
- 47. Will my database server be able to scale another 10x? • Working Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | set: • If you can project it to stay in memory, you have the most predictable growth curve. • Database size: • Not always an issue in itself, but certain operations become harder. • i.e. mysqldump, time it takes to perform DDL. 47
- 48. Will my database server be able to scale another 10x? (cont.) • Active Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Sessions: • You may start experiencing deadlocks and other soft-‐ errors. • Your application is expected to handle these. • Query Volume: • MySQL 5.7 is up to 600K QPS :) • The server itself scales very well. 48
- 49. Hidden Growth Cliffs Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • Watch out for query patterns that are non scalable over time. e.g. • A time series database that recalculates statistics from the beginning of time. • On day 1 when it is built, it will be fast. • On day 500, it’s ~500x more complex. 49
- 50. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 50