Inspecting a multi everything linux system (plmce2k14)
0 likes500 views
This document discusses inspecting and troubleshooting a Linux system. It begins by providing background on the speakers and their experience with systems administration, MySQL, and Percona. It then examines various metrics for understanding system performance, including I/O spikes, load averages, buffer and cache usage. Specific tools are demonstrated for analyzing reads, writes, memory usage, and CPU utilization. The goal is to understand the overall state and behavior of the system.
![IOPS
The formula to calculate IOPS is:
IOPS = 1
[average latency in sec + average read/write seek time in dec ]](https://image.slidesharecdn.com/inspectingamulti-everythinglinuxsystemplmce2k14-140410023343-phpapp02/85/Inspecting-a-multi-everything-linux-system-plmce2k14-51-320.jpg)
![Rotation Latency
HDD Spindle [rpm] Average rotational latency [ms]
4,200 7.14
5,400 5.56
7,200 4.17
10,000 3.00
15,000 2.00](https://image.slidesharecdn.com/inspectingamulti-everythinglinuxsystemplmce2k14-140410023343-phpapp02/85/Inspecting-a-multi-everything-linux-system-plmce2k14-53-320.jpg)
![Example
IOPS = 1 / [ 0.002 + (0.0035+0.004)/2 ]
= 173.91
Total Workload IOPS = 4 x 173.91 = 695 (695.64)
Let's assume 80% reads, 20% writes:
Workload system IOPS = (695 x 0.8)
+ ( (695 x 0.2)/4 )
= 590.75 IOPS
RAID : 5 > penalty = 4
# of disks: 4
Spin Speed (RPM): 15k
Average latency : 2.0ms
Random read seek time : 3.50ms
Random write seek time : 4.0ms](https://image.slidesharecdn.com/inspectingamulti-everythinglinuxsystemplmce2k14-140410023343-phpapp02/85/Inspecting-a-multi-everything-linux-system-plmce2k14-56-320.jpg)
![mpstat
[root@x1 percona]# mpstat P ALL 1
Linux 2.6.32358.11.1.el6.x86_64 (x1) 08/22/2013 _x86_64_ (32 CPU)
05:04:24 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
05:04:25 PM all 13.60 0.00 2.86 0.09 0.00 0.22 0.00 0.00 83.23
05:04:25 PM 0 56.12 0.00 11.22 1.02 0.00 6.12 0.00 0.00 25.51
05:04:25 PM 1 13.13 0.00 4.04 0.00 0.00 0.00 0.00 0.00 82.83
05:04:25 PM 2 39.39 0.00 7.07 1.01 0.00 0.00 0.00 0.00 52.53
05:04:25 PM 3 15.15 0.00 3.03 0.00 0.00 0.00 0.00 0.00 81.82
05:04:25 PM 4 36.73 0.00 6.12 0.00 0.00 0.00 0.00 0.00 57.14
05:04:25 PM 5 8.08 0.00 1.01 0.00 0.00 0.00 0.00 0.00 90.91
05:04:25 PM 6 32.99 0.00 6.19 0.00 0.00 0.00 0.00 0.00 60.82
05:04:25 PM 7 3.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 96.00
05:04:25 PM 8 26.00 0.00 6.00 0.00 0.00 0.00 0.00 0.00 68.00
05:04:25 PM 9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
05:04:25 PM 10 21.00 0.00 5.00 1.00 0.00 0.00 0.00 0.00 73.00
05:04:25 PM 11 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
05:04:25 PM 12 15.15 0.00 4.04 0.00 0.00 0.00 0.00 0.00 80.81
05:04:25 PM 13 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
05:04:25 PM 14 9.00 0.00 3.00 0.00 0.00 0.00 0.00 0.00 88.00
05:04:25 PM 15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
05:04:25 PM 16 39.80 0.00 11.22 1.02 0.00 0.00 0.00 0.00 47.96
05:04:25 PM 17 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 96.00
05:04:25 PM 18 12.00 0.00 3.00 0.00 0.00 0.00 0.00 0.00 85.00
05:04:25 PM 19 1.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.99
05:04:25 PM 20 14.14 0.00 3.03 0.00 0.00 0.00 0.00 0.00 82.83
05:04:25 PM 21 3.96 0.00 0.99 0.00 0.00 0.00 0.00 0.00 95.05
05:04:25 PM 22 2.04 0.00 0.00 0.00 0.00 0.00 0.00 0.00 97.96
05:04:25 PM 23 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
05:04:25 PM 24 5.00 0.00 2.00 0.00 0.00 0.00 0.00 0.00 93.00
...](https://image.slidesharecdn.com/inspectingamulti-everythinglinuxsystemplmce2k14-140410023343-phpapp02/85/Inspecting-a-multi-everything-linux-system-plmce2k14-70-320.jpg)
Inspecting a multi everything linux system (plmce2k14)
- 1. Inspecting a multi-everything Linux system Understand your system Frédéric Descamps Original slides and idea by Aurimas Mikalauskas Percona Live MySQL Conference & Expo April, 2014
- 2. AurimasAurimas dev/ops 1999 dba 2002 consultant 2005 Percona 2007 @ Percona: - Architect - Focus: High performance, troubleshooting, full text search, data recovery
- 3. Me (@lefred)Me (@lefred) dev/ops 1998 consultant 2001 Percona 2011 @ Percona: - Senior Consultant - Focus: High performance, high availability, troubleshooting - devops believer
- 4. - MySQL Consulting - Development - Percona Server - Percona Toolkit - Percona Xtrabackup - Percona XtraDB Cluster - Remote DBA - MySQL support - Training Percona
- 5. IT’S JUSTIT’S JUSTa system.a system.
- 7. 10s10s
- 9. 16kb16kb
- 12. 0.0001s0.0001s
- 13. 0.01s0.01s
- 14. 7.2krpm7.2krpm
- 18. IT’S JUSTIT’S JUSTa system.a system.
- 20. MySQL MyISAM Squid MacOS X SAN InnoDB httpd BIG data FreeBSD MongoDB DRBD varnish DATA Linux XFS
- 25. what aboutwhat about UNBUFFERED I/O?UNBUFFERED I/O?
- 26. Writing big file with dd # dd if=/dev/fioa of=/data/opt/testfile.bin bs=4096 count=7864320 ... 32212254720 bytes (32 GB) copied, 70.7952 s, 455 MB/s // strace ... open("/dev/fioa", O_RDONLY) = 3 dup2(3, 0) = 0 close(3) = 0 lseek(0, 0, SEEK_CUR) = 0 open("/data/opt/testfile.bin", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3 dup2(3, 1) = 1 close(3) = 0 read(0, "353H220202163202740260270002163302163003732760"..., 4096) = 4096 write(1, "353H220202163202740260270002163302163003732760"..., 4096) = 4096 read(0, "00000000000000000000000000000000"..., 4096) = 4096 write(1, "00000000000000000000000000000000"..., 4096) = 4096 ... close(0) = 0 close(1) = 0
- 28. Reads kick in. Writes? # vmstat 1 procs memory swap io system cpu r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 0 64979240 6457316 48724 0 0 0 0 726 177 0 0 100 0 0 1 0 0 64736104 6578076 168964 0 0 121456 0 1769 5468 0 2 98 0 0 1 0 0 63733356 7065556 656456 0 0 487424 0 6133 21445 0 7 93 1 0 1 0 0 62660484 7586900 1177756 0 0 521344 12 6051 22913 0 8 92 0 0 1 0 0 61592424 8105460 1696316 0 0 518656 0 6082 23024 0 7 93 0 0 1 0 0 60440380 8665556 2256852 0 0 559932 44 6266 26413 0 7 92 1 0 1 0 0 59119376 9306964 2898172 0 0 641348 0 7015 31229 0 6 93 0 0 1 0 0 58010208 9845716 3436984 0 0 538752 0 6158 25078 0 7 93 0 0 1 0 0 57018712 10327256 3918336 0 0 481536 24 6004 21398 0 7 93 0 0 1 0 0 55895652 10872924 4464236 0 0 545664 0 6280 25376 0 6 93 0 0 3 0 0 54591312 11506264 5097260 0 0 633344 8 6985 31139 0 8 90 1 0 2 0 0 53263628 12151256 5742488 0 0 644480 0 6971 31668 0 6 93 1 0 1 0 0 51964508 12782296 6373724 0 0 631424 0 7030 31187 0 8 91 1 0 1 0 0 51054968 13224664 6815484 0 0 442212 0 7619 27035 0 5 93 2 0 0 1 0 50891056 13304412 6895208 0 0 79644 12 8448 18104 0 2 92 6 0 0 1 0 49882144 13794140 7385144 0 0 489984 229598 9420 31855 0 6 92 2 0 1 0 0 48985468 14228012 7819192 0 0 433920 434756 10934 30660 0 5 93 1 0 1 1 0 48024344 14694492 8285256 0 0 466432 471500 10914 33897 0 12 84 3 0 1 0 0 47064056 15160924 8751692 0 0 466432 467400 10979 33871 0 7 90 2 0 ... many samples skipped ... r b swpd free buff cache si so bi bo in cs us sy id wa st 1 0 0 1940860 37169660 30761252 0 0 455680 459200 14158 27724 0 6 91 2 0 0 1 0 920864 37665532 31257060 0 0 495744 500200 14708 28307 0 6 92 2 0 2 0 0 3304476 35016696 31506636 0 0 248832 250136 10781 14505 0 9 89 1 0 1 0 0 9349772 28971956 31506812 0 0 0 0 3503 294 0 6 94 0 0
- 32. What if file is small? # dd if=/dev/zero of=/data/opt/testfile.bin bs=4096 count=262144 1073741824 bytes (1.1 GB) copied, 0.957317 s, 1.1 GB/s # vmstat 1 procs memory swap io system cpu r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 0 64976544 6458268 49352 0 0 0 0 701 178 0 0 100 0 0 0 0 0 64976584 6458268 49356 0 0 0 0 488 160 0 0 100 0 0 1 0 0 64491596 6458268 534368 0 0 0 0 4022 191 0 4 96 0 0 0 0 0 63926396 6458276 1097560 0 0 0 12 3315 160 0 5 94 0 0 # ptdiskstats #ts device wr_s wr_avkb wr_mb_s wr_mrg wr_cnc wr_rt busy in_prg {1} sda5 0.0 0.0 0.0 0% 0.0 0.0 0% 0 {1} sda5 0.0 0.0 0.0 0% 0.0 0.0 0% 0 {1} sda5 0.0 0.0 0.0 0% 0.0 0.0 0% 0 {1} sda5 0.0 0.0 0.0 0% 0.0 0.0 0% 0 {1} sda5 0.0 0.0 0.0 0% 0.0 0.0 0% 0
- 34. CACHED Reads are other kernel caches and structures OS page cache
- 36. load average: ? How’s our load test?
- 40. Quiz There’s a server: - it has 2 6-core CPUs - Hyperthreading is off - 32GB of RAM - 8 disks in RAID10 What load average would indicate that server is saturated? a) 4.0 b) 8.0 c) 12.0 d) 16.0 e) 20.0
- 41. Answer Either of the them: a) 4.0 b) 8.0 c) 12.0 d) 16.0 e) 20.0 - 4 processes constantly writing - 8 processes constantly reading - 12 processes constantly computing - 12 computing & 4 writing processes - 12 computing & 8 reading processes
- 43. Latency
- 45. Random -vs- Sequential Random -vs- Sequential Read this post from Chris Buckel (@flashdba) : http://flashdba.com/2013/04/15/understandingiorandomvssequential/
- 48. Sequential IO (BIG random reads) # ptdiskstats wr_s wr_avkb wr_mb_s wr_mrg 13149.5 4.7 60.0 0% 13261.9 4.7 60.5 0% 13324.2 4.7 60.8 0% # ptdiskstats wr_s wr_avkb wr_mb_s wr_mrg 1031.0 456.0 459.1 0% 1027.4 456.0 457.5 0% 1027.4 456.0 457.5 0% # ptdiskstats wr_s wr_avkb wr_mb_s wr_mrg 212.0 20.7 4.3 40% 54.1 55.9 3.0 79% 221.2 19.2 4.2 37% 39.3 25.4 1.0 82%
- 50. IOPSIOPShow manyhow many can your hardware deliver?can your hardware deliver?
- 51. IOPS The formula to calculate IOPS is: IOPS = 1 [average latency in sec + average read/write seek time in dec ]
- 52. RAID Penalty RAID configuration Penalty RAID 0 1 RAID 1 2 RAID 10 2 RAID 5 4 RAID 6 6
- 53. Rotation Latency HDD Spindle [rpm] Average rotational latency [ms] 4,200 7.14 5,400 5.56 7,200 4.17 10,000 3.00 15,000 2.00
- 54. Workload system IOPS formula The formula to calculate workload system IOPS (with the RAID) is: 1(Total Workload IOPS * Percentage of workload read operations) + ( (Total Workload IOPS * Percentage of workload write operations) / RAID IOPS Penalty )
- 55. Example Disk : SEAGATE ST3300655LC (279GB) RAID : 5 # of disks: 4 Spin Speed (RPM): 15k Average latency : 2.0ms Random read seek time : 3.50ms Random write seek time : 4.0ms
- 56. Example IOPS = 1 / [ 0.002 + (0.0035+0.004)/2 ] = 173.91 Total Workload IOPS = 4 x 173.91 = 695 (695.64) Let's assume 80% reads, 20% writes: Workload system IOPS = (695 x 0.8) + ( (695 x 0.2)/4 ) = 590.75 IOPS RAID : 5 > penalty = 4 # of disks: 4 Spin Speed (RPM): 15k Average latency : 2.0ms Random read seek time : 3.50ms Random write seek time : 4.0ms
- 57. Mirroring Striping across 4 stripes Mirroring Mirroring Mirroring BufferingBuffering 8 disk RAID 10
- 65. pt-diskstats wget percona.com/get/pt-diskstats or visit: http://www.percona.com/software/percona-toolkit
- 66. FLASHFLASH a word ona word on
- 67. Reads on Samsung 830 256GB
- 68. Flash
- 70. mpstat [root@x1 percona]# mpstat P ALL 1 Linux 2.6.32358.11.1.el6.x86_64 (x1) 08/22/2013 _x86_64_ (32 CPU) 05:04:24 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 05:04:25 PM all 13.60 0.00 2.86 0.09 0.00 0.22 0.00 0.00 83.23 05:04:25 PM 0 56.12 0.00 11.22 1.02 0.00 6.12 0.00 0.00 25.51 05:04:25 PM 1 13.13 0.00 4.04 0.00 0.00 0.00 0.00 0.00 82.83 05:04:25 PM 2 39.39 0.00 7.07 1.01 0.00 0.00 0.00 0.00 52.53 05:04:25 PM 3 15.15 0.00 3.03 0.00 0.00 0.00 0.00 0.00 81.82 05:04:25 PM 4 36.73 0.00 6.12 0.00 0.00 0.00 0.00 0.00 57.14 05:04:25 PM 5 8.08 0.00 1.01 0.00 0.00 0.00 0.00 0.00 90.91 05:04:25 PM 6 32.99 0.00 6.19 0.00 0.00 0.00 0.00 0.00 60.82 05:04:25 PM 7 3.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 96.00 05:04:25 PM 8 26.00 0.00 6.00 0.00 0.00 0.00 0.00 0.00 68.00 05:04:25 PM 9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 05:04:25 PM 10 21.00 0.00 5.00 1.00 0.00 0.00 0.00 0.00 73.00 05:04:25 PM 11 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 05:04:25 PM 12 15.15 0.00 4.04 0.00 0.00 0.00 0.00 0.00 80.81 05:04:25 PM 13 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 05:04:25 PM 14 9.00 0.00 3.00 0.00 0.00 0.00 0.00 0.00 88.00 05:04:25 PM 15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 05:04:25 PM 16 39.80 0.00 11.22 1.02 0.00 0.00 0.00 0.00 47.96 05:04:25 PM 17 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 96.00 05:04:25 PM 18 12.00 0.00 3.00 0.00 0.00 0.00 0.00 0.00 85.00 05:04:25 PM 19 1.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.99 05:04:25 PM 20 14.14 0.00 3.03 0.00 0.00 0.00 0.00 0.00 82.83 05:04:25 PM 21 3.96 0.00 0.99 0.00 0.00 0.00 0.00 0.00 95.05 05:04:25 PM 22 2.04 0.00 0.00 0.00 0.00 0.00 0.00 0.00 97.96 05:04:25 PM 23 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 05:04:25 PM 24 5.00 0.00 2.00 0.00 0.00 0.00 0.00 0.00 93.00 ...
- 71. UNDERSTANDUNDERSTANDhow your application uses CPUhow your application uses CPU
- 82. Load indices ● On busy systems, the processes queue length is a better way to evaluate the total load. ● For example, on two strongly charged host where the use of the processor continuously reach a maximum of 100%, the system with the higher amount of processes in standby can be considered as more charged.
- 83. To reCAP 1. VMSTAT 1 2. MPSTAT -P ALL 1 3. PT-DISKSTATS
- 84. To reCAP 1. VMSTAT 1 2. MPSTAT -P ALL 1 3. PT-DISKSTATS
- 85. IT’S JUSTIT’S JUSTa system.a system.
- 86. Even more tools ?
- 87. ??
- 88. Aurimas & lered, percona.comAurimas & lered, percona.com