Fosdem managing my sql with percona toolkit

Managing MySQL with
Percona Toolkit

FOSDEM MySQL & Friends devroom 2012

Who am I ?
Frédéric Descamps
@lefred
http://about.me/lefred
Percona Consultant
Managing mysql since 3.23 (as far as I
remember)
devops believer

www.percona.com

Agenda

● What is Percona
Toolkit ?
● What tools are
included and what
can they do ?
● Quick intro to several
tools
● Resources

www.percona.com

Percona Toolkit

● Origin: Maatkit and Aspersa (both projects
started by Baron Schwartz with Daniel Nichter's
contribution)
● Indispensable tools for MySQL administrators
● Opensource GNU GPL v2
● https://launchpad.net/percona-toolkit
● http://www.percona.com/software/percona-toolkit/

www.percona.com

Other Toolkits

www.percona.com

Percona Toolkit ;-)

www.percona.com

More seriously:

www.percona.com

Installation
●
Requirements:
➢
Perl, DBI, DBD::mysql, Term::ReadKey
●
Download tarball, RPM or DEB packages:
➢
wget percona.com/get/perconatoolkit.rpm
➢
It's in Percona repository
●
Download individual tools:
➢
wget percona.com/get/ptdiskstats

www.percona.com

Know your environment:
Hardware & Operating System

www.percona.com 9

Know your environment
(Hardware & OS)
We need to find the answers to these questions:
● On what kind of machine runs my db ? Physical/Virtual ?
● How much memory is used/free ? Does it swap ? Can I add more ?
● How many CPUs and CPU cores ? Speed ?
● Is this a 64bit machine ?
● What kind of RAID controller ? Volumes ? Disks ?
● What about network interfaces ? Connections ? Processes ?
● What's the Operating System ? Kernel ? Release ?
● Which IO scheduler is in use ?
● On which filesystem is the data stored ?
● Do we have free space ?

www.percona.com 10

Summary !
Name: ptsummary
Function: summarize system information in
a nice way
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-summary.html

www.percona.com 11

Taking inventory
$ wget percona.com/get/ptsummary
$ chmod +x ptsummary
$ ./ptsummary
# Percona Toolkit System Summary Report ######################
        Date | 20120202 12:25:18 UTC (local TZ: EET +0200)
    Hostname | db01
      Uptime | 42 days, 16:14, 1 user, load average: 1.15, 1.22, 1.08
      System | HP; ProLiant DL580 G7; vNot Specified (Rack Mount Chassis)
Service Tag | FOSDEM2012
    Platform | Linux
     Release | CentOS release 6.1 (Final)
      Kernel | 2.6.32131.21.1.el6.x86_64
Architecture | CPU = 64bit, OS = 64bit
   Threading | NPTL 2.12
    Compiler | GNU CC version 4.4.5 20110214 (Red Hat 4.4.56).
     SELinux | Disabled
Virtualized | No virtualization detected
# Processor ##################################################
  Processors | physical = 4, cores = 40, virtual = 80, hyperthreading = yes
      Speeds | 80x2397.393
      Models | 80xIntel(R) Xeon(R) CPU E7 4870 @ 2.40GHz

[ Further output omitted ]

www.percona.com 12

Know your environment:
MySQL

www.percona.com 13

Know your environment (MySQL)
We need to find the answers to these questions:
● Version ? Build ?
● How many database ? Where is the data directory ?
● What about replication ? How many replicas ?
● Do you use some Key XtraDB features ?
● What are the Key InnoDB settings ?
● Which storage engines are used ?
● What types of data fields are used ? Index types ?
● Do we use foreign keys ? Views ? Triggers ?
● Do we use Full-text indexes ? Partitioning ?
● What about Privileges ?

www.percona.com 14

Summary... again !
Name: ptmysqlsummary
Function: Summarize MySQL information in
a nice way
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-mysql-summary.html

www.percona.com 15

Taking inventory (cont.)
$ wget percona.com/get/ptmysqlsummary
$ chmod +x ptmysqlsummary
$ ./ptmysqlsummary
# Percona Toolkit MySQL Summary Report #######################
         System time | 20120202 12:24:37 UTC (local TZ: EET +0200)
# Instances ##################################################
  Port  Data Directory             Socket
  ===== ========================== ======
        /mysql
# Report On Port 3306 ########################################
                User | root@localhost
                Time | 20120202 14:24:38 (EET)
            Hostname | db01
             Version | 5.5.1855log Percona
            Built On | Linux x86_64
             Started | 20111226 21:09 (up 37+17:15:35)
           Databases | 6
             Datadir | /mysql/
           Processes | 10 connected, 4 running
         Replication | Is a slave, has 1 slaves connected
             Pidfile | /mysql/mysqld.pid (exists)

[ Further output omitted ]

www.percona.com 16

Master ? Slave ?... both ?
Name: ptslavefind
Function: find and print replication hierarchy
tree of MySQL slaves.
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-slave-find.html

www.percona.com 17

Inventory of Replicas
$ ./ptslavefind localhost
localhost
Version         5.5.1855log
Server ID       11
Uptime          37+17:45:50 (started 20111226T21:09:03)
Replication     Is a slave, has 1 slaves connected, is not read_only
Filters
Binary logging  MIXED
Slave status    0 seconds behind, running, no errors
Slave mode      STRICT
Autoincrement  increment 2, offset 1
InnoDB version  1.1.8rel23.0
+ 10.0.0.17
   Version         5.5.1855log
   Server ID       12
   Uptime          27+21:02:35 (started 20120105T17:52:18)
   Replication     Is a slave, has 1 slaves connected, is not read_only
   Filters
   Binary logging  MIXED
   Slave status    0 seconds behind, running, no errors
   Slave mode      STRICT
   Autoincrement  increment 2, offset 2
   InnoDB version  1.1.8rel23.0

www.percona.com 18

Where's My Disk I/O Going ?
● Use ptdiskstats as an improved iostat
➢ Interactively controllable, slice-and-dice
➢ Doesn't lump reads and writes together

➢ Meaningful, correct columns

● Use ptioprofile to measure & time your I/O
➢ Which files is MySQL using ? How much times does it
spend reading, writing, syncing, etc? How many
reads/writes? How large are they ?

pt-ioprofile can be dangerous in production
www.percona.com 19

Disk I/O
Name: ptdiskstats
Function: Aggregate and summarize
/proc/diskstats
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-diskstats.html

www.percona.com 20

Disk I/O
# ./ptdiskstats d sda$
  #ts device rd_mb_s rd_cnc   rd_rt wr_mb_s wr_cnc   wr_rt busy in_prg
  {1} sda        0.0    0.0     0.0     0.4    0.0     0.1   1%      0
  {1} sda        0.0    0.0     0.0     0.2    0.0     0.0   0%      0
  {1} sda        0.0    0.0     0.0     0.3    0.0     0.1   1%      0
  {1} sda        0.0    0.0     0.0     0.2    0.0     0.1   0%      0
Enter a column pattern:
  #ts device    rd_s rd_avkb rd_mb_s rd_mrg rd_cnc   rd_rt    wr_s wr_avkb wr_mb_s wr_mrg wr_cnc   wr_rt busy in_prg
  {5} sda        0.0     0.0     0.0     0%    0.0     0.0     68.3     7.2     0.2     1%    0.0     0.1   1%      0
  {1} sda        0.0     0.0     0.0     0%    0.0     0.0     39.6     7.1     0.1     0%    0.0     0.2   1%      0
  {1} sda        0.0     0.0     0.0     0%    0.0     0.0    118.2    18.1     1.0    58%    0.0     0.2   0%      0
  {1} sda        0.0     0.0     0.0     0%    0.0     0.0    114.7     7.8     0.4     2%    0.0     0.0   0%      0

www.percona.com 21

I/O Profile
Name: ptioprofile
Function: Watch process IO and print a table
of file and I/O activity.
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-ioprofile.html

www.percona.com 22

I/O profile
# ./ptioprofile
Sat Feb  4 13:44:08 CET 2012
Tracing process ID 24179
     total       read      write       open      close      lseek filename
  0.007547   0.000583   0.004245   0.002181   0.000268   0.000270 /tmp/#sql_5e73_1.MYI
  0.000686   0.000000   0.000000   0.000420   0.000266   0.000000 /tmp/#sql_5e73_1.MYD
  0.000065   0.000000   0.000024   0.000029   0.000012   0.000000 /var/lib/mysql/fred/db.opt
  0.000055   0.000000   0.000000   0.000055   0.000000   0.000000 /var/lib/mysql/mysql/db.opt
  0.000046   0.000000   0.000000   0.000046   0.000000   0.000000 /var/lib/mysql/test/db.opt

www.percona.com 23

Be intimate with your database

www.percona.com 24

Let's try to find the answer to these
indiscreet questions:

● How are the indexes used ?
● Are there duplicate keys ?
● Which queries are eating most of the
resources ?

www.percona.com 25

Analyze Schema & Queries
● Duplicate/redundant indexes or foreign keys?
➢ ptduplicatekeychecker can tell you that
● Which indexes are unused?
➢ ptindexusage can tell you that
➢ It can also show which queries have unstable
plans, which indexes have alternatives that are
never chosen, and lots more
● Bad SQL?
➢ ptqueryadvisor to the rescue

www.percona.com 26

Analyze & Profile Your Workload
● Use ptquerydigest to analyze the “slow
query log” and show a profile of the workload
● It does the Right Thing by default
● Mostly used with slow query logs & tcpdumps

www.percona.com 27

The Report
● Each class of query is ranked by execution time
● There is a report for each class of query
● It's easy to use
● It's a great way to start optimizing queries

www.percona.com 28

Overview Report
# 13903.5s user time, 39.9s system time, 1.78G rss, 1.92G vsz
# Current date: Mon Jan 30 20:36:26 2012
# Hostname: hpdb01
# Files: /var/lib/mysql/db01slow.log
# Overall: 32.98M total, 1.32k unique, 1.73k QPS, 2.06x concurrency ______
# Time range: 20120130 11:08:22 to 16:25:40
# Attribute          total     min     max     avg     95%  stddev  median
# ============     ======= ======= ======= ======= ======= ======= =======
# Exec time         39309s     1us    161s     1ms   515us   109ms    30us
# Lock time          4471s       0    161s   135us    89us    90ms       0
# Rows sent        118.72M       0 1010.29k    3.77    1.96  963.89       0
# Rows examine      15.54G       0   3.51M  505.92    1.96  11.61k       0
# Rows affecte       1.64M       0   2.85k    0.05       0    1.82       0
# Rows read        118.72M       0 1010.29k    3.77    1.96  963.89       0
# Bytes sent        52.40G       0  45.18M   1.67k   5.20k  20.30k  537.02
# Merge passes         128       0       1    0.00       0    0.00       0
# Tmp tables       957.76k       0       4    0.03       0    0.17       0
# Tmp disk tbl     297.92k       0       2    0.01       0    0.10       0
# Tmp tbl size       5.21T       0   2.22G 169.59k       0   1.78M       0
# Query size         8.52G       6 130.40k  277.38  833.10  709.99  174.84
# InnoDB:
# IO r bytes       672.00k       0  32.00k    0.06       0   35.90       0
# IO r ops              42       0       2    0.00       0    0.00       0
# IO r wait           19ms       0     2ms       0       0     1us       0
# pages distin     329.98M       0  32.08k   28.18   42.48  307.62    4.96
# queue wait             0       0       0       0       0       0       0
# rec lock wai       3402s       0    161s   277us       0   147ms       0
# Boolean:
# Filesort       2% yes,  97% no
# Filesort on    0% yes,  99% no
# Full join      0% yes,  99% no
# Full scan      2% yes,  97% no
# QC Hit        55% yes,  44% no
# Tmp table      2% yes,  97% no
# Tmp table on   0% yes,  99% no

www.percona.com 29

Profile Report

# Profile
# Rank Query ID           Response time   Calls    R/Call  Apdx V/M   Item
# ==== ================== =============== ======== ======= ==== ===== ====
#    1 0x844F3DA35A4A426D 8695.6994 22.1%    41470  0.2097 1.00  0.12 SELECT catalog
#    2 0xD566217EB8EB1A27 4098.9154 10.4%    48740  0.0841 1.00  0.11 SELECT geoip
#    3 0xB0A48B410B3CC151 2684.9671  6.8%       55 48.8176 0.00 34.18 UPDATE SELECT sales_
#    4 0x60CDD6CAE6B487BC 2541.5271  6.5%      352  7.2202 0.20  4.01 SELECT catalog catalog_category_product
#    5 0xD9DED53EF74C877F 1285.6118  3.3%  5530829  0.0002 1.00  0.00 SELECT op_image
#    6 0x1F3D58843008ACEC 1249.1639  3.2%      182  6.8635 0.12  1.65 SELECT catalog_product
#    7 0x50175ED6D1B1C8B1 1188.1541  3.0%   153964  0.0077 1.00  0.00 SELECT enterprise
#    8 0x70E2BE84868444A4 1035.3475  2.6%     1786  0.5797 0.98  0.16 SELECT sales_order
#    9 0x125DFE84FFA50675  891.3327  2.3%      382  2.3333 0.97 11... INSERT UPDATE product
#   10 0xBB968DD9BF27E8E5  797.3765  2.0%   153964  0.0052 1.00  0.00 SELECT enterprise

www.percona.com 30

Per Query Report
# Query 1: 2.25 QPS, 0.47x concurrency, ID 0x844F3DA35A4A426D at byte 4779840203
# Attribute    pct   total     min     max     avg     95%  stddev  median
# ============ === ======= ======= ======= ======= ======= ======= =======
# Count          0   41470
# Exec time     22   8696s    97ms      3s   210ms   455ms   159ms   155ms
# Lock time      0      7s   107us    50ms   177us   247us   272us   159us
# Rows sent      1   1.60M      26      50   40.45   49.17   10.74   46.83
# Rows examine  16   2.63G  66.43k  66.60k  66.49k  65.68k       0  65.68k
# Rows affecte   0       0       0       0       0       0       0       0
# Rows read      1   1.60M      26      50   40.45   49.17   10.74   46.83
# Bytes sent     0 200.43M   3.39k   6.14k   4.95k   6.01k   1.08k   5.45k
# Merge passes   0       0       0       0       0       0       0       0
# Tmp tables     4  40.50k       1       1       1       1       0       1
# Tmp disk tbl   0       0       0       0       0       0       0       0
# Tmp tbl size  12 647.83G  16.00M  16.00M  16.00M  16.00M       0  16.00M
# Query size     0  71.78M   1.77k   1.77k   1.77k   1.77k       0   1.77k
# InnoDB:
# IO r bytes     0       0       0       0       0       0       0       0
# IO r ops       0       0       0       0       0       0       0       0
# IO r wait      0       0       0       0       0       0       0       0
# pages distin  47 158.13M   3.90k   3.92k   3.90k   3.88k    0.00   3.88k
# queue wait     0       0       0       0       0       0       0       0
# rec lock wai   0       0       0       0       0       0       0       0
# Boolean:
# Filesort     100% yes,   0% no
# Full scan    100% yes,   0% no
# Tmp table    100% yes,   0% no
# String:
# Databases    prod
# Hosts
# InnoDB trxID 3CB60AED (1/0%), 3CB60B3E (1/0%)... 41468 more
# Last errno   0
# Users        prod
# Query_time distribution
#   1us
#  10us
# 100us
#   1ms
#  10ms
# 100ms  ################################################################
#    1s  #
#  10s+

www.percona.com 31

Maintain your environment

www.percona.com 32

Log & Analyze Errors ?
● Use ptdeadlocklogger to capture
deadlocks
● Use ptfkerrorlogger for foreign key
errors
● The tools give you the possibility to log the
results to a database table for analysis

www.percona.com 33

Monitor dead locks
Name: ptdeadlocklogger
Function: extract and log MySQL deadlock
information
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-deadlock-logger.html
● SHOW ENGINE INNODB STATUS reports only the most
recent deadlock
● Information in status output lacks details about user, client
host, other locks in the deadlock cycle.
● ptdeadlocklogger formats the needed information
nicely and saves it
www.percona.com 34

Monitor Foreign Key Errors
Name: ptfkerrorlogger
Function: extract and log MySQL foreign key
errors.
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-fk-error-logger.html
● Like ptdeadlocklogger, this tool collects more useful
information, formats it nicely and saves it.

www.percona.com 35

Non-Blocking ALTER TABLE
● Use ptonlineschemachange to alter
tables
➢ It makes a “shadow copy” and swaps them

➢ Extremely useful for large, long-running

ALTER
➢ Facebook uses the same technique

www.percona.com 36

Validate Upgrades
● Upgrades are a leading cause of downtime
➢ Are any queries using different indexes ? Is QEP
different ?
➢ Are there new errors or warnings ?

➢ Do queries return the same results ?

➢ Are there significant response time differences ?

● Validate the results before you upgrade!
➢ Capture a snapshot of the data
➢ Capture a log of queries

➢ Compare the queries on two test servers running the

old and the new version
www.percona.com 37

Validate updates
Name: ptupgrade
Function: Execute queries on multiple
servers and check for differences.
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-upgrade.html

www.percona.com 38

pt-upgrade
# Query 2: ID 0xE2C8DC0286B7F44E at byte 0 _______________________________
# Found 23281 differences in 58207 samples:
#   checksums       0
#   column counts   0
#   column types    0
#   query times     23281
#   row counts      0
#   warning counts  0
#   warning levels  0
#   warnings        0
#            AF002020:3306 AF001885:3306
# Errors                 0             0
# Warnings               0             0
# Query_time
#   sum                34s           70s
#   min              395us         342us
#   max              218ms         308ms
#   avg              587us           1ms
#   pct_95           725us           3ms
#   stddev             2ms           4ms
#   median           541us         690us
# row_count
#   sum              1.75k         1.75k
#   min                  0             0
#   max                  1             1
#   avg               0.03          0.03
#   pct_95               0             0
#   stddev            0.17          0.17
#   median               0             0

www.percona.com 39

Tools for Replication
● Verify replication integrity (pttablechecksum)
● Repair out-of-sync replicas (pttablesync)
● Measure delay accurately (ptheartbeat)
● Deliberately delay replication (ptslavedelay)
● Watch & restart replication (ptslaverestart)

www.percona.com 40

Verify replication Integrity
Name: pttablechecksum
Function: perform an online replication
consistency check, or checksum MySQL
tables efficiently on one or many servers.
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-table-checksum.html
● Use it routinely (mandatory for 95% of MySQL users)
● It used to be complicated; no more. Just run it!
● Put it in a weekly crontab
● Repair differences with pttablesync
www.percona.com 41

I feel bad...

www.percona.com 42

Find problems
● What do you advise me doctor ?
● Get the symptoms when it hurts !

www.percona.com 43

Get the symptoms when it hurts !
To perform this action, do a check-up
when you feel bad:

● ptstalk
● ptcollect
● ptsift

www.percona.com 44

Does it hurt ?
Name: ptstalk
Function: wait for a condition to occur then
begin collecting data.
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-stalk.html

www.percona.com 45

We need your health parameters
Name: ptcollect
Function: collect information from a server
for some period of time.
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-collect.html

www.percona.com 46

Let me check...
Name: ptsift
Function: browses files created by pt-collect.
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-collect.html

www.percona.com 47

Very useful
Name: ptmext
Function: Look at many samples of MySQL
SHOW GLOBAL STATUS side-by-side
Url: http://www.percona.com/doc/percona-toolkit/pt-mext.html

● Default STATUS shows counter since the MySQL
instances started.
● It is more helpful to see a delta of recent activity, to check
if a change you applied made a difference.

www.percona.com 48

pt-mext
ptmext r mysqladmin ext i10 c5
All values are interesting but you should pay even more attention to:
● Created_tmp_*
● Handler_read_*
● Innodb_buffer_pool_read*
● Innodb_os_log_written
● Key_read_requests
● Key_reads
● Qchache*
● Select_full_*join
● Select_range
● Select_scan
● Table_locks_waited
● Threads_running

www.percona.com 49

Stop Bad Queries
● Use ptkill to watch for long-running queries
and optionally kill them
● Prevent long-running idle transactions too.

www.percona.com 50

Percona Toolkit's Future
● Rapid development & improvement of tools
● Fully supported via Percona Support contract
● Upcoming:
➢ ptquerydigest “query reviews” improved

➢ ptstalk “magical fault detection algorithm”

➢ new ptdisktats using Perl

➢ Lots more goodies

•https://blueprints.launchpad.net/percona-toolkit

www.percona.com 51

Thank you for your participation to
FOSDEM MySQL & Friends devroom

lefred@percona.com
@lefred

We're Hiring! www.percona.com/about-us/careers/

Fosdem managing my sql with percona toolkit

More Related Content

What's hot (20)

Viewers also liked (8)

Similar to Fosdem managing my sql with percona toolkit (20)

More from Frederic Descamps (20)

Recently uploaded (20)

Fosdem managing my sql with percona toolkit