OSMC 2008 | Monitoring MySQL by Geert Vanderkelen

Geert Vanderkelen
MySQL Senior Support Engineer
Sun Microsystems
Monitoring MySQL

Copyright © 2008 Sun Microsystems, Inc.
Agenda
• Short intro into MySQL, the company
• Monitoring MySQL: some examples
• Nagios plugins for MySQL
• MySQL Enterprise Monitor

From MySQL
to Sun
Whoohoo!

What Makes MySQL Special
• OpenSource!
• Commercial and Community Support
• Storage engines
• Simplicity
• Sun Microsystems
• MySQL 5.1 is coming! MySQL Cluster 6.2 is here!

Monitoring MySQL
Basics and Some examples..

Getting Status Information
• SHOW GLOBAL STATUS
> Global keyword is important!
• Snapshot regularly
> mysqladmin -i 20 extended-status >> status.log
• Also available as information_schema.global_status
• For example:
> Com_xxx counter variables
> Questions for Queries/Sec

Index usage
• Find out if you do lots of table scans
mysql> SHOW GLOBAL STATUS
LIKE ‘Handler_read%’;
Handler_read_first 83574685
Handler_read_key 11274900036
Handler_read_next 31376780526
Handler_read_prev 34482622
Handler_read_rnd 265239832
Handler_read_rnd_next 307182727076
((Handler_read_rnd + Handler_read_rnd_next)
/ (∑(Handler_read_%))*100 = 87.8

Temp. Tables Going To Disk?
LIKE ‘Created_tmp%’;
Created_tmp_disk_tables 5032520
Created_tmp_files 71
Created_tmp_tables 13117923
(Created_tmp_disk_tables
/ Created_tmp_tables) *100 = 38.3

Check # Of Processes
• To much long running process: mostly trouble
• SHOW PROCESSLIST or:
mysql> USE information_schema;
mysql> SELECT COUNT(*) FROM processlist
WHERE user <> 'replication' AND
id <> CONNECTION_ID() AND
time > 60 AND
command <> 'Sleep';

Check uptime
• MySQL might crash and restart quickly
• Other checks might miss it
mysql> SELECT DATE_SUB(NOW(),
INTERVAL variable_value SECOND) AS started,
variable_value AS uptime
FROM global_status WHERE variable_name = 'UPTIME';
+---------------------+--------+
| STARTED | UPTIME |
+---------------------+--------+
| 2008-09-08 10:58:24 | 36604 |
+---------------------+--------+

InnoDB Table Size
• Usually auto-extends
• innodb_data_file_path = ibdata1:200M
• Before MySQL 5.1 in table comments!
mysql> SELECT data_free*1024 AS 'InnoDB free space'
FROM tables WHERE table_name = 'innotest';
+-------------------+
| InnoDB free space |
+-------------------+
| 201326592 |
+-------------------+

InnoDB Buffer Pool Hit Rate
• Is the buffer still big enough?
LIKE 'InnoDB_buffer_pool_read%';
+-----------------------------------+------------+
| Variable_name | Value |
+-----------------------------------+------------+
..
| Innodb_buffer_pool_read_requests | 2226519108 |
| Innodb_buffer_pool_reads | 74845 |
+-----------------------------------+------------+
100 * ( 1 - (Innodb_buffer_pool_reads
/ Innodb_buffer_pool_read_requests)) = 99.99%

Checking configuration
• SHOW GLOBAL VARIABLES
> Also information_schema.global_variables
• What version is running?
• Default storage engine still InnoDB?
• Global character set is still UTF-8?
• Logs enabled?
> slow and general query log (online as of 5.1)

Logging Enabled?
• General log
> Log every statement, useful for debugging
> SHOW GLOBAL VARIABLES LIKE 'general_log';
• Slow query log
> Show long running queries, or not using indexes
> SHOW GLOBAL VARIABLES LIKE 'slow_query_log';
• As of MySQL 5.1
> Can be toggled online, and logged to tables
• Should not be running all the time

Monitoring Replication
• On Master
> Check number of binary logs
> Limit number of slaves
• On Slave
> Use SHOW SLAVE STATUS
> Check seconds behind master
> Check if still reading from Master
> etc.

Master And Her Slaves
• Master can handle lots of slave
> But it’s good to set a limit
• SHOW SLAVE HOSTS
> Does list registered hosts, not connected ones
mysql> SHOW SLAVE HOSTS;
+-----------+--------+------+...+-----------+
| Server_id | Host | Port |...| Master_id |
+-----------+--------+------+...+-----------+
| 2 | cent02 | 3306 |...| 1 |
+-----------+--------+------+...+-----------+
1 row in set (0.00 sec)

Number Of Binary Logs
• Regularly causing disk space issues
> Can use PURGE MASTER LOGS
• SHOW BINARY LOGS
mysql> SHOW BINARY LOGS;
+------------------+-----------+
| Log_name | File_size |
+------------------+-----------+
| cent01bin.000001 | 125 |
| cent01bin.000002 | 1021 |
| cent01bin.000004 | 125 |
| cent01bin.000005 | 462 |
+------------------+-----------+

Slave status
mysql> SHOW SLAVE STATUSG
..
Master_Host: 192.168.14.100
Master_User: replication
Master_Port: 3306
..
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
..
Seconds_Behind_Master: 0
..
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:

Seconds Behind Master
• Number of seconds elapsed since timestamp of last
statement received from Master
• Only useful in fast networks
> Slow I/O gives bad approximation
• Use a heartbeat table
> INSERT INTO replication_ping VALUES (NOW())
> Check on slave when table is updated
> Calculate time difference
> Or use mk-heartbeat from Maatkit

Monitoring Cluster
• Still not easy to do
• Using NDB API
• Parse logs
• Using ndb_mgm
shell> ndb_mgm -e "SHOW"
Connected to Management Server at: localhost:1406
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=3 @10.100.9.8 (mysql-5.1.27 ndb-6.3.17, Nodegroup: 0, Master)
id=4 @10.100.9.9 (mysql-5.1.27 ndb-6.3.17, Nodegroup: 0)
[ndb_mgmd(MGM)] 2 node(s)
id=1 @10.100.9.6 (mysql-5.1.27 ndb-6.3.17)
id=2 (not connected, accepting connect from ndbsup-priv-2)
[mysqld(API)] 7 node(s)
id=5 @10.100.9.6 (mysql-5.1.27 ndb-6.3.17)
id=6 (not connected, accepting connect from any host)

Useful Tools
• MySQL client tools and SQL!
• MySQL Enterprise Monitor
> http://www.mysql.com/products/enterprise/
• innotop
> http://innotop.sourceforge.net/
• Maatkit
> http://www.maatkit.org/

Nagios + MySQL
Monitoring MySQL with Nagios

Configuration tips
• Using hostgroups
> Useful for replication setups or Cluster
define hostgroup { hostgroup_name mysql-masters }
define hostgroup { hostgroup_name mysql-slaves }
define host {
use generic-virtualbox
host_name cent02
alias cent02 (VMWare)
address 192.168.14.101
hostgroups virtualboxes,vmware,mysql-slaves
}
define service {
use generic-service
hostgroup_name mysql-slaves
service_description Slave SQL Thread
check_command mysql_repl_sqlthread
}

Configuration tips (2)
• Secure your setup: SSL or SSH
• Don’t use MySQL root user for checks
• Could use NRPE (Nagios add-on)

Some plugins..
• Oli Sennhauser
> http://www.shinguz.ch/MySQL/mysql_monitoring.html

Some plugins.. (2)
• check_mysql_perf
> http://www.consol.com/opensource/nagios/check-mysql-perf/
• Checks various buffers
> hit rates for qcache, keycache
• InnoDB checks
> bufferpool, etc..
• Checks for slow queries
• and more..

Write your own!
• http://nagiosplug.sourceforge.net/developer-guidelines.html
• Quite easy: here a very simple plugin
#!/bin/sh
mysql -e "SELECT NOW()" -NB -h $1 -u$2 -p$3 2>/dev/null
if [ $? != 0 ];
then
echo "Oops!" ; exit 2 # Criticial
fi
exit 0 # OK
shell> ./simple.sh cent02.kemuri.net agent cop
2008-09-08 13:37:09
shell> ./simple.sh cent02.kemuri.net agent cap
Oops!

Write Your Own with Python
def get_uptime():
global OPTIONS
db = MySQLdb.connection(host=OPTIONS['hostname'],
user=OPTIONS['user'],passwd=OPTIONS['password'])
db.query("SHOW GLOBAL STATUS LIKE 'Uptime'")
res = db.store_result()
row = res.fetch_row(1)
return int(row[0][1])
def check_uptime(uptime):
global OPTIONS, ERRORS
line = 'Uptime %ds' % uptime
err = ERRORS['OK']
if uptime < OPTIONS['critical']:
err = ERRORS['CRITICAL']
elif uptime < OPTIONS['warning']:
err = ERRORS['WARNING']
print line
sys.exit(err)
• Email geert@mysql.com for the full version

MySQL Enterprise Monitor
What is it, and how combining it with Nagios?

MySQL Enterprise Monitor (MEM)
• Single, consolidated view
• Auto discovery of MySQL Servers, Replication
Topologies
• Problem Query Detection, Analysis and Tuning – New!
• Customizable rules-based monitoring and alerts
• Identifies problems before they occur
• Reduces risk of downtime
• Easier to scale-out without requiring more DBAs

Advisors/Rules
• Comparable with Nagios plugins
• Possible using Nagios plugins making custom
Advisors for MEM
• Can use SNMP Traps

MEM, Nagios and SNMP
• Using Passive Service Checks
• Handle SNMP traps
> Using traphandle in snmptrapd configuration
• Nagios and MEM host names must match
MEMView
NagiosView

MEM, Nagios and SNMP (cont.)
• Configure snmptrapd to handle MEM traps
# Usually /etc/snmp/snmptrapd.conf
traphandle default /path/to/handle_mysql_memtrap.sh
• Setup Nagios
define service {
use generic-service
service_description MEM SNMP Trap
hostgroup_name mysql-masters,mysql-slaves
register 0
check_command check_no
is_volatile 1
max_check_attempts 1
retry_check_interval 1
normal_check_interval 4
active_checks_enabled 0
passive_checks_enabled 1
}

SNMP Handler Script (part 1)
read host
read ip
while read oid val
do
val=`expr "$val" : '"(.*)"'`
vars="$vars;; $oid: $val"
case "$oid" in
MYSQLTRAP-MIB::advisor.0)
hostname=$val
;;
memstate=$val
;;
message=$val
;;
esac
done
• Reading SNMP trap information

SNMP Handler Script (part 2)
• Translate and send it to Nagios.
state=$STATE_OK
case "$memstate" in
critical)
state=$STATE_CRITICAL
;;
warning|info|success)
state=$STATE_WARNING
;;
esac
datetime=`date +%s`
cmd="[$datetime] PROCESS_SERVICE_CHECK_RESULT;$hostname;
$service;$state;$message"
echo $cmd >> $nagioscmdfile

Some Links..
• MySQL Enterprise
> http://www.mysql.com/products/enterprise/
• Nagios documentation
> http://nagios.sourceforge.net/docs/2_0/passivechecks.html
> http://www.nagios.org/faqs/viewfaq.php?faq_id=29
• Email Geert for the handler script

Monitoring MySQL
Geert Vanderkelen
geert.vanderkelen@sun.com

OSMC 2008 | Monitoring MySQL by Geert Vanderkelen

More Related Content

What's hot (20)

Similar to OSMC 2008 | Monitoring MySQL by Geert Vanderkelen (20)

Recently uploaded (20)

OSMC 2008 | Monitoring MySQL by Geert Vanderkelen