The 5 Minute DBA-DBA Skills for Non-DBA

THE 5 MINUTE DBA
DBA Skills for Non-DBA’s
Matt Yonkovit ,VP (Consulting) – Percona
Percona University, Washington DC
September 12, 2013

DBA’s are hard to find!
Company Confidential December 2010
-2-

Non-DBA’s in charge of DB’s
-3-
• System Administrators
• Web Administrators
• Developers

You get stretched
-4-

WHAT CAN YOU DO IN 5
MINUTES?
The top tips to ensure your database is running smoothly
-5-

BIG DISCLAIMER!
-6-
• Its important to distinguish this up front:
– I am giving you the easy answers, which are not always
the correct answers..
– If you only have 5 minutes to spend on one of these
database topics, you lose a lot of flexibility
– I could spend lots of time talking about everything here,
instead I am focusing on maximizing your benefit.
• This presentation assumes you are running on
Linux/Unix

The 5 Minute DBA-DBA Skills for Non-DBA

The Common Problems Kill You
-8-

THOU SHALL HAVE BACKUPS!
-9-
They can save your bacon!

• Probably the most common problem to get called
on is some sort of slowness
• Keys to success
– Identify what is slow
– When its slow
– Can you repeat it
• Dig in
The Slow Down Of Doom!
-10-

No sense in targeting area’s that do not matter
Target your pain first
-11-

• Google is your friend to research problems
• But don’t tune via Google
• Trust but verify
Know where to look
-12-

• Know the tools out in the community
• Have them ready to jump in and fight fires at a
moments notice
• Tools like:
– Percona Toolkit
– Cacti
– Home grown scripts
– etc
Know thy tools
-13-

• Define what is slow? If its a single page, the
process is drastically different then if its general
slowness.
• Assume General Slowness
– #1 What has changed? A new deployment? More users?
Etc? Sometimes a roll-back of a buggy code release is
the best fix.
• DO NOT ASSUME ITS THE DATABASE!
• Start with the obvious and the quick and move to
more complex
Tuning walk through
-14-

• 5.1 -vs- 5.0 provides improvements
• 5.5 -vs- 5.1 provides improvements
• 5.6 –vs- 5.5 provides improvements
HUGE Differences in Versions!
-15-

• Often stock OS repos are out of date!
• Watch out for old versions installed as
dependencies of off the shelf software
• Just because a hosting provider installed it does not
mean its the latest or best
Where did you get it?
-16-

• Innodb
• Innodb Plugin
• XtraDB
• TokuDB
• MyISAM
• Aria
• Many Many more....
Storage Engines
-17-

• Old Kernel?
• Multi-core servers, scalability getting better (~32 cores + )
• Fit your hot data into memory!
– Typically more memory is better
• IO is a Major bottleneck
– RAID For your databases is a must
– Battery Backed Controller
– RAID 10 typically better then RAID 5
– SSD's for optimal performance
• Network should not be forgotten
• 32bit binary or OS on 64 bit machine
Hardware & OS
-18-

• Next I start by looking at the server (Sar, vmstat,
top, iostat -x)... I typically start low and work my
way into the database
– Too much io, maybe not enough memory, missing
indexes, or bad config
– Cpu maxed out could be lots of users, lots of concurrent
queries, or something else.
– Is the problem even MySQL? Often times we see issues
with non-database software
– top process is something else ( i.e. apache, php, cron
job, etc )
– Network is often overlooked
Review OS Metrics
-19-

COMMON TOOLS
For monitoring the OS/MySQL in a pinch
-20-

• Sar is your friend, every unix/linux server should
setup sar to collect stats in the background
– Sar Can give you historical reports on : CPU, Disk,
Memory, Network, and a lot more.
– Scheduled to collect data via cron
• Iostat is valuable in gauging disk performance
– avwait+svctim = response time (iostat -x)
Systat
-21-

Sar Network Stats
-22-

Sar CPU
-23-

Sar Memory
-24-

Iostat
-25-

VMSTAT
-26-
• Can give you a realtime view of changes on the
system to CPU/Memory

Top
-27-
• Get a current Snapshot of running processes

Percona Toolkit
-28-
• Collection of scripts used for monitoring system and
DB performance
• pt-stalk
• pt-diskstats
• pt-pmp
• Pt-Ioprofiler
• pt-mext

Cacti
-29-
• Great Visual Presentation of Historical Stats
• MySQL, Apache, Sphinx, Memcached, (Entire
Stack)
• Percona Maintains the MySQL Templates
• Can add a lot of value...
Example:
Mysterious Slowdown

OMG The Server is Slow!
-30-

Dig a little deeper
-31-

Oh Wait it Always Happens
-32-

Percona Monitoring Plugins- Cacti!
-33-
Grab them here:
http://www.percona.com/software/percona-monitoring-plugins/

THE CONFIGURATION
Reasonability Check
-34-

MYSQL CHECKS
Looking at the DB
-35-

Configuration Options – Top 10
-36-
General Configuration Options
• query_cache_size (Disable in many versions)
• log_slow_queries
• long_query_time
• max_connections
• expire_logs_days
MyISAM Specific
• key_buffer_size

Configuration Options
-37-
Innodb Specific Options
• innodb_buffer_pool_size
• innodb_flush_log_at_trx_commit
• innodb_log_file_size
Plugin/Xtradb/Mysql 5.5 Innodb Options
• innodb_io_capacity
• innodb_read_ahead
• innodb_write_io_threads
• innodb_read_io_threads

Online Tools to help you set your my.cnf!
-38-
• https://tools.percona.com/wizard

Memory and Swap
-39-
• Disk IO is bad, Avoiding swapping things that should be
in memory to disk ( do not disable swap however )
• SSD’s/Fast Disks
• Hot Data in Memory!
• read_buffer, read_rnd_buffer, sort_buffer, join_buffer
are all allocated per thread so be careful!
• You want temp tables to be built in memory not on disk

Check Current Activity
• Mysql
– “show full processlist”
– “show global status”
– “Show innodb status”
• Mysqladmin
• Slow Query Log
• pt-query-digest
• Mysqldumpslow
• Innotop
• Mytop
• pt-mext
-40-

Show Processlist
-41-
mysql> show processlistG
*************************** 1. row ***************************
Id: 52
User: bench
Host: domU-12-31-39-0C-41-81.compute-1.internal:42315
db: simple_benchmark
Command: Sleep
Time: 0
State:
Info: NULL
*************************** 14. row ***************************
Id: 65
User: bench
Command: Query
Time: 0
State: init
Info: insert into simple_benchmark.simple_comments (lookup_id, comment, mydate) values ( '8519163', 'Comme
*************************** 15. row ***************************
Id: 66
User: bench
Command: Sleep
Time: 0
State:
Info: NULL

Mysqladmin
-42-
• Mysqladmin extended -r -i1 will give you mysql stats at
1 second intervals, change the -i to increase
decrease...
• A few stats of note:

pt-mext
-43-
• Mext can take the output from mysqladmin and put the
status variables side by side, this makes it very
convenient to look for differences

Slow Query Log
-44-
• The Slow Query log is your friend! As mentioned earlier
queries that take over a certain amount of time are
logged here. (Percona Patches for complete picture )
• Summarize the slow query log with percona toolkit, pt-
query-digest or mysqldumpslow if the percona toolkit is
not available
• Look for not only the longest running query, but also the
one with lots of executions and cumulated time

Pt-query-digest output
-45-

pt-query-digest
-46-

Memcached/Redis/Other Caching
-47-
• Review: More memory the better
• Caching frequently used data is an excellent way to
boost performance
• This typically requires changes to your app
• Performance improvement can be dramatic

Query Tuning
-48-
• Review the slow query log, parse the log with pt-
query-digest.
– If I see something here, I will analyze it
• If your still at a loss, try using tcpdump to dump
SQL statements into pt-query-digest and look for
repeated queries. ( Easier then the general query
log if you need sub second slow queries).

Query Tuning
-49-
• Use Explain Plan
• Add Indexes where needed, do not over index
– pt-duplicate-key-checker
• Lots of small queries can sometimes be more
impactful than 1 big one.

Remember
-50-
• The Disclaimer!
• Questions?

The 5 Minute DBA-DBA Skills for Non-DBA

More Related Content

What's hot (19)

Similar to The 5 Minute DBA-DBA Skills for Non-DBA (20)

Recently uploaded (20)

The 5 Minute DBA-DBA Skills for Non-DBA