Database Hardware Benchmarking

Database Hardware Benchmarking

Greg Smith

2ndQuadrant US

03/25/2009

Greg Smith Database Hardware Benchmarking

About this presentation

The master source for these slides is
http://www.2ndquadrant.com/talks

Slides are released under the Creative Commons Attribution
3.0 United States License
http://creativecommons.org/licenses/by/3.0/us


Why should you always benchmark your hardware?

Many useful tests will only run when the server isn’t being
used yet
Software stacks are complicated
Spending money on upgrades only helps if you upgrade the
right thing usefully
Vendors lie


Systematic Benchmarking

Memory
CPU
Disk
Database server
Application


Memory Tests

DOS: memtest86+ (also on many Linux CDs, like the Ubuntu
installer)
Windows: SiSoftware Sandra
UNIX: sysbench, stream


sysbench memory write and CPU tests

sysbench --test=memory --memory-oper=write
--memory-block-size=64MB --memory-total-size=1024MB
run

sysbench --test=cpu run


stream memory read test

wget http://www.cs.virginia.edu/stream/FTP/Code/stream.c
gcc -O stream.c -o stream
./stream


Memory test comparisons

Processor Frequency RAM Speed memtest86+ sysbench stream

T7200 2.0GHz DDR2/667 2563 1360 2818

Q6600 2.4GHz DDR2/800 3352 2009 4386

T9400 2.53GHz DDR3/1066 3743 2226 ?

Xeon X5450 3.00GHz DDR2/1333 3575 2487 ?

Xeon E5520 2.27GHz DDR3/1333 9252 4230 ?

Xeon X5550 2.67GHz DDR3/1333 10885 5493 ?

i7 860 2.8GHz DDR3/1600 13716 5559 10033


Advanced Clustering STREAM Benchmarking

http://www.advancedclustering.com/company-blog/


Oracle Calling Center OLTP Benchmark

http://it.anandtech.com/IT/showdoc.aspx?i=3769&p=4


Sources for slow memory results

Single channel RAM/slot mistakes
Incorrect SPD/timing/voltage
Bad RAM/CPU multiplier combination
Poor quality RAM


PostgreSQL and the CPU

PostgreSQL uses only a single CPU per query
Queries executing against cached data will bottleneck on CPU
COPY is CPU intensive


CPU Tests

Windows: SiSoftware Sandra
UNIX: sysbench CPU test
Custom test with timing and generate series
pgbench select-only on small database
Vary pgbench client count to test single or multiple CPUs


sysbench CPU comparisons

Processor Frequency sysbench CPU seconds

Intel Q6600 2.40GHz 19.6

Intel T7200 2.0GHz 18.9

Intel T9400 2.53GHz 12.0

Intel Xeon X5450 3.00GHz 8.4

Intel Xeon E5520 2.27GHz 11.7

Intel Xeon X5550 2.67GHz 8.7

Intel i7 860 2.8GHz 9.0


Sources for slow CPU results

Slow memory
Power management throttling
Linux: /proc/cpuinfo shows 1000MHz suggests you need to
adjust the CPUFreq Governor to “performance”


Disk Tests

Sequential write: INSERT, COPY FROM (when not CPU
limited)
Sequential read: SELECT * FROM and similar table
sequential scans
Seeks: SELECT using index, UPDATE
Commit fsync rate: INSERT, UPDATE


dd test

Compute 2X the size of your RAM in 8KB blocks
blocks = 250,000 * gigabytes of RAM

time sh -c "dd if=/dev/zero of=bigfile bs=8k count=X &&
sync"
time dd if=bigfile of=/dev/null bs=8k

Watch vmstat and/or iostat during disk tests
vmstat’s bi and bo will match current read/write rate
Note the CPU percentage required to reach the peak rate


bonnie++

./bonnie++
bon csv2html

Ignore the per-character and create results, look at the block
output/input ones
Random Seeks:
The test runs SeekProcCount processes (default 3) in parallel,
doing a total of 8000 random seek reads to locations in the
ﬁle. In 10% of cases, the block read is changed and written
back.


bonnie++ ZCAV

./zcav -f/dev/sda > t500

Must get a recent version of bonnie++ for ZCAV to scale
properly for TB drives (1.03e works)
ZCAV on experimental branch (1.96) gave useless results for
me
Download somewhat broken gnuplot script sample and typical
results from:

http://www.coker.com.au/bonnie++/zcav/results.html


Improved bonnie++ ZCAV gnuplot script

unset autoscale x
set autoscale xmax
unset autoscale y
set autoscale ymax
set xlabel "Position GB"
set ylabel "MB/s"
set key right bottom
set terminal png
set output "zcav.png"
plot "raid0" title "7200RPM RAID 0 3 Spindles",
"single" title "7200RPM Single Drive"


bonnie++ ZCAV: Laptop 7200RPM Disk


Scaling of 3-Disk RAID0 with 7200RPM SATA Disks


3-Disk Short-Stroked RAID0, larger 7200RPM SATA


Read seeks/second - sysbench

THREADS=1
GB=10
MODE=rndrd
OPTIONS="--test=fileio --num-threads=$THREADS
--file-block-size=8K --file-test-mode=$MODE
--file-num=$GB --file-total-size=${GB}G
--file-fsync-freq=0 --file-fsync-end=no"
sysbench prepare $OPTIONS
sysbench run --max-time=60 $OPTIONS
sysbench cleanup $OPTIONS


Sample sysbench random read results

Read 78.125Mb Written 0b
Total transferred 78.125Mb (1.0059Mb/sec)
128.75 Requests/sec executed

That’s 128.75 seeks/second over 10GB, resulting in a net
throughput of 128.75 * 8KB/s = 1.01MB/s
Consider both the size of the disk used and the number of
clients doing seeks


More customizable seek tests

bonnie++ experimental (currently at 1.96)
iozone
ﬁo
Windows: HD Tune does everything but commit rate


Sources for slow disk results

Poor mapping to underlying hardware
Buggy driver
Insufficient bandwidth to storage
Bottlenecking at CPU/memory limits
Bad performing filesystem or filesystem misaligned with stripe
sizes
Writes faster than reads? Probably low read-ahead settings
somewhere.
Vibration: don’t shout at your JBODs! They don’t like it!
http://it.toolbox.com/blogs/database-soup/the-problem-with-iscsi-30602
http://blog.endpoint.com/2008/09/filesystem-io-what-we-presented.html
http://www.youtube.com/watch?v=tDacjrSCeq4


fsync tests

sysbench --test=fileio --file-fsync-freq=1 --file-num=1
--file-total-size=16384 --file-test-mode=rndwr run
| grep "Requests/sec"

pgbench insert-only test
PostgreSQL contrib/test fsync might work, but isn’t really
reliable


Sample laptop disk speciﬁcation

ST9320423AS Momentus 7200.4 320GB
7200 RPM
16MB Cache
Average seek: 11ms read/13ms write
Average rotational latency: 4.17ms


Computed parameters

Rotational latency = 1 / RPM / 60 / 2
IOPS=1/(latency+seek)
IOPS = 1/(((1/(RPM/60))/2) + S)
IOPS = 1/(4.17ms + 11ms) = 65.9 IOPS


IOPS Calculators and Info

http://www.wmarow.com/strcalc/
http://www.dbasupport.com/oracle/ora10g/disk_IO_02.shtml
http://storageadvisors.adaptec.com/2007/03/20/sata-iops-measurement/


Sample disk results

Disk Seq Seq bonnie++ Read-only Commit Drive
Count Read Write seeks seeks Rate Model

1 71 58 232 @ 4GB 194 @ 4GB 105/s 7200.4

1 59 54 177 @ 16GB 56 @ 100GB 10212/s WD160

3 125 119 371 @ 16GB 60 @ 100GB 10855/s RAID0
Commit rate for 7200.4 laptop drive is 1048/s with unsafe
volatile write cache
Non-laptop drives include a 256MB battery-backed write
cache, Linux SW RAID


Custom PostgreSQL tests
Quick CPU test (about 1140ms on my laptop):

timing
select sum(generate series) from
generate series(1,1000000);

Quick insert/plan test:

timing
CREATE TABLE test (id INTEGER PRIMARY KEY);
INSERT INTO test VALUES (generate series(1,100000));
EXPLAIN ANALYZE SELECT COUNT(*) FROM test;


Benchmarking functions

http://justatheory.com/computers/databases/postgresql/benchmarking_upc_validation.html
http://justatheory.com/computers/databases/postgresql/benchmarking_functions.html

pg stat user functions handles this speciﬁc job in 8.4
The general technique is applicable for all sorts of custom
benchmarks


pgbench select only test

set naccounts 100000 * :scale
setrandom aid 1 :naccounts
SELECT abalance FROM accounts WHERE aid = :aid;


pgbench custom test: insert only, to measure commit rate

set nbranches :scale
set ntellers 10 * :scale
set naccounts 100000 * :scale
setrandom bid 1 :nbranches
setrandom tid 1 :ntellers
setrandom delta -5000 5000
BEGIN
INSERT INTO history (tid, bid, aid, delta, mtime)
VALUES (:tid, :bid, :aid, :delta, CURRENT TIMESTAMP);
END;

pgbench standard test

BEGIN;
UPDATE accounts SET abalance = abalance + :delta
WHERE aid = :aid;
SELECT abalance FROM accounts WHERE aid = :aid;
UPDATE tellers SET tbalance = tbalance + :delta
WHERE tid = :tid;
UPDATE branches SET bbalance = bbalance + :delta
WHERE bid = :bid;
INSERT INTO history (tid, bid, aid, delta, mtime)
VALUES (:tid, :bid, :aid, :delta, CURRENT TIMESTAMP);
END;


pgbench-tools

Available from git.postgresql.org and github
Set of shell scripts to automate running many pgbench tests
Results are saved into a database for analysis
Saves TPS, latency, background writer statistics
Most of the cool features require PostgreSQL 8.3
Inspired by the dbt2 benchmark framework


Server conﬁguration for pgbench results

Quad-Core Intel Q6600
8GB DDR2-800 RAM
Areca ARC-1210 SATA II PCI-e x8 RAID controller, 256MB
write cache
DB: 3x640GB Western Digital SATA disks, short-stroked,
Linux software RAID-0
WAL: 160GB Western Digital SATA disk
CentOS 5.4, Linux Kernel 2.6.18-164.11.1.el5xen x86 64
OS on separate disk
Untuned ext3 ﬁlesystems


PostgreSQL Configuration

PostgreSQL 9.0-alpha4 (with multi-threaded pgbench patch)
shared buffers = 2GB
checkpoint segments = 32
checkpoint completion target = 0.9
wal buffers = 16MB
max connections = 300


pgbench read-only size scaling


pgbench read-only client scaling


pgbench TCP-B-like size scaling


pgbench TPC-B-like client scaling


What should you do?

Trust no one
Don’t start on application benchmarks until you’ve proven
basic performance
Don’t spend too long on basic performance if you can switch
to application benchmarks
Vendors alternate among lying, misunderstanding what you
want, and trying to make you feel dumb
Use simple, standard tools whenever possible to minimize
vendor disputes
Be prepared to translate to your vendor’s language and
subvert their agenda
Never spend real money on hardware unless you can return it
if it sucks


Credits

Special thanks to:
Mark Wong and the other contributors to dbt-2
Truviso for providing some of the test hardware included here


For more information...

Some slides are taken from the upcoming book:
“High Performance PostgreSQL” by Greg Smith
Planned for release this summer by Packt Publishing
Performance tuning of PostgreSQL 8.1 through 9.0, from
hardware to scaling via replication


Questions?

Remember that your coﬀee break awaits you


Database Hardware Benchmarking

More Related Content

What's hot (20)

Viewers also liked (9)

Similar to Database Hardware Benchmarking (20)

More from Command Prompt., Inc (20)

Database Hardware Benchmarking