Database Benchmarking: Miseries, Myths and Misconceptions
Download as PPTX, PDF1 like391 views
The document discusses the common pitfalls in database benchmarking, emphasizing that many efforts fail due to inadequate time, improper skill sets, insufficient resources, and a lack of understanding of benchmarks. It highlights the importance of preparation, using the correct tools, and following industry standards like TPC benchmarks. Furthermore, it addresses misconceptions about metrics, particularly the distinction between transactions per second and average response time.
Database Benchmarking: Miseries, Myths and Misconceptions
- 1. © 2017 IDERA, Inc. All rights reserved. Proprietary and confidential. DATABASE BENCHMARKING: MISERIES, MYTHS AND MISCONCEPTIONS Bert Scalzo, PhD. & Oracle ACE June 2017
- 2. 2© 2017 IDERA, Inc. All rights reserved. Proprietary and confidential. BOOKS BY AUTHOR Fall 2017
- 3. 3© 2017 IDERA, Inc. All rights reserved. Proprietary and confidential.
- 4. 4© 2017 IDERA, Inc. All rights reserved. Proprietary and confidential. MOST DB BENCHMARKING EFFORTS FAIL MISERABLY
- 5. 5© 2017 IDERA, Inc. All rights reserved. Proprietary and confidential. WHY DB BENCHMARKING FAILS Insufficient time allocated • 1-3 weeks under-estimated • 1-2 months necessary (or more) Wrong skill sets assigned • DBA – i.e. strong DB knowledge • Good OS, network & storage skills Insufficient hardware/software Wrong setup / configuration • Often out-of-box defaults (wrong!) Lack of benchmarking knowledge • Almost no one reads the specs!!! • Not sure what to measure and why • No experience interpreting results
- 6. 6© 2017 IDERA, Inc. All rights reserved. Proprietary and confidential. DB BENCHMARKING REQUIRES PREPARATION
- 7. 7© 2017 IDERA, Inc. All rights reserved. Proprietary and confidential. DB BENCHMARKING IS NOT EASY! 7 No tool can think for you or do your job – no big red easy button
- 8. 8© 2017 IDERA, Inc. All rights reserved. Proprietary and confidential. DB BENCHMARKING REQUIRES TOOLS & EXPERT Database Benchmarking Tool: stress the database & make it sweat – that’s all they do (no built-in smarts) Database Monitor Expert – must know database, OS, hardware, software, tuning, & have read the benchmark specs Expert decides what treadmill inclination, what speed, what duration, what to monitor, how to diagnose, how to treat, etc. … ?
- 9. 9© 2017 IDERA, Inc. All rights reserved. Proprietary and confidential. DB BENCHMARKING – COMMON, SIMPLE FAILURES Wrong person trying to do the heavy lifting Right person trying to do tests without right monitoring tools Not familiar with benchmark specs and/or how well tool adheres to it
- 10. 10© 2017 IDERA, Inc. All rights reserved. Proprietary and confidential. KNOW THE INDUSTRY STANDARD BENCHMARKS TPC-C older OLTP benchmark basic “order entry” type app TPC-H basic data warehousing 22 queries – star schema=NO TPC-E newer OLTP benchmark simulates online brokerage firm TPC-DS newer data warehousing 99 queries – star schema=YES TPC-DI data integration/ETL ETL brokerage firm into DW TPC-VMS virtualized database standard DB tests on VM’s
- 11. 11© 2017 IDERA, Inc. All rights reserved. Proprietary and confidential. If you don’t know this, how can you set either benchmark tool or DB parameters ??? http://tpc.org/tpcc/spec/tpcc_current.pdf
- 12. 12© 2017 IDERA, Inc. All rights reserved. Proprietary and confidential. UNDERSTAND THE DB DESIGN – TPC-C
- 13. 13© 2017 IDERA, Inc. All rights reserved. Proprietary and confidential. UNDERSTAND THE DB DESIGN – TPC-H
- 14. 14© 2017 IDERA, Inc. All rights reserved. Proprietary and confidential. CREATE DATA MODEL IF UNSURE (TPC-H)
- 15. 15© 2017 IDERA, Inc. All rights reserved. Proprietary and confidential. KNOW WHAT’S ALLOWED
- 16. 16© 2017 IDERA, Inc. All rights reserved. Proprietary and confidential. UNDERSTAND THE DB DESIGN – TPC-E
- 17. 17© 2017 IDERA, Inc. All rights reserved. Proprietary and confidential. UNDERSTAND THE DB DESIGN – TPC-DS – PART 1
- 18. 18© 2017 IDERA, Inc. All rights reserved. Proprietary and confidential. UNDERSTAND THE DB DESIGN – TPC-DS – PART 2
- 19. 19© 2017 IDERA, Inc. All rights reserved. Proprietary and confidential. FREE BENCHMARKING TOOLS HammerDB (formerly HammerORA) • Includes 2 benchmarks: TPC-C and TPC-H • http://www.hammerdb.com Swingbench • Includes 6 benchmarks: OrderEntry, SalesHistory, TPC-DS Like, JSON, CallingCircle and StressTest • http://dominicgiles.com/swingbench.html Benchmark Factory (freeware) • Inlcudes 3 benchmarks: TPC-C, TPC-E and TPC-H • https://www.toadworld.com/m/freeware/555 • NOTE: only commercial version on Quest website 19
- 20. 20© 2017 IDERA, Inc. All rights reserved. Proprietary and confidential. TOP-10 BENCHMARKING MISCONCEPTIONS
- 21. 21© 2017 IDERA, Inc. All rights reserved. Proprietary and confidential.
- 22. 22© 2017 IDERA, Inc. All rights reserved. Proprietary and confidential.
- 23. 23© 2017 IDERA, Inc. All rights reserved. Proprietary and confidential.
- 24. 24© 2017 IDERA, Inc. All rights reserved. Proprietary and confidential.
- 25. 25© 2017 IDERA, Inc. All rights reserved. Proprietary and confidential.
- 26. 26© 2017 IDERA, Inc. All rights reserved. Proprietary and confidential.
- 27. 27© 2017 IDERA, Inc. All rights reserved. Proprietary and confidential.
- 28. 28© 2017 IDERA, Inc. All rights reserved. Proprietary and confidential.
- 29. 29© 2017 IDERA, Inc. All rights reserved. Proprietary and confidential. READ FULL DISCLOSURE REPORTS FOR SIMILAR SETUPS
- 30. 30© 2017 IDERA, Inc. All rights reserved. Proprietary and confidential. LOOK AROUND – FIND CLOSEST MATCH POSSIBLE
- 31. 31© 2017 IDERA, Inc. All rights reserved. Proprietary and confidential. Very common technique to load very large data sets into ETL or staging tables, and then to do create table as select (CTAS) to populate the benchmark tables APPENDIX B WILL SHOW DDL FOR OPTIMAL DESIGN
- 32. 32© 2017 IDERA, Inc. All rights reserved. Proprietary and confidential. Transactions per Second • Gets far too much attention • Meaningless to most users • Sort of like automobile RPM’s (how fast internal engine is working – not how fast car is moving or how soon we’ll arrive) • Misconception that TPS equates to IOPS (IO Operations per Second) – ignores database memory caching and logging Average Response Time • Gets far too less attention • Meaningful to most users • Sort of like MPH (or KPH) (how fast car actually is moving – so infers how soon we’ll arrive or amount of fuel we’ll use) • When examined in conjunction with TPS, then a generally observable and clear pattern often emerges (next slide) … METRICS – TPS VS. AVG. RESPONSE TIME
- 33. 33© 2017 IDERA, Inc. All rights reserved. Proprietary and confidential. 0 1000 2000 3000 4000 5000 6000 7000 100 200 300 400 500 600 700 800 900 1000 TPS Avg Resp Time Looking for inflection point whereTPS is still increasing or just decreasing and close to max where average response time is below customer defined SLA Notice the line characteristics between roughly 750 and 850 concurrent users – for current configuration and optimization benchmark results interesting Common mistake to simply attempt maximizeTPS – rememberTPS is not IOPS BENCHMARK PATTERN – TRUE POINT OF SATURATION
- 34. 34© 2017 IDERA, Inc. All rights reserved. Proprietary and confidential.