RMOUG 2013 - Where did my CPU go?
5 likes986 views
Karl Arao presented on monitoring and capacity planning on consolidated environments. He discussed comparing CPU speeds using benchmarks like TPC-C and SPECint, the difference between cores and threads, common CPU events like CPU wait and scheduler, and tools for CPU monitoring like AWR, Tableau, vmstat and collectl. He showed examples of using these tools to analyze CPU usage across multiple hosts on a consolidated Exadata platform and identify optimization opportunities.
RMOUG 2013 - Where did my CPU go?
- 1. “Where did my CPU go?” monitoring & capacity planning adventures on a consolidated environment Presented by: Karl Arao 1
- 2. whoami Karl Arao • Senior Technical Consultant @ Enkitec • Performance and Capacity Planning Enthusiast 6 years 11 months 12 days DBA experience Oracle ACE, OCP-DBA, RHCE, OakTable Blog: karlarao.wordpress.com Wiki: karlarao.tiddlyspot.com Twitter: @karlarao www.enkitec.com 2
- 3. Agenda • HOWTO compare CPU speeds • Cores vs Threads • The different CPU events • CPU Monitoring/Capacity Planning on consolidated environments www.enkitec.com 3
- 4. 12:27:15 SYS@DEMO1> show parameter cpu_count NAME TYPE VALUE ------------------------------------ ----------- -------- cpu_count integer 16 www.enkitec.com 4
- 5. Exadata V2 => 2s8c16t CPU0 CPU8 CPU1 CPU9 CPU2 CPU10 CPU3 CPU11 Core0 Core1 Core2 Core3 Socket0 CPU4 CPU12 CPU5 CPU13 CPU6 CPU14 CPU7 CPU15 Core0 Core1 Core2 Core3 Socket1 www.enkitec.com 5
- 6. PART1: compare CPU speeds www.enkitec.com 6
- 7. Different methods: • Published benchmarks – TPC-C – SPECint_rate2006 • Actual Benchmarking – cputoolkit – SLOB (lio test) www.enkitec.com 7
- 8. TPC-C • Transaction Processing Performance Council (TPC) • Throughput => transactions per minute (tpmC) • Price/Performance => USD / tpmC • CPU performance => tpmC / core • 1609186.39 / 16 = 100574 www.enkitec.com 8
- 9. SPECint_rate2006 • Standard Performance Evaluation Corporation (SPEC) • SPECint_rate2006 • Integer performance • All CPUs are used • Used by OEM12c Consolidation Planner (SYSMAN.EMCT_SPEC_RATE_LIB) • CPU performance => SPECint_rate2006/core • 702/16 = 43.875 $ cat spec.txt | grep -i sun | grep -i x3-2 | sort -rnk1 44.0625, 16, 2, 8, 2, 632, 705, Oracle Corporation, Sun Blade X3-2B (Intel Xeon E5-2690 2.9GHz) 44.0625, 16, 2, 8, 2, 630, 705, Oracle Corporation, Sun Server X3-2L (Intel Xeon E5-2690 2.9GHz) 43.875, 16, 2, 8, 2, 628, 702, Oracle Corporation, Sun Server X3-2 (Intel Xeon E5-2690 2.9GHz) www.enkitec.com 9
- 10. 2007 vs 2012 www.enkitec.com 10
- 11. Actual Benchmarking • cputoolkit and SLOB (lio test) • LIOs/sec cputoolkit SLOB ./runcputoolkit-auto <start CPU> <end CPU> <db name> ./runit.sh <writers> <readers> ./runcputoolkit-auto 1 2 dw while :; do ./runit.sh 0 2; done CPU1 CPU1 CPU2 CPU2 CPU3 CPU3 CPU4 Both at 25% CPU4 CPU utilization CPU5 CPU5 CPU6 CPU6 CPU7 CPU7 CPU8 CPU8 www.enkitec.com 11
- 12. V2 and X2 CPU perf comparison 3.6M LIOs/sec 2.1M LIOs/sec www.enkitec.com 12
- 13. V2 -> X2 migration V2 X2 chip efficiency factor = (source LIOs/sec) / (destination LIOs/sec) = 2.1M / 3.6M = .5833 X2 CPU requirement = source host CPUs * utilization * chip efficiency factor = 16 * .46 = 7.36 * .5833 = 4.29 CPUs X2 CPU Utilization = CPU requirement / CPU capacity = 4.29 / 24 = 17.8 % www.enkitec.com 13
- 14. Core0 Core1 Core2 Core3 Socket0 PART2: Cores vs Threads CPU1 CPU5 CPU2 CPU6 CPU3 CPU7 CPU4 CPU8 Core0 Core1 Core2 Core3 Socket0 www.enkitec.com 14
- 15. www.enkitec.com 15
- 16. ~30% depends on the workload cputoolkit SLOB 17% 21% Intel HT Technology Technical User's Guide http://goo.gl/3Ec5Z www.enkitec.com 16
- 17. PART3: Different CPU events CPU CPU Wait CPU Scheduler www.enkitec.com 17
- 18. AAS CPU www.enkitec.com 18
- 19. www.enkitec.com 19
- 20. CPU Wait www.enkitec.com 20
- 21. www.enkitec.com 21
- 25. CPU Scheduler www.enkitec.com 25
- 26. www.enkitec.com 26
- 27. Putting it all together Problem: A single SQL Stmt. overwhelming Instances Caged SQL Applied to lock CPU resources. at 12 CPU’s ea. in good plan. www.enkitec.com 27
- 28. BEFORE: procs ----- r 52 58 50 58 49 load average: 10.36, 17.76, 36.42 AFTER: procs ------ r 13 13 After Before 12 12 14 www.enkitec.com 28
- 29. PART4: CPU monitoring and Capacity Planning www.enkitec.com 29
- 30. OS Tools • The usual Operating System commands – vmstat – top – mpstat –P ALL 1 5 • Cool tools – collectl –sC (http://collectl.sourceforge.net) – turbostat.c – dcli (Exadata) • dcli -l oracle -g /home/oracle/dbs_group --vmstat 2 • dcli -l oracle -g /home/oracle/dbs_group uptime www.enkitec.com 30
- 31. Load Map www.enkitec.com 31
- 32. Performance Page – Historical View www.enkitec.com 32
- 33. AWR Toolkit • DIY performance data warehouse run_awr Extract AWR data points as csv files 1 Customer site run_extract Package all the csv files FRESH_LOAD Create new client “dimension” tables 2 DIY DW server CHECK_LOAD Check new data points DELTA_LOAD Load new data points awr_topevents_(ClientNameX) awr_topevents_(ClientNameY) awr_topevents_(ClientNameZ) awr_cpuwl_(ClientNameX) awr_cpuwl_(ClientNameY) awr_cpuwl_(ClientNameZ) awr_iowl_(ClientNameX) awr_iowl_(ClientNameY) awr_iowl_(ClientNameZ) 3 Tableau Analytics www.enkitec.com 33
- 34. www.enkitec.com 34
- 35. • Tableau auto creates a time dimension for the time column “MM/DD/YY HH24:MI:SS” of AWR csv output www.enkitec.com 35
- 36. • Summary and Underlying data 1-2AM 2-3AM www.enkitec.com 36
- 37. CPU usage across half rack Exadata www.enkitec.com 37
- 38. CPU usage per host www.enkitec.com 38
- 39. CPU redistribution across nodes www.enkitec.com 39
- 40. Wrap up! • HOWTO compare CPU speeds o SPECint_rate2006, TPC-C, Actual benchmarking • Cores vs Threads o Always have HT on o ~30% performance benefit after core count • The different CPU events o 1 AAS CPU = 1 CPU core o Oracle CPU may not correlate with Host CPU if you have a lot of CPU activity outside of the database • CPU Monitoring/Capacity Planning on consolidated environments o AWR analytics www.enkitec.com 40
- 41. Resources • cputoolkit - http://karlarao.wordpress.com/scripts-resources/ • AWR Tableau and R toolkit Visualization Examples - http://goo.gl/xZHHY • AAS investigation - http://goo.gl/5WaAg • Cores vs Threads - http://goo.gl/1MLFf • Turbostat.c - http://goo.gl/jDUKg • cpu_topology - http://goo.gl/EUDG7 • CPU centric benchmark comparisons - http://goo.gl/nR9Yy • SLOB - http://goo.gl/yKa45 • Kyle Hailey - http://dboptimizer.com/2011/07/21/oracle-cpu-time/ • The mindmap of this presentation - http://goo.gl/XeY0e karl.arao@enkitec.com www.enkitec.com 41
- 42. Fastest Growing Companies in Dallas 42