Using AWR/Statspack for Wait Analysis

1. Using AWR For Wait AnalysisMike Ault, Oracle GuruMarch 2011The World’s Fastest Storage® for over thirty years!

2. Michael R. Ault, Oracle Guru Nuclear Navy 6 years

3. Nuclear Chemist/Programmer 10 years

4. Kennedy Western University Graduate

5. Bachelors Degree Computer Science

6. Certified in all Oracle Versions Since 6

7. Oracle DBA, author, since 1990Books by Michael R. Ault

8. StatspackAnalyzer.comFree Statspack/AWR Analysis Sponsored by Texas Memory SystemsLooks for IO bottlenecks and other configuration issues.

9. Straightforward tuning adviceIntroductionStatspack was introduced in 8.1.7AWR came out in Oracle10gBoth are very similarBoth provide a top-down look at performance statistics

10. What Is AWR?A background processA set of tablesA set of reportsTakes snapshots of statistics every hourTakes snapshot of high-cost SQL every hour

11. Preparation for AnalysisKnow your systems normal performance fingerprintBe familiar with Concepts and Tuning GuidesHave “normal” AWR/Statspacks for comparison

12. Top-Down ApproachReport starts with settings overviewNext provides Top-5 waitsUse the Waits to guide further investigation

13. AWR Report HeaderWORKLOAD REPOSITORY report forDB Name DB Id Instance Inst Num Startup Time Release RAC------------ ----------- ------------ -------- --------------- ----------- ---AULTDB 4030696936 aultdb1 1 04-Aug-08 10:16 11.1.0.6.0 YESHost Name Platform CPUs Cores Sockets Memory(GB)---------------- -------------------------------- ---- ----- ------- ----------aultlinux3 Linux IA (32-bit) 2 1 1 2.97 Snap Id Snap Time Sessions Curs/Sess--------- ------------------- -------- ---------Begin Snap: 91 04-Aug-08 12:00:15 41 1.2 End Snap: 92 04-Aug-08 13:00:28 47 1.1Elapsed: 60.22 (mins) DB Time: 139.52 (mins)Cache Sizes Begin End~~~~~~~~~~~ ---------- ---------- Buffer Cache: 1,312M 1,312M Std Block Size: 8K Shared Pool Size: 224M 224M Log Buffer: 10,604K

22. Know Your Load Type!Online Transaction ProcessingFew readsMany writesMany small transactionsLook for redo/undo and sequential read issuesDecision Support/Data WarehouseMany readsFew writes (other then possible temp)Few transactionsLook for sort/workarea and scattered read issuesMixed or Hybrid

23. Load Profile SectionLoad Profile Per Second Per Transaction Per Exec Per Call~~~~~~~~~~~~ --------------- --------------- ---------- ---------- DB Time(s): 2.3 7.1 0.63 1.05 DB CPU(s): 0.3 0.9 0.07 0.13 Redo size: 800.5 2,461.8 Logical reads: 6,307.6 19,396.7 Block changes: 3.6 10.9 Physical reads: 2,704.9 8,317.8 Physical writes: 86.9 267.3 User calls: 2.2 6.8 Parses: 2.0 6.1 Hard parses: 0.0 0.1W/A MB processed: 932,965.4 2,868,990.9 Logons: 0.1 0.2 Executes: 3.7 11.3 Rollbacks: 0.1 0.3 Transactions: 0.3

31. What Are Your EfficienciesShould be close to 100%Parse issues usually are a result of:Bad bind variable usageInsufficient memoryWill also be co-indicated by low percentage of memory for multiple SQL execution

32. Load Profile SectionInstance Efficiency Percentages (Target 100%)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Buffer Nowait %: 100.00 Redo NoWait %: 99.97 Buffer Hit %: 96.09 In-memory Sort %: 100.00 Library Hit %: 98.17 Soft Parse %: 97.88 Execute to Parse %: 45.80 Latch Hit %: 99.95Parse CPU to ParseElapsd %: 0.00 % Non-Parse CPU: 99.77Shared Pool Statistics Begin End ------ ------ Memory Usage %: 81.53 85.39 % SQL with executions>1: 79.29 79.48 % Memory for SQL w/exec>1: 76.73 78.19

45. Top 5 Waits SectionCritical to look closely at this sectionUse highest wait times to guide investigationDB FILE type waits – physical IOBUFFER type waits – Logical IOLOG type waits – Redo relatedPX – Parallel QueryGC – Global Cache (RAC related)Undo – Undo or rollback segment related

46. Top 5 Waits SectionTop 5 Timed Foreground Events~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Avg wait % DBEvent Waits Time(s) (ms) time Wait Class------------------------------ ------------ ----------- ------ ------ ----------db file sequential read 465,020 3,969 9 47.4 User I/ODB CPU 995 11.9db file parallel read 2,251 322 143 3.8 User I/Odb file scattered read 15,268 153 10 1.8 User I/Ogc current block 2-way 108,739 116 1 1.4 Cluster

52. DB File Type WaitsDB File Sequential Reads – memory starvation, non-selective indexesDB File Scattered Reads – full table scans, insufficient indexingDirect Path Writes – Appends, data loadsDirect Path Reads – Parallel slaves used to retrieve dataDB File Parallel Writes – Backup and partition useDB File Parallel Reads – Partition useDB File Single Write – File header writes, excessive data filesDirect path read temp – Temp file activity (sorts, hashes, temp tables, bitmaps)Direct path write temp – Temp file activity (sorts, hashes, temp tables, bitmaps)

60. Buffer Type Waitslatch: cache buffers chains – Hot blocks, check for hot objectsfree buffer waits – Insufficient buffers, processes holding buffers too long, IO subsystem over loadedbuffer busy waits – See what is causing them further along in reportgc buffer busy – Overloaded interconnect, find problem objects and tunelog buffer space – High load, too small a log buffer, increase log buffer sizelatch: cache buffers lru chain – Freelist issues, hot blockslatch: cache buffer handles – Freelist issues, hot blocksbuffer busy - See what is causing them further along in reportno free buffers – Insufficient buffers, dbwr contention

65. Log Type Waitslog file parallel write – Look for log file contentionlog buffer space – Look at increasing log buffer sizelog file switch (checkpoint incomplete) – May indicate excessive db files or slow IO subsystemlog file switch (archiving needed) – Indicates archive files are written too slowlylog file switch completion – May need more log files per threadlog file sync – Could indicate excessive commits

69. PX Type WaitsPX Deq: Msg Fragment – PEM maybe too smallPX qref latch - Data is produced faster than it is consumed, look at PEMPX Deq Credit: send blkd – Look at PEM size and parallel query into non-parallel DML

70. GC Eventsgccr multi block request – Full table or index scansgc current multi block request – Full table or index scansgccr block 2-way – Blocks are busy in another instance, check for block level contention or hot blocksgccr block 3-way – Blocks are busy in another instance, check for block level contention or hot blocksgccr block busy – Blocks are busy in another instance, check for block level contention or hot blocksgccr block congested – cr block congestion, check for hot blocks or busy interconnectgccr block lost – Indicates interconnect issues and contentiongc current block 2-way – Blocks are busy in another instance, check for block level contention or hot blocksgc current block 3-way – Blocks are busy in another instance, check for block level contention or hot blocksgc current block busy – Block is already involved in GC operation, shows hot blocks or congestiongc current block congested – current block congestion, check for hot blocks or busy interconnectgc current block lost - Indicates interconnect issues and contention

73. Undo Eventsundo segment extension – If excessive, tune undolatch: In memory undo latch – If excessive could be bug, check for your version, may have to turn off in memory undowait for a undo record – Usually only during recovery of large transactions, look at turning off parallel undo recovery.

76. What Next?Determine wait events of concernDrill down to specific sections of report for deeper analysisUse custom scripts, ADDM and Ash to investigate issues

77. Thank You!Mike AultOracle@RamSan.comhttp://www.statspackanalyzer.com/

Using AWR/Statspack for Wait Analysis

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Using AWR/Statspack for Wait Analysis (20)

Recently uploaded (20)

Using AWR/Statspack for Wait Analysis