ASH Archit ecture and Advanced Usage.pdf
- 1. ASH Architecture and Advanced Usage Graham Wood, Uri Shaft, John Beresniewicz Oracle America
- 2. The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
- 3. Agenda • ASH Fundamentals and Architecture • Estimating DB Time Using ASH • The ASH Fix-up Mechanism • Estimating Event Counts Using ASH • Additional ASH Arcana • Lessons Learned
- 4. Motivation for ASH in 10i Performance diagnostics tool with: • Always on (unlike SQL trace) • History of database activity • Fine grained detail Other requirements: • Very small performance overhead • Uses no locks of any kind • Works well even when system is overloaded
- 5. Active Session History (time based) A sample of activity taken at regular intervals t1 Session 1 Session 2 Session 3 Session Time Duration Activity SQL Object 1 t1 null CPU SQL1 null 2 t1 100ms I/O SQL2 EMP 3 t1 5ms I/O SQL1 EMP 3 t2 3sec Row lock SQL1 DEPT t2 t3
- 6. ASH Architecture Every 1 hour or out-of-space AWR Circular buffer in SGA (2MB per CPU) DBA_HIST_ACTIVE_SESS_HISTORY V$ACTIVE_SESSION_HISTORY Session state objects MMON Lite (MMNL) V$SESSION V$SESSION_WAIT Variable length rows Direct-path INSERTS Write 1 out of 10 samples
- 7. ASH Architecture Readers go unlatched Writer goes one direction Every 1 hour or out-of-space AWR Circular buffer in SGA (2MB per CPU) DBA_HIST_ACTIVE_SESS_HISTORY V$ACTIVE_SESSION_HISTORY MMON Lite (MMNL) Session state objects V$SESSION V$SESSION_WAIT Readers go the opposite way Indexe d on time Indexe d on time
- 8. ASH Pros and Cons PROS • Supports the DB Time method of performance analysis • Historically for large or systemic problems • Recently or “now” (V$SESSION still available) for emergency • Always available (modulo licensing restrictions) • Minimal cost to server performance CONS • Queries become estimates • We query a sample and not the full data set • Difficult to form semantically meaningful queries • Probability distribution of a query result matters
- 9. Database Time and Active Sessions Database Time = Total time spent by sessions in the database server actively working (on CPU) or actively waiting (non-idle wait) Active Sessions = The number of sessions active (working or waiting) in the database server at a particular time Active Sessions is the rate of change of Database Time over time, also referred to as load
- 10. ASH and Database Time i S T DBT DB Time is approximated by multiplying sampling interval by the total count of the samples DBT » T · (rows in ASH) Each ASH sample represents 1 second of DB Time
- 11. Basic ASH Math COUNT(*) = DB Time GROUP BY ?
- 12. Active Sessions and DB Time • The number of active sessions at any time is the rate of change of the “DB Time function” at that time. • DB Time is the integral of the Active Session function. 1 0 t t ions ActiveSess DBtime ions ActiveSess t DBtime /
- 13. time t0 t1 Active Sessions and DB Time t = 1 sec Active sessions DB Time
- 14. ASH DB Time Query (basic ASH math) • What are the top SQL by database time in the last 5 minutes? SELECT * FROM (SELECT NVL(SQL_ID,'NULL') as SQL_ID ,SUM(1) as DBtime_secs FROM V$ACTIVE_SESSION_HISTORY WHERE sample_time > SYSDATE - 5/24/60 GROUP BY SQL_ID ORDER BY 2 DESC ) WHERE rownum < 6; NOTE: SUM(:T) can be substituted for SUM(1) to account for non-default sampling intervals
- 15. ASH Math: EM Top Activity
- 16. ASH Math: ASH Analytics
- 17. Multi-dimensional DB Time Analysis
- 18. 18 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
- 19. Common Mistakes ASH = Complete record of database activity • SUM(time_waited) = total time spent in database • COUNT(*) = total count of wait events • AVG(time_waited) = average latency of waits ASH = Random sample of wait events • AVG(time_waited) = estimate of average wait latency • Top SQL by SUM(time_waited) = Top SQL by time spent in the server
- 20. Queries Over Samples • ASH is a time sample of the true workload • One can imagine multiple ASH mechanisms each producing different results from the exact same true workload • ASH rows are random variables extracted from the true data • It is a time sample: the probability an event is sampled is proportional to the event duration • A 10ms event is 10 times more likely to be in ASH than a 1ms event
- 21. Query Results Are Estimates • ASH does not contain all the data, just a sample • Any aggregate over ASH data is a random variable • It estimates something or is a sample of something. • Any query over samples is an unbiased estimator of a property value if the expected value of the query equals that property value
- 22. Compare Estimated to Actual DB Time
- 23. Bad ASH Math • SQL observed using 9 secs of CPU every 10 secs
- 24. The ASH “Fix-up” • ASH columns may be unknown at sampling time • TIME_WAITED: session is still waiting • PLAN_HASH: session is still optimizing SQL • GC events: event details unknown at event initiation • Certain time model bit vector columns • ASH “fixes up” data during subsequent sampling • TIME_WAITED fixed up in first sample after event completes • Long events: last sample gets fixed up time_waited (all others stay 0) • Querying current ASH may return un-fixed rows • Should not be a problem generally
- 25. ASH Fix-up 1: Sampling Session State Objects ASH Sampler MMNL ON CPU ON CPU ON CPU WAITING WAITING IO WAIT V$ASH TIME_WAITED PLAN_HASH TIME_WAITED Fix-up List Post-wait callback registered by session
- 26. ASH Fix-up 2: Fixing Up Callback Session State Objects ASH Sampler MMON ON CPU ON CPU ON CPU TIME_WAITED=n WAITING V$ASH TIME_WAITED=n PLAN_HASH Fix-up List
- 27. Estimating Event Counts With ASH :T = ASH sampling interval TIME_WAITED = length of sampled event N = Number of events the ASH sample represents :T ~ TIME_WAITED * N OR N ~ :T / TIME_WAITED
- 28. Why Event Count = :T / Time_Waited? 250 ms Sally 250 ms 250 ms 250 ms 250 ms User1 User2 User3 User4 tN tN+:T :T elapsed time (1000 ms) ASH sample time Sally’s row in ASH estimates 1000/250 ms = 4 waits during time interval
- 29. ASH Event Count Query (1) • Top 5 objects by User I/O requests last 15 minutes SELECT W.*,O.object_name FROM (SELECT current_obj# ,ROUND(SUM(100000/time_waited)) as est_IOwaits ,SUM(1) as est_DBtime FROM V$ACTIVE_SESSION_HISTORY WHERE sample_time > SYSDATE - 15/24/60 AND time_waited > 0 AND event IN (‘db file sequential read’ ,’db file scattered read’) GROUP BY current_obj# ORDER BY 2 DESC ) W , DBA_OBJECTS O WHERE o.object_id = W.current_obj# AND ROWNUM < 6;
- 30. Long Events • Events that are longer than :T are always sampled • No sampler “bias” for these • Event may be sampled multiple times • Only final (“fixed-up”) ASH row for long events has TIME_WAITED > 0 • The event count of long events is known: • 1 for each row with TIME_WAITED >= :T • The DB Time estimate is still COUNT(*)
- 31. ASH Event Count Query (2) • Top objects by I/O event counts adjusted for long events SELECT * FROM (SELECT current_obj# ,ROUND(SUM(CASE WHEN time_waited >= 1000000 THEN 1 ELSE 1000000 / time_waited END)) as est_IOwaits FROM V$ACTIVE_SESSION_HISTORY WHERE sample_time > SYSDATE - 15/24/60 AND time_waited > 0 AND wait_class = 'User I/O' GROUP BY current_obj# ORDER BY 2 DESC ) WHERE ROWNUM < 6;
- 32. Estimating Event Latencies 1. We estimate total DB time on events 2. We can now estimate total event occurrences 3. Therefore, we can compute average latency Est_avg_latency_ms = est_Dbtime_ms / est_waits
- 33. User I/O Events Ordered by Latency SELECT event ,ROUND(est_DBtime_ms/est_waits,1) as est_avg_latency_ms FROM (SELECT event ,ROUND(SUM(GREATEST(1,1000000/time_waited))) as est_waits ,SUM(1000) as est_DBtime_ms FROM V$ACTIVE_SESSION_HISTORY WHERE sample_time > SYSDATE - 15/1440 AND time_waited > 0 AND wait_class = 'User I/O' GROUP BY event ) ORDER BY 2 DESC; • Note preferred syntax for computing event counts
- 34. DEMO?
- 35. Additional ASH Arcana • ASH Samples “ON CPU” • Challenges of SQL_ID tracking • Sampling CPU-bound systems • Current_OBJ# and when to use it • Blocking session information
- 36. ON CPU and ASH • ASH row status “ON CPU” derived, not observed • Session is in a database call • Session is NOT in a wait event (idle or non-idle) • Un-instrumented waits => “ON CPU” • These are bugs and should be rare, but have happened • Session on run queue may be WAITING or ON CPU • Depends on state prior to going onto run queue
- 37. Evolution of SQL_ID •Objective is to get lowest level user level SQL •Prior to 11.1.0.7 various schemes tried •Varying levels of success, but each had problems •11.1.0.7 and above: it is the currently executing SQL • except recursions inside the server code, triggers •PL/SQL has both top level and current level calls •ENTRY_LEVEL and columns for applications that start with generic PL/SQL and end up doing interesting things underneath – need to find what the applications initially called.
- 38. Why does ASH work when the server is CPU bound? 1. ASH sampler is very efficient, and does not lock • Therefore, in almost all cases it takes a single CPU slice to finish a full sample. 2. After a sample is done, the sampler computes next scheduled sample time and sleeps until then 3. Upon scheduled wake-up, it waits for CPU (runq) and samples again • Thus, CPU bound samples are shifted by one runq but stay about 1 per second
- 39. ASH Sampler and Run-queue S_t0 S_t2 S_t1 Run queue Run queue A_t1 A_t0 Run queue A_t2 Sleep until next time Sleep until next Sample Sample Sample If run queue times are consistent sampling interval will be preserved but sample times shifted
- 40. What Is CURRENT_OBJ# It is the ID of the segment we operate on when ASH is sampled. However, this is only valid in specific wait events: •I/O events on data blocks •Cluster (global cache) events on data blocks •Row Locks •Table Locks •Buffer busy (and associated RAC events) DANGER: the column is not cleared when it is invalid.
- 41. Blocking Session BLOCKING_SESSION_STATUS BLOCKING_SESSION BLOCKING_SESSION_SERIAL# BLOCKING_INST_ID Prior to 11.2: Finding blocking session in most cases can create a lock/crash Therefore, blocking session only in same instance, only for some cases From 11.2: We copy information from Hang Manager. Therefore, events that are long enough (3 seconds in same instance, 10 seconds in RAC) get the hang information copied into ASH.
- 42. Lessons Learned • ASH is a time-based and not event-based sample of database activity • ASH is an excellent representation of activity history • How to estimate and rank DB Time spent over ASH dimensions using basic ASH Math • ASH estimates underlie key DB Time performance analysis use cases exposed by EM • The ASH fix-up is a critical (and unique) mechanism • How to estimate event counts and latencies using TIME_WAITED
- 43. DEMO?