Oracle RAC 12c Practical Performance Management and Tuning OOW13 [CON8825]

9/30/13&
1&
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.1
Oracle RAC 12c Practical
Performance Management
and Tuning
Markus Michalewicz,
Director of Product Management
Michael Zoll,
Oracle RAC Performance Architect

9/30/13&
2&
Safe Harbor Statement
The following is intended to outline our general product direction. It is
intended for information purposes only, and may not be incorporated
into any contract. It is not a commitment to deliver any material, code, or
functionality, and should not be relied upon in making purchasing
decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole
discretion of Oracle.
Agenda
!  Introduction
!  Fundamentals
!  Network Recommendations
!  Application Considerations
!  Testing
!  Identifying Issues
!  Monitoring & Analysis
!  Summary and Q&A

9/30/13&
3&
The Rule of “3” for Oracle RAC Performance
“3” layers of performance consideration:
1.  The application
2.  The database including servers
3.  The storage
The “3” performance relevant stages:
1.  Preparation / planning
2.  Test / product (monitoring)
3.  Tuning
Oracle Real Application Clusters (RAC)
Oracle Grid Infrastructure (GI):
Automatic Storage Management (ASM)
Oracle Clusterware (OCW)
CRM$
Preparation of the Servers
!  Setup of the Servers include:
–  Setting up network connections
!  Public and private with switch
–  Setting up shared storage for all servers
!  Shared storage required
–  Setting up the OS image – here OL 6.4
!  Use pre-install package
!  And FixUp scripts to optimize
!  The “3” areas to optimize:
–  The servers (memory / CPU)
–  The network (interconnect mainly)
–  The storage (IOPs / throughput)
dasher
OS OL 6.4
dancer
OS OL 6.4
vixen
OS OL 6.4
comet
OS OL 6.4
OS image setup and more
Example attached

9/30/13&
4&
Preparation of the Servers
!  Recommendations:
–  Use OL 6.4+ UEK
–  Use the Oracle preinstall package
!  Available for
–  Oracle Database 11g Rel. 2
–  Oracle Database 12c Rel. 1
–  Additional steps to perform after using the
pre-install package for an Oracle RAC cluster:
!  Add a “grid” user & “dba” group
!  Align UIDs / GUIDs across servers
!  Set SELinux to “permissive”, if possible
!  Check large / huge pages settings
Kernel and Oracle preinstall package
[root@dasher ~]# uname –a
Linux dasher 2.6.39-400.17.1.el6uek.x86_64 #1 SMP Fri Feb 22
18:16:18 PST 2013 x86_64 x86_64 x86_64 GNU/Linux
#Get the pre-install package
[root@dasher Desktop]# yum list oracle-*
oracle-rdbms-server-11gR2-preinstall.x86_64 1.0-7.el6 ol6_latest
oracle-rdbms-server-12cR1-preinstall.x86_64 1.0-8.el6 ol6_latest
Install Oracle Grid Infrastructure 12c
Use FixUp scripts to omptimize the configuraiton – part 1

9/30/13&
5&
Install Oracle Grid Infrastructure 12c
Use FixUp scripts to omptimize the configuraiton – part 2
Do fix
later!
Agenda
!  Introduction
!  Fundamentals
!  Testing

9/30/13&
6&
Oracle RAC Fundamentals
!  Instances communicate over the
private interconnect for two purposes:
1.  Function / message shipping
2.  Data shipping (block transfer)
!  Spinning Disk access is minimized
as much as possible
–  Flash Cache is utilized, if present
dasher dancer
vixencomet
Communication flows in the cluster
racdb1_2
racdb1_3
racdb1_1
racdb1_4
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
dasher
comet
Processes and Functions per instance
racdb1_3
racdb1_1
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
dancer
vixen
racdb1_2
racdb1_4
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
LM
Dictionary
Cache
Log buffer
LGWRDBW0
Library
Cache
Buffer Cache
LMONLMD0
SGA
LMSx
Cluster Private High Speed Network
Global Enqueue Service Global Cache Service
Oracle
Process
Oracle
Process
Runs in
Real
Time
Priority
LM
HB

9/30/13&
7&
dasher dancer
vixencomet
“3” ways to get access to the data
racdb1_2
racdb1_3
racdb1_1
racdb1_4
!  Data is either stored
1.  Locally (local cache) " access time: nanoseconds
2.  Remote (global cache) " access time: micros.
3.  “On disk”
!  Flash cache " access time: microseconds
!  Disk controller cache " access time: micros.
!  Spinning disk " access time: milliseconds
3
21
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
dasher dancer
vixencomet
Maximum “3” way communication to access data
racdb1_2
racdb1_3
racdb1_1
racdb1_4
!  In the worst case,
–  The requester asks
–  for data held in a remote instance
–  mastered in a third instance
master
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC

9/30/13&
8&
Agenda
!  Introduction
!  Fundamentals
!  Testing
Network Recommendations
What is “normal”?
vixencomet
racdb1_3 racdb1_4
Block
size
RT (usec)
2K 4K 8K 16K
UDP/GE 300 310 360 460
RDS/IB 100 130 160 200
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC

9/30/13&
9&
Interconnect Latency Variation
vixencomet
racdb1_3 racdb1_4
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
<250 500 2000 8000 >16000
usec per message
Distribution of message roundtrip times
pct
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
<250 500 2000 8000 >16000
usecs per message
Distribution of message roundtrip times ( under
load )
pct
Mean: 250 µsecs
Mean: 800 µsecs
5% > 8 ms!  Interconnect Utilization or Network Configuration
!  High CPU Utilization and Process Concurrency
!  LMS utilization and priorities
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
vixencomet
How to prepare to get to a “normal” state?
racdb1_3 racdb1_4
!  Size Interconnect for aggregated throughput
–  ROT approx1Gb/sec per 32 cores
!  10Gb/sec or IB generally good for database
consolidations and large SMP nodes
!  Bonded 1GbE solutions with
load balancing are supported
–  Use same type NICs for LB and failover
–  Use different subnets
!  Use Jumbo frames wherever possible
!  Check for IP & NIC send and receive buffer size
!  Interconnect should be stress
tested with Iperf or Netperf
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC

9/30/13&
10&
vixencomet
Use Jumbo Frames wherever possible
!  Instances communicating can require database
blocks to be transferred over the network
!  The size of a block exceeding the size of the
MTU will be fragmented and reassembled
–  Ethernet IP:
!  Costs charged to driver and OS Kernel
!  Increases chance of “losing” a block
!  UDB unreliable
!  Detection and retry for 8K block should
be avoided. It is avoided by using Jumbo
Frames for 8K blocks
–  IB has larger MTU sizes
!  RDS is reliable
8K Data
Block
1500 byte MTU
Send() Receive()
ReassemblyFragmentation
Oracle RAC Oracle RAC
Transfer Patch Length
usersys
Context switchNIC
Driver and IP stack IPC
Global Cache (RDBMS)
Wire
CPUSocketPackets
Packet Headers
Queues
Time
!  Wire latency is very small
–  ~ 50% of fixed overhead is in kernel
–  Protocol ( e.g. UDP, RDS ) dependent
!  IPC queue lengths are variable
–  Depends on incoming rate and service time
!  Context switch and scheduling delay
(CPU queue ) are variable
–  Depends on process concurrency & CPU load
!  Hence: time in queues can vary under load
Performance of immediate message
transfers depends practically on minimizing
queue and context switch time

9/30/13&
11&
Agenda
!  Introduction
!  Fundamentals
!  Testing
Application Considerations
vixencomet
How to avoid “Write Hot Spots” in applications – part 1
racdb1_3 racdb1_4
!  Frequent transactional changes to the same
data blocks in all instances may result in “write
hot spots”
–  In 99% of OLTP performance issues,
write hot spots occur on indexes
!  Block with pending changes may be
“pinged” by other instance
!  Pending redo must be written to log
before the block can be transferred
!  Only for very frequently modified data
!  Latency for a deferred block transfer
becomes dependent on delay for log IO
!  LGWR and LMS interact asynchronously
NUM
INC1()
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC

9/30/13&
12&
vixencomet
racdb1_3 racdb1_4
!  Use non-ordered & cached sequences if
sequence is used to generate primary key
–  ALTER SEQUENCE S1 … CACHE 10000+
!  Symptoms if not cached:
–  Enq; SQ – contention
–  Row cache lock
!  Ordered Sequences
–  Do not scale well in Oracle RAC
–  Symptoms:
!  enq: SV – contention / DFS lock handle
–  Solution: Use them only on one instance
in active-passive configuration
!  Create multiple per application and route
NUM
INC1()
Sequence
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
vixencomet
racdb1_3 racdb1_4
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
!  Possible:
–  Consolidate applications to use only
one server and route via services
!  Optimize log flush :
–  Place redo logs on fast storage
if performance critical; e.g. SSDs
–  Separate disks for logs from other IO busy disks
!  Implemented in 11.2.2.4 of Exadata and
Oracle Database Appliance by default
(Smart Logs and SSDs, respectively)
!  Schema tuning only involves minimal
modification and is the preferred option

9/30/13&
13&
vixencomet
racdb1_3 racdb1_4
!  Global hash partitioned indexes
!  Locally partitioned indexes
–  Both solutions achieve
better cache locality
!  Drop unused indexes
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
vixencomet
How to avoid Resource Contention in applications
racdb1_3 racdb1_4
Connection
Pool
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
!  Foreground processes are in time-share class
!  Scheduling delays on high context switch rates
on busy systems may increase the variation in
the cluster traffic times
!  Latch and mutex contention can cause priority
inversion issues for critical background procs.
!  More processes imply higher memory
utilization and higher risk of paging
!  Control the number of concurrent processes
–  Use connection pooling
–  Avoid connection storms
(pool and process limits )
!  Ensure that load is well-balanced over nodes

9/30/13&
14&
vixencomet
Optimize memory locally to prevent cluster-wide impact
racdb1_3 racdb1_4
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
!  Avoid memory pressure!
–  Paging and Swapping activity on one node
affects performance on all nodes
–  Severe Paging and Swapping activity on
one node can cause instance evictions
!  #1 cause for service disruptions in clusters
!  Must use Huge pages for SGA (Linux)
–  Saves memory for page tables
–  Pins pages for SGA
!  Use Memory Guard
–  QoS feature – available in monitoring only mode
–  Prevents new connections from coming in to a
server that is already under memory pressure
Configure Huge Pages for Oracle RAC
!  Use Huge pages for SGA (Linux)
–  Dramatic reduction in memory for page tables
–  SGA pages pinned in memory
!  More information:
–  My Oracle Support note 361323.1 –
HugePages on Linux: What It Is... and What It Is Not...
–  My Oracle Support note 401749.1 – Shell Script to
Calculate Values Recommended Linux HugePages /
HugeTLB Configuration
!  Engineered systems provide templates for
pre-configuration of huge pages for the SGA

9/30/13&
15&
vixencomet
What to avoid in any case …
racdb1_3 racdb1_4
!  Do not use (named) pipes
–  A pipe on one server may not exist on the other
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
Consolidation Tips and Tricks
vixencomet
What to consider when using more than one instance per server – part 1
racdb2_1 racdb2_2
racdb1_3
Data Structures
Concurrency
Parallelism
Processes
Memory Allocation
Load Calculation
fx(CPU_COUNT) 2.  Use Instance Caging /
set CPU_COUNT
3.  Number of real-time
processes needs to be
taken into consideration
racdb1_4
1.  Manage Memory Carefully
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
!  More details:
–  http://www.oracle.com/technetwork/database/focus-areas/
database-cloud/database-cons-best-practices-1561461.pdf

9/30/13&
16&
Consolidation Tips and Tricks
What to consider when using more than one instance per server – part 2
1.  Switch to Oracle Multitenant
vixen srvpool: frontoffice
dancer srvpool: backofficedasher srvpool: frontoffice
Oracle GI | HUB Oracle GI | HUB
Oracle GI | HUB
Oracle RAC Oracle RAC
Oracle RAC
raccdb1
raccdb1_3
raccdb1_4 raccdb1_2
comet srvpool: backoffice
Oracle GI | HUB
Oracle RAC
raccdb1_1
vixencomet
Summary
racdb1_3 racdb1_4
1.  Focus on the INSERT, UPDATE and DELETE
intensive indexes and tablespace (READS will
always scale well)
2.  Random access to database is not an issue,
frequently modified small tables are interesting
3.  A small subset of statements and objects
causes 90% of performance issues
4.  Standard SQL and schema tuning
solves > 80% of performance problems
5.  Standard common sense system
tuning practices should be applied
6.  Almost everything can be scaled out quickly
with load-direction and load balancing via
services
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC

9/30/13&
17&
Agenda
!  Introduction
!  Fundamentals
!  Testing
Engineered Systems
Optimized and already tested
Oracle Exadata
Database Machine
Oracle Database
Appliance
Quarter Rack
Half Rack
Full Rack
• IB Interconnect
• Low Overhead IPC Prot.
Smart Logs or
SSDs for log files
Preconfigured

9/30/13&
18&
Exadata Smart Flash Log
Accelerate Transaction Response Time Using Flash
!  Uses Flash for Database Logs in a clever way
–  Flash is fast but has slow outliers
!  Smart Flash Log feature transparently
uses Flash as a parallel write cache to
disk controller cache
–  Whichever write completes first wins (disk or flash)
!  Better response time and more throughput
!  Uses almost no flash capacity (0.1% of capacity)
!  Automatically activated with 11.2.2.4.2
Smart Log
Enabled
Default
• Choppy
Response
• High Outliers
• 3x faster
response
• Much lower
outliers
Oracle Real Application Testing
Gather and deploy workload on any database type…
ERP
CRM
DW

9/30/13&
19&
Standardize on Oracle RAC (One Node)
Here: to simplify Oracle RAC testing …
Oracle GIDBA
DBA
Oracle GIDBA
DBA
DBA
Oracle RAC One Node
Oracle RAC
Flexibility
Scalability
Re-configuration
to enable cluster
Online
upgrade
DBA
Oracle Restart
Oracle GI
Standardize on Oracle RAC (One Node)
Oracle RAC testing made simple:
1.  Standardize on Oracle RAC (One Node)
2.  Run tests on Oracle RAC One Node instance
–  RAC One Node behaves basically like a single instance
3.  Online Upgrade to Oracle RAC for testing only
4.  Run tests on Oracle RAC
5.  Decide whether the application provides service goals
when load balanced over multiple active instances
–  Identify bottlenecks and tuning actions as required
To simplify Oracle RAC testing
vixencomet
RON RON_2
Online
upgrade
RON_1
AWR
---
ASH
AWR
---
ASH
Decide
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC

9/30/13&
20&
Agenda
!  Introduction
!  Fundamentals
!  Testing
Identifying Issues
dasher dancer
vixencomet
Immediate Global Cache Access – no contention
racdb1_2
racdb1_3
racdb1_1
racdb1_4
Shadow Process LMS
1.  Shadow process on node dasher
requests block from remote instance
2.  LMS on remote instance sends block or grant
3.  Shadow process / instance receives block
2
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
1
3

9/30/13&
21&
Identifying Issues
No contention Global Cache Access – how it looks in AWR
Accurate average:
109 µsecs
• Latency in Cluster
has small impact
• Average
Performance is good
!  Factors Affecting Performance of
Immediate Global Cache Access
–  Machine Load
!  Process concurrency for CPU
!  Scheduling
!  CPU utilization
–  Interconnect Bandwidth
!  Total bandwidth utilization for the database(s)
–  LMS processes
!  Real time
!  CPU busy
!  No application tuning required
Identifying Issues
dasher dancer
vixencomet
Global Cache Access – with application contention
racdb1_2
racdb1_3
racdb1_1
racdb1_4
1.  Shadow process on node dasher has a local cache
miss and tries getting data from another instance
2.  LMS on node vixen receives the request
3.  LMS on node vixen posts the LGWR to write the redo
for the data as the data buffer is dirty in “vixen’s cache”
4.  LGWR on node vixen performs the log IO
5.  Once the log IO is complete, LMS picks up the buffer
handle and sends the block directly to the requesting
process on node dasher.
6.  The shadow process on node dasher
wakes up and continues with the data
The Redo log IO latency will dominate the roundtrip time.
(block is busy)
2
Shadow Process LMS LGWR
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
Oracle RAC
Oracle GI Oracle GI
Oracle RAC
6
4
1
5
3

9/30/13&
22&
Identifying Issues
Global Cache Access with application contention – how it looks in AWR
!  Factors Affecting Performance
with Application Contention on Data
–  Log File IO latency
–  LGWR responsiveness
!  Schema tuning may be required
–  If the application response time or
throughput do not meet objectives
Impact of
Application
Contention
Index
Contention
Block on the way
from another
instance
Block on the way
from another
instance
Transfer delayed
by log flush on
other node(s)
Identifying Issues
SQL and Schema Optimization: Identifying SQL incurring highest Cluster Wait Time
Indexes with High Contention, 1 accounting for 84%

9/30/13&
23&
Quick Application Fixes
vixencomet
Without having to change the application
racdb1_3 racdb1_4
!  Indexes with right-growing characteristics
–  Cache sequence numbers per instance
–  Hash or range partition table with LOCAL indexes
!  Frequent updates of “small” and compact tables
–  Reduce block size ( 2K ) and row
density of blocks (PCTFREE 99 )
!  Non-Ordered sequences
–  Large caches:
ALTER SEQUENCE S1 … CACHE 10000
!  Ordered Sequences
–  Do not scale well in RAC if frequently used
–  Shard application objects and its objects
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
Identifying Issues
dasher
vixencomet
Cause and Effect are distributed – Example: Log-File IO problem
racdb1_3
racdb1_1
racdb1_4
!  Example:
–  Node dasher sees
!  gc current block busy
!  gc buffer busy acquire
!  gc buffer busy release
!  The root cause lies in the storage system
–  Log writes on nodes comet and vixen are slow
–  Disk Bottleneck
–  SCSI queue length
–  Intermittent controller failures
A
Shadow Process LGWR
B
gc current block busy
gc buffer busy acquire
gc buffer busy releaseOracle GI
Oracle RAC
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC

9/30/13&
24&
Identifying Issues
Cause and Effect are distributed – Example: LGWR stall
Shadow Process LGWR
comet
racdb1_3B
Enqueue: CF –
contention (LGWR)
Oracle GI
Oracle RAC
dasher
racdb1_1
A
gc current block busy
gc buffer busy acquire
gc buffer busy releaseOracle GI
Oracle RAC
!  Example:
–  Node dasher sees
!  gc current block busy
!  gc buffer busy acquire
!  gc buffer busy release
!  The root cause is a blocked LGWR
–  LGWR waits for enq: CF – contention which
may be held by another background process
–  A background process may hold a CF
enqueue while updating the control file
Identifying Issues
Cause and Effect are distributed – How to read the Global Impact
dasher
racdb1_1
Local sessions
waiting for
transfer
Block pinged out;
sessions waiting
for its return
Transfer delayed
by log flush on
other node(s)
Global cache wait events:
35% significant
# higher than expected
comet
racdb1_3
Variance and Outliers indicate
That IO to the log file disk
group affects performance
In the cluster
26.1 / 73.1

9/30/13&
25&
Agenda
!  Introduction
!  Fundamentals
!  Testing
Monitoring Essentials
dasher dancer
vixencomet
Use Cluster Health Monitor for OS (CHM/OS)
1.  First failure diagnostics: Collects as much
system metrics and data as feasible
2.  Collects system metrics on all
nodes in the cluster synchronized
3.  Persistent storage of all
collected data in a database
4.  Integrated with Oracle Grid Infrastructure
starting with 11.2.0.2 (12c: uses GIMR)
5.  Robust collection in situations
of resource starvation
6.  Data dump or visualization
LOGGERD
sysmond
sysmond
Oracle GI Oracle GI
Oracle GI Oracle GI
sysmond
sysmond
sysmond
LOGGERD

9/30/13&
26&
Log Collection Essentials
dasher dancer
vixencomet
Runtime and Emergency Data Collections
!  Save database statistics
–  For baselines and reference: export AWR
!  For diagnostics:
–  Always get AWR and ASH reports
!  For emergencies and “last gasp”
–  get ASH dump (e.g. oradebug ashdunp level 0)
!  CHM/OS
–  oclumon dumpnodeview -n pecdb08 pecdb09 -s
"2011-09-30 15:00:00" -e "2011-09-30 15:30:00" -v
–  oclumom –manage –report <ret. time in secs>
!  CSS logs from all nodes
–  Node and instance evictions
–  Interconnect issues (e.g. gc blocks lots )
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
LOGs
LOGs
LOGs
LOGs
dasher
comet
Use Trace File Analyzer (TFA) to collect Data
!  Goals
–  Improved comprehensive first
failure diagnostics collection
–  Efficient collection, packaging
and transfer of data for customers
–  Reduce round trips between
customers and Oracle
–  Supports 10.2, 11.1, 11.2 and above
–  Included in the 11.2.0.4
patch set and future versions
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
LOGs
LOGs
TFA

9/30/13&
27&
dasher
comet
Use Trace File Analyzer (TFA) to collect Data
!  Approach
–  Collect for all relevant components
(OS, Grid Infrastructure, ASM, RDBMS)
–  One command to collect all information
–  Prune large files based on temporal criteria
–  Collect time relevant IPS (incident)
packages on RAC nodes
–  Collect time relevant CHM/OS,
OSWatcher data on all Oracle RAC nodes
–  On-demand (default) and
Event Driven diagnostic collections
!  TFA Collector – MOS note ID 1513912.1
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
LOGs
LOGs
TFA
TFA – Usage Example
Sending diagcollect request to host : myhost2
Getting list of files satisfying time range [Tue Sep 03 14:17:43 PDT 2013, Tue Sep 03 18:17:43 PDT 2013]
myhost1: Zipping File: /opt/oracle/oak/oswbb/archive/oswiostat/myhost1_iostat_13.09.03.1500.dat.gz
myhost1: Zipping File: /u01/app/oracle/diag/rdbms/bill/bill1/trace/alert_bill1.log
Trimming file : /u01/app/oracle/diag/rdbms/bill/bill1/trace/alert_bill1.log with original file size : 109kB
myhost1: Zipping File: /opt/oracle/oak/oswbb/archive/oswtop/myhost1_top_13.09.03.1500.dat.gz
myhost1: Zipping File: /opt/oracle/oak/log/myhost1/oak/oakd.log
Trimming file : /opt/oracle/oak/log/myhost1/oak/oakd.log with original file size : 9.2MB
myhost1: Zipping File: /u01/app/11.2.0.3/grid/log/myhost1/gipcd/gipcd.log
myhost1: Zipping File: /u01/app/11.2.0.3/grid/log/myhost1/agent/ohasd/oraagent_grid/oraagent_grid.log
Trimming file : /u01/app/11.2.0.3/grid/log/myhost1/agent/ohasd/oraagent_grid/oraagent_grid.log with original filesize 4.3MB
myhost1: Zipping File: /var/log/messages
…Truncated for brevity
myhost1: Zipping File: /opt/oracle/oak/oswbb/archive/oswslabinfo/myhost1_slabinfo_13.09.03.1800.dat
Collecting ADR incident files...
Total Number of Files checked : 10543
Total Size of all Files Checked : 3.9GB
Number of files containing required range : 68
Total Size of Files containing required range : 129MB
Number of files trimmed : 10
Total Size of data prior to zip : 144MB
Saved 63MB by trimming files
Zip file size : 8.6MB
Total time taken : 47s.
Logs are collected to:
/opt/oracle/tfa/tfa_home/repository/collection_Tue_Sep_3_18_17_24_PDT_2013_node_all/myhost1.tfa_Tue_Sep_3_18_17_24_PDT_2013.zip
/opt/oracle/tfa/tfa_home/repository/collection_Tue_Sep_3_18_17_24_PDT_2013_node_all/myhost2.tfa_Tue_Sep_3_18_17_24_PDT_2013.zip
OS Watcher files
Pruning
47 seconds! – 1 command, 2 nodes, 4 databases, ASM, Clusterware, OS
Relevant files only
$ sudo ./tfactl diagcollect
144MB pruned and compressed down to 8.6MB
One simple command
ADR Incident files

9/30/13&
28&
Analysis Essentials
Time Series Analysis – AWR Reports are not enough due to average…
Occurrence of an interconnect
problem ~18 secs avg !!!!
!  Solution
–  ASH reports
Every
1 hour
or
out-of-space
AWR
Circular buffer
in SGA
(2MB per CPU)
DBA_HIST_ACTIVE_SESS_HISTORYV$ACTIVE_SESSION_HISTORY
Session
state
objects
MMON Lite
(MMNL)
V$SESSION
V$SESSION_WAIT
Write 1
out of 10
samples
Variable
length rows
Direct path
inserts
Analysis Essentials
Time Series Analysis – AWR Reports are not enough …
2011-10-03 15:25:25.662: [ CSSD][39]clssnmPollingThread:
node pecdb07 (1) is impending reconfig, flag 199692, misstime
15626
node 2, pecdb08, ninfmisstime 267, misstime 267, skgxnbit 4,
vcwmisstime 0, syncstage 0
local diskTimeout set to 27000 ms, remote disk timeout set to
27000, impending reconfig status(1)
2011-10-03 15:25:25.663: [ CSSD]
[35]clssnmvDHBValidateNCopy: node 1, pecdb07, has a disk
HB, but no network HB, DHB has rcfg 207057058, wrtcnt,
17468860, LATS 534607234, lastSeqNo 17462672, uniqueness
1317671435, timestamp 1317673524/229299074
...
2011-10-03 15:30:21.811: [ CSSD]
[35]clssnmvDHBValidateNCopy: node 1, pecdb07, has a disk
HB, but no network HB, DHB has rcfg

9/30/13&
29&
Analysis Essentials
Temporary Cluster Wait Spike
SQL with
High Global
Cache Wait
Time
Spike in Global Cache
Response Time
Agenda
!  Introduction
!  Fundamentals
!  Testing

9/30/13&
30&
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template59
Graphic Section Divider
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template60

Oracle RAC 12c Practical Performance Management and Tuning OOW13 [CON8825]

More Related Content

What's hot (20)

Viewers also liked (6)

Similar to Oracle RAC 12c Practical Performance Management and Tuning OOW13 [CON8825] (20)

More from Markus Michalewicz (20)

Recently uploaded (20)

Oracle RAC 12c Practical Performance Management and Tuning OOW13 [CON8825]