Troubleshooting Tips and Tricks for Database 19c - EMEA Tour Oct 2019

Sandesh Rao
VP AIOps , Autonomous Database
19 Troubleshooting Tips and
Tricks for Database 19c
@sandeshr
https://www.linkedin.com/in/raosandesh/
https://www.slideshare.net/SandeshRao4

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not
a commitment to deliver any material, code, or functionality, and should not be
relied upon in making purchasing decisions. The development, release,
timing, and pricing of any features or functionality described for Oracle’s products
may change and remains at the sole discretion of Oracle Corporation.

oracle.com/gbtour
New Free Tier Always Free
Oracle Cloud Infrastructure
Services you can use for unlimited time
30-Day Free Trial
Free credits you can use for more services
+

1 – Best Practices
Automation and visualization

Combined installation of TFA, ORAchk and EXAchk
TFA and ORAchk/EXAchk are installed via a single platform specific installation
available at doc: 2550798.1
Updated on a quarterly basis
Included within Database & Grid Infrastructure install
Included within Release Updates (RUs)

Automatic notifications
ORAchk / EXAchk results
older than two weeks are
automatically purged
Full ORAchk / EXAchk
compliance run at
3am once a week
TFA automatically monitors
for significant events,
collects, analyses
diagnostics then notifies you
Most impactful compliance
checks from
ORAchk/EXAchk are run
2am every day
tfactl <orachk|exachk> -autostart

Automatically upload all collection results, company wide to AHF Service
TFA & ORAchk/EXAchk collections can be
automatically uploaded to AHF Service
AHF Service
Processes and analyses collection results
Can be used to create custom checks
(user defined)
RPM download to be available soon
tfactl -setserviceupload
tfactl <orachk|exachk> -setserviceupload

Troubleshooting Tips and Tricks for Database 19c - EMEA Tour Oct 2019

2 – ADDM in a
Multitenant Environment

ADDM in a multitenant environment
Starting with Oracle Database 12c, ADDM is enabled by default in the root
container of a multitenant container database (CDB)
You can also use ADDM in a pluggable database (PDB)
In a CDB, ADDM works in the same way as it works in a non-CDB
ADDM analysis is performed each time an AWR snapshot is taken on a CDB
root or a PDB
ADDM does not work in a PDB by default, because automatic AWR snapshots
are disabled

ADDM in a multitenant environment
To enable ADDM in a PDB:
Set the AWR_PDB_AUTOFLUSH_ENABLED initialization parameter to TRUE in
the PDB using the following command:
Set the AWR snapshot interval greater than 0 in the PDB using the command as
shown in the following example:
Results on a PDB provide only PDB-specific findings and recommendations
SQL> ALTER SYSTEM SET AWR_PDB_AUTOFLUSH_ENABLED=TRUE;
SQL> EXEC
dbms_workload_repository.modify_snapshot_settings(inter
val=>60);

3 – Analyze logs and
look for errors

Investigate logs and look for errors
tfactl analyze -since 1d
INFO: analyzing all (Alert and Unix System Logs) logs for the last 1440
minutes...
...
Unique error messages for last ~1 day(s)
Occurrences percent server name error
----------- ------- -------------------- -----
1 100.0% myserver1 Errors in file
/u01/oracle/diag/rdbms/orcl2/orcl2/trace/orcl2_ora_12272.trc
(incident=10151):
ORA-00600: internal error code, arguments: [600], [], [], [], [], [], [], [],
[], [], [], []
Incident details in:
/u01/oracle/diag/rdbms/orcl2/orcl2/incident/incdir_10151/orcl2_ora_12272_i101
51.trc
...

tfactl analyze -search "ORA-04031" -last 1d
INFO: analyzing all (Alert and Unix System Logs) logs for the last 1440
minutes...
...
Matching regex: ORA-04031
Case sensitive: false
Match count: 1
[Source: /u01/oracle/diag/rdbms/orcl2/orcl2/trace/alert_orcl2.log, Line: 1941]
Sep 15 12:09:05 2019
Errors in file /u01/oracle/diag/rdbms/orcl2/orcl2/trace/orcl2_ora_6982.trc
(incident=7665):
ORA-04031: unable to allocate bytes of shared memory ("","","","")
Incident details in:
/u01/app/oracle/diag/rdbms/orcl2/orcl2/incident/incdir_7665/orcl2_ora_6982_i76
65.trc
...

Examples
tfactl analyze -since 5h
#Show summary of events from alert logs,
system messages in last 5 hours
tfactl analyze -comp os -since 1d
#Show summary of events from system
messages in last 1 day
tfactl analyze -search "ORA-" -since 2d
#Search string ORA- in alert and system
logs in past 2 days
tfactl analyze -search "/Starting/c" -
since 2d
#Search case sensitive string "Starting"
in past 2 days
tfactl analyze -comp os -for "Feb/24/2019
11" -search "."
#Show all system log messages at time
Feb/24/2019 11
tfactl analyze -comp osw -since 6h
#Show OSWatcher Top summary in last 6
hours
tfactl analyze -comp oswslabinfo -from
"Feb/26/2019 05:00:01" -to "Feb/26/2019
06:00:01"
#Show OSWatcher slabinfo summary for
specified time period
tfactl analyze -since 1h -type generic
#Analyze all generic messages in last one
hour

$ ./tfactl analyze -type generic -since 7d
INFO: analyzing all (Alert and Unix System Logs) logs for the last 10080 minutes...
...
Total message count: 54,807, from 28-Jan-2019 04:26:28 PM PST to
03-Mar-2019 02:41:34
Messages matching last ~7 day(s): 3,139, from 24-Feb-2019 02:46:23 PM PST to
03-Mar-2019 02:41:34
last ~7 day(s) generic count: 3,139, from 24-Feb-2019 02:46:23 PM PST to
03-Mar-2019 02:41:34
last ~7 day(s) unique generic count: 94
Message types for last ~7 day(s)
Occurrences percent server name type
----------- ------- -------------------- -----
3,139 100.0% myhost1 generic
...

Unique generic messages for last ~7 day(s)
Occurrences percent server name generic
----------- ------- -------------------- -----
1,504 47.9% myhost1 : [crflogd(13931)]CRS-9520:The storage of Grid
Infrastructure Managem...
487 15.5% myhost1 : [crflogd(13931)]CRS-9520:The storage of Grid
Infrastructure Managem...
336 10.7% myhost1 myhost1 smartd[13812]: Device: /dev/sdv, SMART
Failure: FAILURE...
336 10.7% myhost1 myhost1 smartd[13812]: Device: /dev/sdag, SMART
Failure: FAILURE ...
103 3.3% myhost1 myhost1 last message repeated 9 times
103 3.3% myhost1 myhost1 kernel: oracle: sending ioctl 2285 to a
partition!
...snipping for brevity...

Pattern match search output
tfactl analyze -search "ORA-" -since 7d
...
[Source: /u01/app/oracle/diag/rdbms/ratoda/RATODA1/trace/alert_RATODA1.log, Line:
9494]
Feb 25 22:00:02 2014
Errors in file
/u01/app/oracle/diag/rdbms/ratoda/RATODA1/trace/RATODA1_j003_10948.trc:
ORA-12012: error on auto execute of job "ORACLE_OCM"."MGMT_CONFIG_JOB_2_1"
ORA-29280: invalid directory path
ORA-06512: at "ORACLE_OCM.MGMT_DB_LL_METRICS", line 2436
ORA-06512: at line 1
End automatic SQL Tuning Advisor run for special tuning task
"SYS_AUTO_SQL_TUNING_TASK”
...

OS Watcher top data
tfactl analyze -comp osw -since 6h
...
statistic: t first highest (time) lowest (time) average non zero 3rd last 2nd last last trend
top.cpu.util.id: % 98.0 99.7 @10:35AM 72.8 @03:11PM 97.3 2,059 95.2 96.8 96.0 -2%
top.cpu.util.st: % 0.1 0.1 @09:14AM 0.0 @09:14AM 0.0 889 0.0 0.0 0.0 -100%
top.cpu.util.us: % 0.1 8.8 @11:31AM 0.0 @09:14AM 0.6 1,966 4.3 0.8 3.4 3300%
top.cpu.util.wa: % 1.7 18.7 @03:11PM 0.1 @10:35AM 1.1 2,059 0.3 0.4 0.4 -76%
top.loadavg.last01min: 1.17 3.12 @09:44AM 0.07 @12:45PM 0.93 1,823 0.31 0.26 0.22 -81%
top.loadavg.last05min: 0.94 2.26 @09:44AM 0.27 @12:45PM 0.93 1,823 0.82 0.79 0.77 -18%
top.loadavg.last15min: 0.79 1.60 @09:46AM 0.44 @01:18PM 0.92 1,823 0.96 0.95 0.94 18%
top.mem.buffers: k 808232 808388 @09:41AM 785608 @02:57PM 796511 2,093 785744 785744 785744 -2%
top.mem.free: k 1130332 1291344 @10:02AM 927576 @09:43AM 1188576 2,093 1244020 1265248 1265188 11%
top.swap.used: k 47556 48088 @03:00PM 47556 @09:14AM 47828 2,097 48088 48088 48088 1%
top.tasks.running: 1 4 @12:04PM 1 @09:14AM 1 1,996 1 2 2 100%
top.tasks.total: 514 527 @02:57PM 509 @09:18AM 514 1,996 518 521 520 1%
top.tasks.zombie: 0 5 @11:04AM 0 @09:14AM 0 62 0 0 0 n/a
top.users: 5 6 @03:00PM 5 @09:14AM 5 1,823 6 6 6 20%
...

OS Watcher slabinfo data
tfactl analyze -comp oswslabinfo -from "Feb/26/2019 05:00:01" -to "Feb/26/2019
06:00:01"
...
statistic: t first highest (time) lowest (time) average non zero 3rd last 2nd last last trend
slabinfo.acfs_ccb_cache.active_objs: 4 38 @05:52AM 0 @05:01AM 10 294 3 1 8 100%
slabinfo.inet_peer_cache.active_objs: 23 39 @05:59AM 23 @05:00AM 23 351 23 23 39 69%
slabinfo.sigqueue.active_objs: 385 768 @05:28AM 285 @05:27AM 554 351 712 621 577 49%
slabinfo.skbuff_fclone_cache.active_objs: 55 133 @05:51AM 11 @05:20AM 69 351 56 77 70 27%
slabinfo.names_cache.active_objs: 126 180 @05:00AM 110 @05:23AM 146 351 171 166 156 23%
slabinfo.sgpool-8.active_objs: 135 228 @05:31AM 59 @05:11AM 152 351 180 165 157 16%
slabinfo.UDP.active_objs: 568 675 @05:28AM 492 @05:17AM 597 351 630 596 626 10%
slabinfo.size-8192.active_objs: 174 209 @05:36AM 160 @05:14AM 181 351 205 187 188 8%
slabinfo.task_delay_info.active_objs: 1477 1856 @05:28AM 1334 @05:57AM 1574 351 1529 1411 1579 6%
slabinfo.pid.active_objs: 1608 1980 @05:29AM 1452 @05:21AM 1678 351 1564 1487 1689 5%
slabinfo.blkdev_requests.active_objs: 720 880 @05:04AM 651 @05:54AM 745 351 707 736 761 5%
slabinfo.ip_dst_cache.active_objs: 1497 1800 @05:28AM 1279 @05:36AM 1517 351 1594 1466 1560 4%
slabinfo.sock_inode_cache.active_objs: 2168 2329 @05:11AM 2106 @05:56AM 2225 351 2322 2278 2232 2%
...

4 – Keep track of the attributes
of important files pre-post
patching

Track attribute changes on important files
Start tracking using –fileattr start
Automatically discovers Grid Infrastructure and Database directories and files
Prevent discovery using –excludediscovery
Further configure the list of monitored directories using –includedir
tfactl <orachk|exachk> -fileattr start -includedir "/root/myapp/config"
...
List of directories(recursive) for checking file attributes:
/u01/app/oradb/product/11.2.0/dbhome_11203
/root/myapp/config
orachk has taken snapshot of file attributes for above directories at:
/orahome/oradb/orachk/orachk_mysrv21_20170504_041214

Compare current attributes against first snapshot using –fileattr check
When checking, use the same include/exclude arguments you started with
tfactl <orachk|exachk> -fileattr check -includedir "/root/myapp/config”
...
List of directories(recursive) for checking file attributes:
/root/myapp/config
Checking file attribute changes...
"/root/myapp/config/myappconfig.xml" is different:
Baseline : 0644 oracle root
/root/myapp/config/myappconfig.xml
Current : 0644 root root
/root/myapp/config/myappconfig.xml
...

Automatically proceeds to run compliance checks after file attribute checks
Only run attribute checks by using -fileattronly
File Attribute Changes are shown in HTML report output

ORAchk | EXAchk email notification
Automatically running
Critical checks once a day at 2am
Full checks once a week at 3am Sunday
You only need to configure your email for notification – if not already done during install
tfactl <orachk|exachk> -set “NOTIFICATION_EMAIL=SOME.BODY@COMPANY.COM

Critical event notification
TFA can send email notification when faults are detected – only required if not set during install
Notification for all problems:
Notification for all problems on database owned by oracle user:
Optionally configure an SMTP server:
Confirm email notification work:
tfactl set notificationAddress=some.body@example.com
tfactl set notificationAddress=oracle:another.person@example.com
tfactl set smtp
tfactl sendmail <email_address>

Copyright © 2019 Oracle and/or its affiliates.
Event: ORA-29770
Event time: Fri Sep
13 07:13:09 PDT 2019
File containing
event:
/u01/app/oracle/diag
/rdbms/orcl/orcl/tra
ce/alert_orcl.log
Logs will be
collected at:
/opt/oracle.ahf/data
/repository/auto_srd
c_ORA-
29770_2019_07_18:09_
myserver1.zip

Symptom
LCK0 (ospid:NNNN)
has not called a
wait for <n_secs>
secs.
Call stack:
ksedsts <-
kjzdssdmp <-
kjzduptcctx <-
kjzdicrshnfy <-
ksuitm <-
kjgcr_KillInstance
<- kjgcr_Main <-
kjfmlmhb_Main <-
ksbrdp

Action
Apply the one-off
patch 18795105 to
resolve this issue
For further
information see
Doc :1998445.1 and
Doc :18795105.8
Cause
Instance crash due
to ORA-29770 LCK0
hung

Evidence
Orcl_lmhb_23242.trc
(15):
ksedsts()+465<-
kjzdssdmp()+267<-
kjzduptcctx()+232<-
kjzdicrshnfy()+63<-
ksuitm()+5570<-
kjgcr_KillInstance()
+125
alert_orcl.log(140):
ORA-29770: global
enqueue process LMS0
(OSID 11912) is hung
for more than 70
seconds

6 – Self Analysis in MOS
using TFA uploads

One command SRDC
tfactl diagcollect –srdc <srdc_type>
Scans system to identify recent events
Once the relevant event is chosen, proceeds with diagnostic collection
tfactl diagcollect -srdc ORA-00600
Enter the time of the ORA-00600 [YYYY-MM-DD HH24:MI:SS,<RETURN>=ALL] :
Enter the Database Name [<RETURN>=ALL] :
1. Sep/07/2019 05:29:58 : [orcl2] ORA-00600: internal error code,
arguments: [600], [], [], [], [], [], [], [], [], [], [], []
2. Aug/16/2019 06:55:08 : [orcl2] ORA-00600: internal error code,
arguments: [600], [], [], [], [], [], [], [], [], [], [], []
Please choose the event : 1-2 [1]
Selected value is : 1 ( Sep/07/2019 05:29:58 )

One command SRDC
All required files are identified
Trimmed where applicable
Package in a zip ready to provide to support
...
2019/09/07 06:14:24 EST : Getting List of Files to Collect
2019/09/07 06:14:27 EST : Trimming file :
myserver1/rdbms/orcl2/orcl2/trace/orcl2_lmhb_3542.trc with original file
size : 163MB
...
2019/09/07 06:14:58 EST : Total time taken : 39s
2019/09/07 06:14:58 EST : Completed collection of zip files.
...
/opt/oracle.ahf/data/repository/srdc_ora600_collection_Tue_Sep_7_06_14_17
_EST_2017_node_local/myserver1.tfa_srdc_ora600_Tue_Sep_7_06_14_17_EST_201
9.zip

Sanitize sensitive
information

Sanitize or mask sensitive information
Sensitive information can be hidden from diagnostics
Machine learning algorithms determine sensitive data like:
Host names
IP addresses
MAC addresses
Oracle Database names
Tablespace names
Service names
Ports
Operating system user names

Sanitize or mask sensitive information
Add –sanitize or –mask to any command
–sanitize replaces a sensitive value with random characters
myhost123 >>>> JnsF3km9
–mask replaces a sensitive value with a series of ‘X’
myhost123 >>>> XXXXXXXX

Sanitized hostname
Check ID
tfactl orachk –preupgrade -sanitize

Sanitized hostname
Sanitized hostname
tfactl orachk –preupgrade -sanitize

Reverse map the sanitization
tfactl orachk -rmap qzh024703246tsa1
TFA using ORAchk : /opt/oracle.ahf/orachk/orachk
________________________________________________________________________________
| Entity Type | Substituted Entity Name | Original Entity Name |
________________________________________________________________________________
| hostname | qzh024703246tsa1 | myserver1 |
________________________________________________________________________________

Sanitized hostname
Repair command
Check ID

Repair command
Check IDCheck ID
Repair command

Understand what the repair command does
Understand what the repair command will do with:
tfactl orachk -showrepair 8300E0A2FFE48253E053D298EB0A76CC
TFA using ORAchk : /opt/oracle.ahf/orachk/orachk
Repair Command:
currentUserName=$(whoami)
if [ "$currentUserName" = "root" ]
then
repair_report=$(rpm -e stix-fonts 2>&1)
else
repair_report="$currentUserName does not have priviedges to run
$CRS_HOME/bin/crsctl set resource use 1"
fi
echo -e "$repair_report"

Run the repair command
Run the checks again and repair everything that fails
Run the checks again and repair only the specified checks
Run the checks again and repair all checks listed in the file
tfactl orachk -repaircheck all
tfactl orachk -repaircheck <check_id_1>,<check_id_2>
tfactl orachk -repaircheck <file>

Pre and post upgrade
compliance checking

ORAchk/EXAchk provides a single source for all upgrade checks
ORAchk checks
EXAchk checks
Database
AutoUpgrade checks
Cluster Verification
Utility (CVU) checks
Compare
Contrast
Combine
Consolidate
Resulting ORAchk / EXAchk
checks

ORAchk/EXAchk provides a single source for all upgrade checks
To check an environment before upgrading run:
To check an environment after upgrade run:
tfactl <orachk|exachk> –u -o pre
tfactl <orachk|exachk> –u -o post

8 – Restrict the maximum size
of files collected for data upload

Optionally restrict the collection of excessively large files
During collection TFA will verify the size of each file to be collected
Files greater than MaxFileCollectSize MB trimmed to last 1,000 lines
skipped_files.txt in collection shows all the files, which were too large
tfactl set MaxFileCollectSize <size_mb>

Optionally restrict the collection of excessively large files
tfactl print config
.------------------------------------------------------------------------------------.
| myserver |
+-----------------------------------------------------------------------+------------+
| Configuration Parameter | Value |
+-----------------------------------------------------------------------+------------+
| TFA Version | 19.3.0.0.0 |
…
…
| Max Collection Size of Core Files (MB) | 500 |
| Max File Collection Size (MB) | 5120 |
| Minimum Free Space to enable Alert Log Scan (MB) | 500 |
| Time interval between consecutive Disk Usage Snapshot(minutes) | 60 |
| Age of Purging Collections (Hours) | 12 |
| TFA IPS Pool Size | 5 |
…

9 – REST Interfaces to
programmatically execute
diagnostics

REST service
Install includes
Oracle REST Data
Services (ORDS)
tfactl rest -start https://host:port/ords/<api>
§ -start #Starts TFA REST services
§ -stop #Stops TFA REST services
§ -status #Show current status
§ -dir #Configuration location
§ -port #Port to listen on (default 9090)
§ -user #User to run as (default Grid Owner)

REST service
Start diagnostic
collection with a
POST to API:
<tfactl/diagcollect>
Download collection with a
GET to API:
</tfactl/download/{collectionid}>
Returns JSON with
collectionId:
testuser: {
"collectionId" : "20190901121115slc13lyb",
"zipName" : "TFA_DEF_ZIP_20190901121115",
"tagName" : "TFA_DEF_TAG_20190901121115",
"message" : [ "Diagcollect request will be
processed soon by TFA" ]
}

REST service
Customize diagnostic collection via JSON
POST https://host:port/ords/tfactl/diagcollect
[{
"components": "-database -asm -tns -crs -acfs -install -cfgtools -os",
"timePeriod": "-since n[d|h] | -last n[d|h] | -for date |
-from date -to date",
"tagName": "crs_crash_collection",
"nodeList": "node1,node2",
"options": "-nocopy | -notrim | -silent | -nocores |
-collectalldirs | -collectdir dir1,dir2..."
}]

REST service other key API methods
tfactl/run/alertsummary #Shows summary of events from alert logs
tfactl/run/calog #Shows major events from the cluster event log
tfactl/run/changes #Shows system changes including DB & OS parameters & patches
tfactl/run/events #Reports warnings and errors seen in the logs
tfactl/run/history #Reports history of commands for the tfactl shell
tfactl/orachk/start_client #start orachk/exachk run
tfactl/orachk/profile/<profile1>,<profile2> #run specific orachk/exachk profiles
tfactl/orachk/check/<check_id>,<check_id> #run specific orachk/exachk checks
tfactl/orachk/status/<job_id> #query status of an ORAchk/EXAchk request
tfactl/orachk/download/<job_id> #download results of an ORAchk/EXAchk request

Deploy REST service via Apache Tomcat
1. Deploy the WAR file located at /opt/oracle.ahf/common/jlib/tfa.war to your Tomcat server
2. Change the tfaadmin user password
3. Change the tfarest user password
4. Add the user Tomcat runs as to the TFA access list
curl -k --user tfaadmin:tfaadmin https://host/tfa/tfactl/user/update
‘[{"password" : "some_new_password" }]’
curl -k --user tfarest:tfarest https://host/tfa/tfactl/user/update
‘[{"password" : "some_new_password" }]’
tfactl access add -user <tomcat_user>

10 – Blackout option
for scheduled events

Temporarily restrict automatic collections for specific events
Temporarily prevent automatic collection using tfactl blackout
Can be set for certain targets, events & durations
Do not collect ORA-00600 events on mydb for the next 24hrs (default time)
Do not collect ORA-04031 events on any database for the next hour
Do not collect any events (during patching)
tfactl blackout add -targettype database -target mydb -event "ORA-00600"
tfactl blackout add -targettype database -target mydb -event "ORA-04031"
–timeout 1h
tfactl blackout add -targettype all -event all -target all -timeout 1h
-reason "Disabling all events during patching"

11 - Detect and Collect
using SRDC’s

Some problem areas covered in SRDCs
Database areas
Errors / Corruption
Performance
Install / patching / upgrade
RAC / Grid Infrastructure
Import / Export
RMAN
Transparent Data Encryption
Storage / partitioning
Undo / auditing
Listener / naming services
Spatial / XDB
Other Server Technology
Enterprise Manager
Data Guard
GoldenGate
Exalogic
Full list in documentationAround 100 problem types covered
tfactl diagcollect –srdc <srdc_type>
[-sr <sr_number>]

Manual collection vs TFA SRDC for database performance
Manual method TFA SRDC
1. Generate ADDM reviewing Document 1680075.1 (multiple steps)
2. Identify “good” and “problem” periods and gather AWR
reviewing Document 1903158.1 (multiple steps)
3. Generate AWR compare report (awrddrpt.sql) using “good” and
“problem” periods
4. Generate ASH report for “good” and “problem” periods
reviewing Document 1903145.1 (multiple steps)
5. Collect OSWatcher data reviewing Document 301137.1 (multiple
steps)
6. Collect Hang Analyze output at Level 4
7. Generate SQL Healthcheck for problem SQL id using Document
1366133.1 (multiple steps)
8. Run support provided sql scripts – Log File sync diagnostic
output using Document 1064487.1 (multiple steps)
9. Check alert.log if there are any errors during the “problem”
period
10. Find any trace files generated during the “problem” period
11. Collate and upload all the above files/outputs to SR
1. Run
tfactl diagcollect –srdc dbperf
[-sr <sr_number>]

Automatic Database Log Purge
TFA can automatically purge database logs
Purging automatically removes logs older than 30 days
Configurable with
Purging runs every 60 minutes
Configurable with:
tfactl set manageLogsAutoPurge=ON
tfactl set manageLogsAutoPurgePolicyAge=<n><d|h>
tfactl set manageLogsAutoPurgeInterval=<minutes>

Manual Database Log Purge
TFA can manage ADR log and trace files
tfactl managelogs <options>
–show usage #Show disk space usage per diagnostic directory for both GI and database logs
-show variation –older <n><m|h|d> #Show disk space growth for specified period
-purge –older <n><m|h|d> #Remove ADR files older than the time specified
–gi #Restrict command to only files under the GI_BASE
–database [all | dbname] #Restrict command to only files under the database directory
-dryrun #Use with –purge to estimate how many files will be affected and how much disk
space will be freed by a potential purge command

Understand Database log disk space usage
tfactl managelogs -show usage
...
.---------------------------------------------------------------------------------.
| Grid Infrastructure Usage |
+---------------------------------------------------------------------+-----------+
| Location | Size |
+---------------------------------------------------------------------+-----------+
| /u01/app/crsusr/diag/afdboot/user_root/host_309243680_94/alert | 28.00 KB |
| /u01/app/crsusr/diag/afdboot/user_root/host_309243680_94/incident | 4.00 KB |
| /u01/app/crsusr/diag/afdboot/user_root/host_309243680_94/trace | 8.00 KB |
...
+---------------------------------------------------------------------+-----------+
| Total | 739.06 MB |
'---------------------------------------------------------------------+-----------’
...
Use -gi to only show grid infrastructure

Understand Database log disk space usage
...
.---------------------------------------------------------------.
| Database Homes Usage |
+---------------------------------------------------+-----------+
| Location | Size |
+---------------------------------------------------+-----------+
| /u01/app/crsusr/diag/rdbms/cdb674/CDB674/alert | 1.06 MB |
| /u01/app/crsusr/diag/rdbms/cdb674/CDB674/incident | 4.00 KB |
| /u01/app/crsusr/diag/rdbms/cdb674/CDB674/trace | 146.19 MB |
| /u01/app/crsusr/diag/rdbms/cdb674/CDB674/cdump | 4.00 KB |
| /u01/app/crsusr/diag/rdbms/cdb674/CDB674/hm | 4.00 KB |
+---------------------------------------------------+-----------+
| Total | 147.26 MB |
'---------------------------------------------------+-----------'
Use -database to only show database

Understand Database log disk space usage variations
tfactl managelogs -show variation -older 30d
Output from host : myserver74
------------------------------
2019-09-20 12:30:42: INFO Checking space variation for 30 days
.---------------------------------------------------------------------------------------------.
| Grid Infrastructure Variation |
+---------------------------------------------------------------------+-----------+-----------+
| Directory | Old Size | New Size |
+---------------------------------------------------------------------+-----------+-----------+
| /u01/app/crsusr/diag/asm/user_root/host_309243680_96/alert | 22.00 KB | 28.00 KB |
+---------------------------------------------------------------------+-----------+-----------+
| /u01/app/crsusr/diag/clients/user_crsusr/host_309243680_96/cdump | 4.00 KB | 4.00 KB |
+---------------------------------------------------------------------+-----------+-----------+
| /u01/app/crsusr/diag/tnslsnr/myserver74/listener/alert | 15.06 MB | 244.10 MB |
+---------------------------------------------------------------------+-----------+-----------+
...

Understand Database log disk space usage variations
...
.---------------------------------------------------------------------------.
| Database Homes Variation |
+---------------------------------------------------+-----------+-----------+
| Directory | Old Size | New Size |
+---------------------------------------------------+-----------+-----------+
| /u01/app/crsusr/diag/rdbms/cdb674/CDB674/hm | 4.00 KB | 4.00 KB |
+---------------------------------------------------+-----------+-----------+
| /u01/app/crsusr/diag/rdbms/cdb674/CDB674/trace | 16.63 MB | 146.19 MB |
+---------------------------------------------------+-----------+-----------+
| /u01/app/crsusr/diag/rdbms/cdb674/CDB674/cdump | 4.00 KB | 4.00 KB |
+---------------------------------------------------+-----------+-----------+
| /u01/app/crsusr/diag/rdbms/cdb674/CDB674/incident | 4.00 KB | 4.00 KB |
+---------------------------------------------------+-----------+-----------+
| /u01/app/crsusr/diag/rdbms/cdb674/CDB674/alert | 1.06 MB | 1.06 MB |
'------------------------------------------------------------+-------------+-------------'

Run a database log purge dryrun
tfactl managelogs -purge -older 30d -dryrun
------------------------------
Estimating files older than 30 days
Estimating purge for diagnostic destination "diag/afdboot/user_root/host_309243680_94" for files ~ 2 files deleted , 22.58 KB freed ]
Estimating purge for diagnostic destination "diag/afdboot/user_crsusr/host_309243680_94" for files ~ 2 files deleted , 11.72 KB freed ]
Estimating purge for diagnostic destination "diag/asmtool/user_root/host_309243680_96" for files ~ 2 files deleted , 21.36 KB freed ]
Estimating purge for diagnostic destination "diag/asmtool/user_crsusr/host_309243680_96" for files ~ 3 files deleted , 23.22 KB freed ]
Estimating purge for diagnostic destination "diag/tnslsnr/myserver74/listener" for files ~ 23 files deleted , 225.33 MB freed ]
Estimating purge for diagnostic destination "diag/diagtool/user_root/adrci_309243680_96" for files ~ 73 files deleted , 517.69 KB freed ]
Estimating purge for diagnostic destination "diag/clients/user_crsusr/host_309243680_96" for files ~ 38 files deleted , 17.15 KB freed ]
Estimating purge for diagnostic destination "diag/asm/+asm/+ASM" for files ~ 0 files deleted , 0 bytes freed ]
Estimating purge for diagnostic destination "diag/asm/user_root/host_309243680_96" for files ~ 1 files deleted , 19.52 KB freed ]
Estimating purge for diagnostic destination "diag/asm/user_crsusr/host_309243680_96" for files ~ 1 files deleted , 20.25 KB freed ]
Estimating purge for diagnostic destination "diag/crs/myserver74/crs" for files ~ 40 files deleted , 219.39 MB freed ]
Estimation for Grid Infrastructure [ Files to delete : ~ 185 files | Space to be freed : ~ 445.36 MB ]
Estimating purge for diagnostic destination "diag/rdbms/cdb674/CDB674" for files ~ 27760 files deleted , 66.57 MB freed ]
Estimation for Database Home [ Files to delete : ~ 27760 files | Space to be freed : ~ 66.57 MB ]

Run a database log purge
tfactl managelogs -purge -older 30d
------------------------------
Purging files older than 30 days
Cleaning Grid Infrastructure destinations
Purging diagnostic destination "diag/afdboot/user_root/host_309243680_94" for files - 0 files deleted , 0 bytes freed
Purging diagnostic destination "diag/afdboot/user_crsusr/host_309243680_94" for files - 1 files deleted , 10.16 KB freed
Purging diagnostic destination "diag/asmtool/user_root/host_309243680_96" for files - 1 files deleted , 10.16 KB freed
Purging diagnostic destination "diag/asmtool/user_crsusr/host_309243680_96" for files - 2 files deleted , 29.18 KB freed
Purging diagnostic destination "diag/tnslsnr/myserver74/listener" for files - 2 files deleted , 29.18 KB freed
Purging diagnostic destination "diag/diagtool/user_root/adrci_309243680_96" for files - 2 files deleted , 29.18 KB freed
Purging diagnostic destination "diag/clients/user_crsusr/host_309243680_96" for files - 2 files deleted , 29.18 KB freed
Purging diagnostic destination "diag/asm/+asm/+ASM" for files - 2 files deleted , 29.18 KB freed
Purging diagnostic destination "diag/asm/user_root/host_309243680_96" for files - 2 files deleted , 29.18 KB freed
Purging diagnostic destination "diag/asm/user_crsusr/host_309243680_96" for files - 2 files deleted , 29.18 KB freed
Purging diagnostic destination "diag/crs/myserver74/crs" for files - 2 files deleted , 29.18 KB freed
...

Run a database log purge
...
Grid Infrastructure [ Files deleted : 18 files | Space Freed : 253.75 KB ]
.-----------------------------------------------------------------------------------------------.
| File System Variation : /u01/app/crsusr/12.2.0/grid2 |
+--------+-----------------------------------+----------+----------+---------+----------+-------+
| State | Name | Size | Used | Free | Capacity | Mount |
+--------+-----------------------------------+----------+----------+---------+----------+-------+
| Before | /dev/mapper/vg_rws1270665-lv_root | 51475068 | 46597152 | 2256476 | 96% | / |
| After | /dev/mapper/vg_rws1270665-lv_root | 51475068 | 46597152 | 2256476 | 96% | / |
'--------+-----------------------------------+----------+----------+---------+----------+-------'

Disk Usage Snapshots
TFA will track disk usage and record snapshots to:
/opt/oracle.ahf/data/repository/suptools/<node>/managelogs/usage_snapshot
Disk usage monitoring is ON by default, configurable with:
Snapshot happens every 60 minutes, configurable with:
tfactl set diskUsageMonInterval=<minutes>
tfactl set diskUsageMon=<ON|OFF>

13 - Find events and execute
code based on them

Find events
tfactl events
INFO :0
ERROR :0
WARNING :0
Event Timeline:
No Events Found
INFO :0
ERROR :0
WARNING :0
Event Timeline:
No Events Found
...

Find events
...
INFO :1
ERROR :1
WARNING :0
Event Timeline:
[Sep/18/2019 02:38:25.000]: [db.ogg11204.ogg112041]: Incident details in:
/scratch/app/oradb/diag/rdbms/ogg11204/ogg112041/incident/incdir_102702/ogg
112041_ora_5001_i102702.trc
[Oct/18/2018 02:38:25.000]: [db.ogg11204.ogg112041]: ORA-00600: internal
error code, arguments: [ksprcvsp2], [1596993584], [], [], [], [], [], [],
[], [], [], []

Search metadata
All metadata stored in the TFA index is searchable:
Search for all events for a database between certain dates:
tfactl search -showdatatypes|-json [json_details]
tfactl search -json
‘{
“data_type” : “event”,
“content”: “oracle”,
“database” : “rac11g”,
“from”: “09/01/2019 00:00:00”,
“to” : “09/21/2019 00:00:00”
}’

Search metadata
Listing all index events:
Listing all available datatypes:
tfactl search -json ‘{“data_type” : “event”}’
tfactl search -showdatatypes

tail files
tfactl tail alert
------------------------------
/scratch/app/11.2.0.4/grid/log/myserver69/alertmyserver69.log
2019-09-25 23:28:22.532:
[ctssd(5630)]CRS-2409:The clock on host myserver69 is not synchronous with
the mean cluster time. No action has been taken as the Cluster Time
Synchronization Service is running in observer mode.
2019-09-25 23:58:22.964:
[ctssd(5630)]CRS-2409:The clock on host myserver69 is not synchronous with
the mean cluster time. No action has been taken as the Cluster Time
Synchronization Service is running in observer mode.
...

tail files
...
/scratch/app/oradb/diag/rdbms/apxcmupg/apxcmupg_2/trace/alert_apxcmupg_2.log
Wed Sep 25 06:00:00 2018 VKRM started with pid=82, OS id=4903
Wed Sep 25 06:00:02 2018 Begin automatic SQL Tuning Advisor run for special
tuning task "SYS_AUTO_SQL_TUNING_TASK"
Wed Sep 25 06:00:37 2018 End automatic SQL Tuning Advisor run for special
tuning task "SYS_AUTO_SQL_TUNING_TASK"
Wed Sep 25 23:00:28 2018 Thread 2 advanced to log sequence 759 (LGWR switch)
Current log# 3 seq# 759 mem# 0:
+DATA/apxcmupg/onlinelog/group_3.289.917164707
Current log# 3 seq# 759 mem# 1:
+FRA/apxcmupg/onlinelog/group_3.289.917164707
...

tail files
...
/scratch/app/oradb/diag/rdbms/ogg11204/ogg112041/trace/alert_ogg112041.log
Clearing Resource Manager plan via parameter
Wed Sep 25 05:59:59 2018
Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter
Wed Sep 25 05:59:59 2018
Starting background process VKRM
Wed Sep 25 05:59:59 2018
VKRM started with pid=36, OS id=4901
Wed Sep 25 22:00:31 2018
Thread 1 advanced to log sequence 305 (LGWR switch)
Current log# 1 seq# 305 mem# 0: +DATA/ogg11204/redo01.log
...

tail files
...
/scratch/app/oragrid/diag/asm/+asm/+ASM1/trace/alert_+ASM1.log <==
Sun Sep 22 04:42:22 2018
NOTE: [ocrcheck.bin@myserver69 (TNS V1-V3) 2323] opening OCR file
Mon Sep 23 01:05:39 2018
Mon Sep 23 01:05:41 2018
Mon Sep 23 01:21:12 2018
Mon Sep 23 01:21:12 2018
...

tail files
tfactl vi alert
2019-09-25 19:58:19.481:
[ctssd(5630)]CRS-2409:The clock on host myserver69 is not synchronous
with the mean cluster time. No action has been taken as the Cluster
Time Synchronization Service is running in observer mode.
2019-09-25 20:28:19.911:
2019-09-25 20:58:20.346:

16 - Monitor Database
performance

oratop (Support Tools Bundle)
Near real-time Database monitoring
Single instance & RAC
Monitoring current database activities
Database performance
Identifying contentions and bottleneck
Process & SQL Monitoring
Real time wait events
Active Data Guard support
Multitenant Database (CDB) support

Monitor Database performance
tfactl run oratop -database ogg19c

Monitor Database performance
Section 1 DATABASE:
Global database
information
Section 2 INSTANCE:
Database instance
Activity
Section 3 EVENT:
AWR like “Top 5
Timed Events“
Section 4 PROCESS |
SQL: Processes or
SQL mode
information more info 1500864.1

OS Watcher (Support Tools Bundle)
Collect & Archive OS Metrics
Executes standard UNIX utilities (e.g. vmstat, iostat, ps,
etc) on regular intervals
Built in Analyzer functionality to summarize, graph and
report upon collected metrics
Output is Required for node reboot and performance
issues
Simple to install, extremely lightweight
Runs on ALL platforms (Except Windows)

Analyse OS Metrics
tfactl run oswbb
Starting OSW Analyzer V8.1.2
OSWatcher Analyzer Written by Oracle Center of Expertise
Copyright (c) 2017 by Oracle Corporation
Parsing Data. Please Wait...
Scanning file headers for version and platform info...
Parsing file rws1270069_iostat_18.11.24.0900.dat ...
Parsing file rws1270069_iostat_18.11.24.1000.dat ...
...

Analyse OS Metrics
...
Enter 1 to Display CPU Process Queue Graphs
Enter 2 to Display CPU Utilization Graphs
Enter 3 to Display CPU Other Graphs
Enter 4 to Display Memory Graphs
Enter 5 to Display Disk IO Graphs
Enter GC to Generate All CPU Gif Files
Enter GM to Generate All Memory Gif Files
Enter GD to Generate All Disk Gif Files
Enter GN to Generate All Network Gif Files
Enter L to Specify Alternate Location of Gif Directory
Enter Z to Zoom Graph Time Scale (Does not change analysis dataset)
...

Analyse OS Metrics
...
Enter B to Returns to Baseline Graph Time Scale (Does not change
analysis dataset)
Enter R to Remove Currently Displayed Graphs
Enter X to Export Parsed Data to Flat File
Enter S to Analyze Subset of Data(Changes analysis dataset including
graph time scale)
Enter A to Analyze Data
Enter D to Generate DashBoard
Enter Q to Quit Program
Please Select an Option:1

Analyse OS Metrics
myserver69
more info 301137.1

Cluster Health Monitor (CHM)
GIMR
ologgerd
(master)
osysmon
d
osysmon
d
osysmon
d
osysmon
d
12c Grid Infrastructure
Management Repository
Generates view of Cluster and
Database diagnostic metrics
Always on - Enabled by default
Provides Detailed OS Resource Metrics
Assists Node eviction analysis
Locally logs all process data
User can define pinned processes
Listens to CSS and GIPC events
Categorizes processes by type
Supports plug-in collectors (ex. traceroute,
netstat, ping, etc.)
New CSV output for ease of analysis
OS Data OS Data
OS Data
OS Data

Cluster Health Monitor (CHM)
Confidential – Oracle Internal/Restricted/Highly
Restricted
Oclumon CLI or full integration
with EM Cloud Control

Cluster Health Advisor (CHA)*
Always on - Enabled by default
Detects node and database performance problems
Provides early-warning alerts and corrective action
Supports on-site calibration to improve sensitivity
Integrated into EMCC Incident Manager and
notifications
Standalone Interactive GUI Tool
OS Data
GIMR
ochad
DB Data
CHM
Node
Health
Prognostic
s
Engine
Database
Health
Prognostic
s
Engine
* Requires and Included with RAC or R1N License

Calibrating CHA to your RAC deployment
Choosing a Data Set for Calibration – Defining “normal”
chactl query calibration –cluster –timeranges ‘start=2016-10-28
07:00:00,end=2016-10-28 13:00:00’
Cluster name : mycluster
Start time : 2019-09-28 07:00:00
End time : 2019-09-28 13:00:00
Total Samples : 11524
Percentage of filtered data : 100%
1) Disk read (ASM) (Mbyte/sec)
MEAN MEDIAN STDDEV MIN MAX
0.11 0.00 2.62 0.00 114.66
<25 <50 <75 <100 >=100
99.87% 0.08% 0.00% 0.02% 0.03%
...

...
2) Disk write (ASM) (Mbyte/sec)
0.01 0.00 0.15 0.00 6.77
<50 <100 <150 <200 >=200
100.00% 0.00% 0.00% 0.00% 0.00%
...

...
3) Disk throughput (ASM) (IO/sec)
2.20 0.00 31.17 0.00 1100.00
<5000 <10000 <15000 <20000 >=20000
100.00% 0.00% 0.00% 0.00% 0.00%
4) CPU utilization (total) (%)
9.62 9.30 7.95 1.80 77.90
<20 <40 <60 <80 >=80
92.67% 6.17% 1.11% 0.05% 0.00%
...

Create and store a new model
Begin using the new model
Confirm the new model is working
chactl query calibrate cluster –model daytime –timeranges
‘start=2018-10-28 07:00:00, end=2018-10-28 13:00:00’
chactl monitor cluster –model daytime
chactl status –verbose
monitoring nodes svr01, svr02 using model daytime
monitoring database qoltpacdb, instances oltpacdb_1, oltpacdb_2 using
model DEFAULT_DB

Command line operations
Enable CHA monitoring on RAC database with optional model
Enable CHA monitoring on RAC database with optional verbose
chactl monitor database –db oltpacdb [-model model_name]
chactl status –verbose
monitoring nodes svr01, svr02 using model DEFAULT_CLUSTER
monitoring database oltpacdb, instances oltpacdb_1, oltpacdb_2 using
model DEFAULT_DB

Check for Health Issues and Corrective Actions with CHACTL QUERY DIAGNOSIS
chactl query diagnosis -db oltpacdb -start "2016-10-28 01:52:50" -end "2016-10-28 03:19:15"
2019-09-28 01:47:10.0 Database oltpacdb DB Control File IO Performance (oltpacdb_1) [detected]
2019-09-28 02:59:35.0 Database oltpacdb DB Log File Switch (oltpacdb_1) [detected]
Problem: DB Control File IO Performance
Description: CHA has detected that reads or writes to the control files are slower than expected.
Cause: The Cluster Health Advisor (CHA) detected that reads or writes to the control files were
slow because of an increase in disk IO.
The slow control file reads and writes may have an impact on checkpoint and Log Writer (LGWR)
performance.
Action: Separate the control files from other database files and move them to faster disks or Solid
State Devices.
Problem: DB Log File Switch
Description: CHA detected that database sessions are waiting longer than expected
for log switch completions.
Cause: The Cluster Health Advisor (CHA) detected high contention during log switches
because the redo log files were small and the redo logs switched frequently.
Action: Increase the size of the redo logs.

HTML diagnostic health output
available (-html <file_name>)

Diagnose cluster health
chactl query diagnosis -db oltpacdb -start "2019-09-26 02:52:50.0" -end "2019-09-26 03:19:15.0"
2019-09-26 02:52:15.0 Database oltpacdb DB CPU Utilization (oltpacdb_2) [detected]
2019-09-26 02:52:50.0 Database oltpacdb DB CPU Utilization (oltpacdb_1) [detected]

19 - Find if anything
has changed

Has anything changed recently?
tfactl changes
------------------------------
[Sep/17/2019 04:54:15.397]: Parameter: fs.aio-nr: Value: 95488 => 97024
[Sep/17/2019 04:54:15.397]: Parameter: fs.inode-nr: Value: 764974 131561 => 740744
131259
[Sep/17/2019 04:54:15.397]: Parameter: kernel.pty.nr: Value: 2 => 1
[Sep/17/2019 04:54:15.397]: Parameter: kernel.random.entropy_avail: Value: 189 => 158
[Sep/17/2019 04:54:15.397]: Parameter: kernel.random.uuid: Value: 36269877-9bc9-40a3-
82e0-1619865096f2 => 7551c5e7-c59f-40fa-b55f-5bd170e8b1ab
[Sep/17/2019 05:46:15.397]: Parameter: fs.inode-nr: Value: 1580316 810036 =>
1562320 768555
[Sep/17/2019 05:46:15.397]: Parameter: kernel.random.uuid: Value: 37cc31aa-ee31-459e-
8f2a-0766b34b1b64 => f5176cdc-6390-415d-882e-02c4cff2ae4e
...

Has anything changed recently?
...
------------------------------
[Sep/17/2019 04:54:15.397]: Parameter: fs.inode-nr: Value: 764974 131561 => 740744
131259
[Sep/17/2019 04:54:15.397]: Parameter: kernel.random.entropy_avail: Value: 189 => 158
[Sep/17/2019 04:54:15.397]: Parameter: kernel.random.uuid: Value: 36269877-9bc9-40a3-
82e0-1619865096f2 => 7551c5e7-c59f-40fa-b55f-5bd170e8b1ab
[Sep/17/2019 05:46:15.397]: Parameter: fs.inode-nr: Value: 1580316 810036 =>
1562320 768555
[Sep/17/2019 05:46:15.397]: Parameter: kernel.random.uuid: Value: 37cc31aa-ee31-459e-
8f2a-0766b34b1b64 => f5176cdc-6390-415d-882e-02c4cff2ae4e

Troubleshooting Tips and Tricks for Database 19c - EMEA Tour Oct 2019

More Related Content

What's hot (20)

Similar to Troubleshooting Tips and Tricks for Database 19c - EMEA Tour Oct 2019 (20)

More from Sandesh Rao (20)

Recently uploaded (20)

Troubleshooting Tips and Tricks for Database 19c - EMEA Tour Oct 2019