NZOUG - GroundBreakers-2018 -Using Oracle Autonomous Health Framework to Preserve Performance and Rapidly Recover
0 likes235 views
The document outlines the Oracle Autonomous Health Framework, designed to maintain database performance and facilitate rapid recovery through tools like TFA (Trace File Analyzer) and health checks (orachk/exachk). It emphasizes proactive problem detection and efficiency in diagnosing issues within Oracle systems, reducing reliance on manual diagnostics. The provided agenda includes installation, configuration, and usage details, with instructions for both reactive and proactive approaches to database health management.
![Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Manual Method
1. Generate ADDM reviewing Document 1680075.1 (multiple steps)
2. Identify “good” and “problem” periods and gather AWR reviewing
Document 1903158.1 (multiple steps)
3. Generate AWR compare report (awrddrpt.sql) using “good” and
“problem” periods
4. Generate ASH report for “good” and “problem” periods reviewing
Document 1903145.1 (multiple steps)
5. Collect OSWatcher data reviewing Document 301137.1 (multiple
steps)
6. Collect Hang Analyze output at Level 4
7. Generate SQL Healthcheck for problem SQL id using Document
1366133.1 (multiple steps)
8. Run support provided sql scripts – Log File sync diagnostic output using
Document 1064487.1 (multiple steps)
9. Check alert.log if there are any errors during the “problem” period
10. Find any trace files generated during the “problem” period
11. Collate and upload all the above files/outputs to SR
Automated One Command TFA SRDC
1. Run
32
Targeted Diagnostics – Service Request Data Collections (SRDCs)
tfactl diagcollect –srdc dbperf
[-sr <sr_number>]
Confidential – Oracle Internal](https://image.slidesharecdn.com/nzoug-groundbreakers-2018-usingoracleautonomoushealthframeworktopreserveperformanceandrapidlyrecover-190305192926/85/NZOUG-GroundBreakers-2018-Using-Oracle-Autonomous-Health-Framework-to-Preserve-Performance-and-Rapidly-Recover-32-320.jpg)
![Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Use ‘tfactl’ to check for recent Errors
bash-4.1# tfactl events
Output from host : myserver69
INFO :2
ERROR :2
WARNING :0
Event Timeline:
[Oct/18/2018 02:38:25.000]: [db.ogg11204.ogg112041]: Incident details in:
/scratch/app/oradb/diag/rdbms/ogg11204/ogg112041/incident/incdir_102702/ogg112041_ora_5001_i102
702.trc
[Oct/18/2018 02:38:25.000]: [db.ogg11204.ogg112041]: ORA-00600: internal error code, arguments:
[ksprcvsp2], [1596993584], [], [], [], [], [], [], [], [], [], []
[Oct/18/2018 02:38:37.000]: [db.ogg11204.ogg112041]: Incident details in:
/scratch/app/oradb/diag/rdbms/ogg11204/ogg112041/incident/incdir_102703/ogg112041_ora_5001_i102
703.trc
[Oct/18/2018 02:38:37.000]: [db.ogg11204.ogg112041]: ORA-00600: internal error code, arguments:
[ktfbtgex-7], [1015817], [1024], [1015816], [], [], [], [], [], [], [], []
34](https://image.slidesharecdn.com/nzoug-groundbreakers-2018-usingoracleautonomoushealthframeworktopreserveperformanceandrapidlyrecover-190305192926/85/NZOUG-GroundBreakers-2018-Using-Oracle-Autonomous-Health-Framework-to-Preserve-Performance-and-Rapidly-Recover-34-320.jpg)
![Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Check to see if a change may have caused the issue ?
-bash-4.1# tfactl changes
Output from host : myserver69
------------------------------
[Oct/17/2018 04:54:15.397]: [RDBMS.myDB1]: Parameter: parallel_max_servers: Value: 8 => 16
[Oct/17/2018 05:12:13.344]: [RDBMS.myDB1]: Parameter: log_archive_dest_1: Value: /var => /opt
35](https://image.slidesharecdn.com/nzoug-groundbreakers-2018-usingoracleautonomoushealthframeworktopreserveperformanceandrapidlyrecover-190305192926/85/NZOUG-GroundBreakers-2018-Using-Oracle-Autonomous-Health-Framework-to-Preserve-Performance-and-Rapidly-Recover-35-320.jpg)
![Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Metadata search capability
• All metadata stored in the TFA index is searchable:
• Searching for all events for a database between certain dates:
36
tfactl search -showdatatypes|-json [json_details]
tfactl search -json
‘{
"data_type":"event",
"content":"oracle",
"database":"rac11g",
"from":“10/01/2018 00:00:00",
"to":"10/21/2018 00:00:00"
}’](https://image.slidesharecdn.com/nzoug-groundbreakers-2018-usingoracleautonomoushealthframeworktopreserveperformanceandrapidlyrecover-190305192926/85/NZOUG-GroundBreakers-2018-Using-Oracle-Autonomous-Health-Framework-to-Preserve-Performance-and-Rapidly-Recover-36-320.jpg)
![Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Collect ORA-00600 SRDC
bash-4.1$ ./tfactl diagcollect -srdc ORA-00600
Enter the time of the ORA-00600 [YYYY-MM-DD HH24:MI:SS,=ALL] :
Enter the Database Name [=ALL] :
1. Oct/18/2018 02:38:37 : [ogg11204] ORA-00600: internal error code, arguments: [ktfbtgex-7], [1015817],
[1024], [1015816], [], [], [], [], [], [], [], []
2. Oct/18/2018 02:38:25 : [ogg11204] ORA-00600: internal error code, arguments: [ksprcvsp2],
[1596993584], [], [], [], [], [], [], [], [], [], []
Please choose the event : 1-2 [1]
38](https://image.slidesharecdn.com/nzoug-groundbreakers-2018-usingoracleautonomoushealthframeworktopreserveperformanceandrapidlyrecover-190305192926/85/NZOUG-GroundBreakers-2018-Using-Oracle-Autonomous-Health-Framework-to-Preserve-Performance-and-Rapidly-Recover-38-320.jpg)
![Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Manual Database Log Purge
• TFA can manage ADR log and trace files
– Show disk space usage of individual diagnostic destinations
– Purge these file types based on diagnostic location and or age:
• "ALERT“, "INCIDENT“, "TRACE“, "CDUMP“, "HM“, "UTSCDMP“, "LOG“
tfactl managelogs <options>
Runs as the ADR home
owner. So will only be able
to purge files this owner
has permission to delete
Option Description
–show usage Shows disk space usage per diagnostic directory for both GI and database logs
-show variation –older <n><m|h|d> Use to determine per directory disk space growth.
Shows the disk usage variation for the specified period per directory.
-purge –older <n><m|h|d> Remove all ADR files under the GI_BASE directory, which are older than the time specified
–gi Restrict command to only diagnostic files under the GI_BASE
–database [all | dbname] Restrict command to only diagnostic files under the database directory. Defaults to all,
alternatively specify a database name
-dryrun Use with –purge to estimate how many files will be affected and how much disk space will be
freed by a potential purge command.
May take a while for a
large number of files
45](https://image.slidesharecdn.com/nzoug-groundbreakers-2018-usingoracleautonomoushealthframeworktopreserveperformanceandrapidlyrecover-190305192926/85/NZOUG-GroundBreakers-2018-Using-Oracle-Autonomous-Health-Framework-to-Preserve-Performance-and-Rapidly-Recover-45-320.jpg)
![Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Cluster Health Advisor – Command Line Operations
Confidential – Oracle Restricted 53
Monitoring Your Databases and Nodes with CHACTL
Enable CHA monitoring on RAC database with optional model
$ chactl monitor database –db oltpacdb [-model model_name]
Enable CHA monitoring on RAC database with optional verbose
$ chactl status –verbose
monitoring nodes svr01, svr02 using model DEFAULT_CLUSTER
monitoring database oltpacdb, instances oltpacdb_1, oltpacdb_2 using model DEFAULT_DB](https://image.slidesharecdn.com/nzoug-groundbreakers-2018-usingoracleautonomoushealthframeworktopreserveperformanceandrapidlyrecover-190305192926/85/NZOUG-GroundBreakers-2018-Using-Oracle-Autonomous-Health-Framework-to-Preserve-Performance-and-Rapidly-Recover-53-320.jpg)
![Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
CHA Command Line Operations
Confidential – Oracle Restricted 54
Checking for Health Issues and Corrective Actions with CHACTL QUERY DIAGNOSIS
$ chactl query diagnosis -db oltpacdb -start "2016-10-28 01:52:50" -end "2016-10-28 03:19:15"
2016-10-28 01:47:10.0 Database oltpacdb DB Control File IO Performance (oltpacdb_1) [detected]
2016-10-28 01:47:10.0 Database oltpacdb DB Control File IO Performance (oltpacdb_2) [detected]
2016-10-28 02:59:35.0 Database oltpacdb DB Log File Switch (oltpacdb_1) [detected]
2016-10-28 02:59:45.0 Database oltpacdb DB Log File Switch (oltpacdb_2) [detected]
Problem: DB Control File IO Performance
Description: CHA has detected that reads or writes to the control files are slower than expected.
Cause: The Cluster Health Advisor (CHA) detected that reads or writes to the control files were
slow because of an increase in disk IO.
The slow control file reads and writes may have an impact on checkpoint and Log Writer (LGWR) performance.
Action: Separate the control files from other database files and move them to faster disks or Solid
State Devices.
Problem: DB Log File Switch
Description: CHA detected that database sessions are waiting longer than expected
for log switch completions.
Cause: The Cluster Health Advisor (CHA) detected high contention during log switches
because the redo log files were small and the redo logs switched frequently.
Action: Increase the size of the redo logs.](https://image.slidesharecdn.com/nzoug-groundbreakers-2018-usingoracleautonomoushealthframeworktopreserveperformanceandrapidlyrecover-190305192926/85/NZOUG-GroundBreakers-2018-Using-Oracle-Autonomous-Health-Framework-to-Preserve-Performance-and-Rapidly-Recover-54-320.jpg)
![Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Database Writes Error to Alert Log
Errors in file /cdb1810_1/trace/cdb1810_1_lmhb_16476.trc (incident=11791) (PDBNAME=CDB$ROOT):
ORA-29770: global enqueue process LMS0 (OSID 16458_16462) is hung for more than 70 seconds
Incident details in: /cdb1810_1/incident/incdir_11791/cdb1810_1_lmhb_16476_i11791.trc
LMHB (ospid: 16476): terminating the instance due to ORA error 29770
Cause - 'ERROR: Some process(s) is not making progress.
LMHB (ospid: 16476) is terminating the instance.
ERROR: Some process(s) is not making progress.
'System state dump requested by (instance=1, osid=16476 (LMHB)), summary=[abnormal instance
termination]. error - 'Instance is terminating.'
System State dumped to trace file /cdb1810_1/trace/cdb1810_1_diag_16435_20181016124352.trc
Instance terminated by USER, pid = 30173
Starting ORACLE instance (normal) (OS id: 30489)
82](https://image.slidesharecdn.com/nzoug-groundbreakers-2018-usingoracleautonomoushealthframeworktopreserveperformanceandrapidlyrecover-190305192926/85/NZOUG-GroundBreakers-2018-Using-Oracle-Autonomous-Health-Framework-to-Preserve-Performance-and-Rapidly-Recover-82-320.jpg)
![Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
TFA Detects and Generates a Collection
[INFO] [Thread-328-4437] addCriticalEvent: Thu Oct 18 14:37:13 PDT
2018:/cdb1810_1/trace/alert_cdb1810_1.log ORA-29770: global enqueue process LCK0 (OSID 26445) is
hung for more than 70 seconds
[INFO] [Thread-2801-9133] Sleeping for 5 minutes to collect events
[INFO] [Thread-328-4437] addCriticalEvent: Thu Oct 18 14:37:29 PDT
2018:/cdb1810_1/trace/alert_cdb1810_1.log System State dumped to trace file
/cdb1810_1/trace/cdb1810_1_diag_26239_20181018143729.trc
[INFO] [Thread-2801-9133] Processing event ORA-29770: global enqueue process LCK0 (OSID 26445) is
hung for more than 70 seconds at .... Thu Oct 18 14:37:13 PDT 2018
[INFO] [Thread-2801-9133] Processing event System State dumped to trace file
/cdb1810_1/trace/cdb1810_1_diag_26239_20181018143729.trc at .... Thu Oct 18 14:37:29 PDT 2018
[INFO] [Thread-2801-9133] Creating a composite event at Thu Oct 18 14:37:29 PDT 2018
[INFO] [Thread-2801-9133] compositeEvents: Thu Oct 18 14:37:29 PDT 2018
[INFO] [Thread-2801-9133] Done processing events
83](https://image.slidesharecdn.com/nzoug-groundbreakers-2018-usingoracleautonomoushealthframeworktopreserveperformanceandrapidlyrecover-190305192926/85/NZOUG-GroundBreakers-2018-Using-Oracle-Autonomous-Health-Framework-to-Preserve-Performance-and-Rapidly-Recover-83-320.jpg)
NZOUG - GroundBreakers-2018 -Using Oracle Autonomous Health Framework to Preserve Performance and Rapidly Recover
- 6. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Avoid the Pitfalls of Inefficient and Incomplete Diagnostics Collection 6 Become Proactive and Avoid Encountering Known Issues Help Us Help You!
- 9. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 9 Lots of Pings Customer Experience Before TFA Oracle Grid Infrastructure & Databases Oracle Support 1 Open new Service Request Collect data from all nodes without regard to relevance2 3 Upload data Collect more missing data (ping) 4 5 Upload more missing data 6 Download tools/scripts (ping) 7 Run tools/scripts Upload results of tools/scripts8 Confidential – Oracle Internal Multiple iterations & pings during SR resolution
- 10. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 10 Lots of Pings Support Experience Before TFA 1 Download all files to laptop or view separately in ISDE End of shift handover write-ups / discussion after which the new analysts frequently asked for some of the same data already uploaded Navigate 100s of log files to figure out problem areas 3 Search SR’s, bugs, MOS notes to find solutions 4 Oracle Support 2Multiple iterations & pings during SR resolution Confidential – Oracle Internal
- 11. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal 11 Customer Experience Before ORAchk/EXAchk Oracle Databases “Hi Oracle Support, my database just fell over” “ahh yer….we published a note on that a while ago…. …didn’t you see it?” “hmmmmm”
- 18. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 18 Experience Today With TFA & ORAchk/EXAchk @ Customer @ Oracle Add advice to SR 1 1 Detect Issue 3 5 Trim, Capture, Package & Optionally Upload Diagnostics 6 Integrate with AHF 4 OthersCollection Manager Integrate & Display Health Checks Results 2 TFA UI (TFA Web) Diagnose SR & Recommend Solution 2 Notification of Issues TFA / ORAchk/EXAchk Auto Proactive Health checks
- 20. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | • Quarterly release cycle – Follow similar release number formatting to DB – Current release 18.3.0 • Installed by default for Grid Infrastructure • Available for install from database homes • Updated via Release Updates • Also available on My Oracle Support (MOS): Doc 1513912.1 – Supported to patch GI Homes 20 Getting TFA (Which Includes ORAchk/EXAchk)
- 21. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Installation / Upgrade Using My Oracle Support Download Continuous Service Mode (Preferred) 1. Transfer zip to required machine 2. Unzip 3. Execute self extracting install script as root user • Will install/upgrade on all cluster nodes • Will auto discover relevant Oracle Software & Exadata Storage Servers • Will start monitoring all discovered items for significant events & collect diagnostics when necessary 21 ./installTFA-<platform>
- 22. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Installation / Upgrade Using My Oracle Support Download Standalone Mode Installation 1. Transfer zip to required machine 2. Unzip 3. Execute self extracting install script as the Oracle Software Owner 22 ./installTFA-<platform> -extractto <path>
- 23. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Configuration • No Configuration Required However you can:- – Change the location where collections are Generated – Change the amount of space allocated to the Collection Location. – Set up email notifications – Configure Log retention and purging. – Many More Options Documented in the User Guide. 23 tfactl set reposizeMB=20480
- 26. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Command line • Specify all command options at the command line Shell 1. Set and change context 2. Run commands from within the shell 26 Menu • Select menu navigation options then choose the command you want to run REST • Invoke commands over HTTPS TFA Command Interfaces tfactl <command> tfactl tfaclt > database MyDB MyDB tfactl > oratop tfactl menu tfactl rest -start https://host:port/ords/{api}
- 27. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | TFA Utilities To Detect and Analyze Issues Tool Description ORAchk or EXAchk Provides health checks for the Oracle stack. Oracle Trace File Analyzer will install: - • EXAchk for Engineered Systems, see document 1070954.1 for more details • ORAchk for all non-Engineered Systems, see document 1268927.2 for more details OSWatcher Oswatcher Collects and archives OS metrics. These are useful for instance or node evictions & performance Issues. See document 301137.1 for more details oratop Provides near real-time database monitoring. See document 1500864.1 for more details. alertsummary Provides summary of events for one or more database or ASM alert files from all nodes ls Lists all files TFA knows about for a given file name pattern across all nodes pstack Generate process stack for specified processes across all nodes grep Search alert or trace files with a given database and file name pattern, for a search string. summary Provides high level summary of the configuration 27 tfactl <tool>
- 28. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | TFA Utilities To Detect and Analyze Issues Tool Description vi Opens alert or trace files for viewing a given database and file name pattern in the vi editor tail Runs a tail on an alert or trace files for a given database and file name pattern param Shows all database and OS parameters that match a specified pattern dbglevel Sets and unsets multiple CRS trace levels with one command history Shows the shell history for the tfactl shell changes Reports changes in the system setup over a given time period. This includes database parameters, OS parameters and patches applied calog Reports major events from the Cluster Event log events Reports warnings and errors seen in the logs managelogs Shows disk space usage and purges ADR log and trace files ps Finds processes triage Summarize oswatcher/exawatcher data 28
- 29. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Collecting Diagnostics with TFA Standard Diag Collection Targeted Diag Collection via SRDC 1. Run 2. Upload resulting zip file to SR tfactl diagcollect –srdc <srdc>1. Run OR Run OR Run 2. Upload resulting zip file to SR tfactl diagcollect –last <n><d>|<h> tfactl diagcollect –from <date> -to <time> tfactl diagcollect 29
- 30. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 1. Enter default for event date/time and database name 2. Scans system to identify recent 10 events in the system (ORA600 example shown) 3. Once the relevant event is chosen, proceeds with diagnostic collection 30 One Command SRDC tfactl diagcollect –srdc <srdc_type> 4. All required files are identified 5. Trimmed where applicable 6. Package in a zip ready to provide to support Interactive Mode
- 31. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 31 Faster & Easier SR Data Collection tfactl diagcollect –srdc <srdc_type> -sr <SR#> Type of Problem SRDC ORA Errors • ORA-00020 • ORA-00060 • ORA-00600 • ORA-00700 • ORA-01555 • ORA-01628 • ORA-04030 • ORA-04031 • ORA-07445 • ORA-27300 • ORA-27301 • ORA-27302 • ORA-30036 Other internal database errors • internalerror Database performance • dbperf Database patching • dbpatchinstall • dbpatchconflict Database resource • dbunixresources XDB installation or invalid object • dbxdb Database install / upgrade • dbinstall • dbupgrade • dbpreupgrade Type of Problem SRDC Database storage • asm Excessive SYSAUX Space used by the Automatic Workload Repository (AWR) • dbawrspace Database startup / shutdown • dbshutdown • dbstartup Data Guard • dbdataguard Enterprise Manager tablespace usage metric • emtbsmetrics Enterprise Manager general metrics page or threshold problems - Run all three SRDCs • emdebugon • emdebugoff • emmetricalert Enterprise Manager target discovery / add • emcliadd • emclusdisc • emdbsys • emgendisc • emprocdisc Enterprise Manager OMS restart • emrestartoms
- 32. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Manual Method 1. Generate ADDM reviewing Document 1680075.1 (multiple steps) 2. Identify “good” and “problem” periods and gather AWR reviewing Document 1903158.1 (multiple steps) 3. Generate AWR compare report (awrddrpt.sql) using “good” and “problem” periods 4. Generate ASH report for “good” and “problem” periods reviewing Document 1903145.1 (multiple steps) 5. Collect OSWatcher data reviewing Document 301137.1 (multiple steps) 6. Collect Hang Analyze output at Level 4 7. Generate SQL Healthcheck for problem SQL id using Document 1366133.1 (multiple steps) 8. Run support provided sql scripts – Log File sync diagnostic output using Document 1064487.1 (multiple steps) 9. Check alert.log if there are any errors during the “problem” period 10. Find any trace files generated during the “problem” period 11. Collate and upload all the above files/outputs to SR Automated One Command TFA SRDC 1. Run 32 Targeted Diagnostics – Service Request Data Collections (SRDCs) tfactl diagcollect –srdc dbperf [-sr <sr_number>] Confidential – Oracle Internal
- 36. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Metadata search capability • All metadata stored in the TFA index is searchable: • Searching for all events for a database between certain dates: 36 tfactl search -showdatatypes|-json [json_details] tfactl search -json ‘{ "data_type":"event", "content":"oracle", "database":"rac11g", "from":“10/01/2018 00:00:00", "to":"10/21/2018 00:00:00" }’
- 37. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Metadata search capability • Listing all index events: • Listing all available datatypes: 37 tfactl search -json ‘{"data_type":"event"}’ tfactl search -showdatatypes
- 38. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Collect ORA-00600 SRDC bash-4.1$ ./tfactl diagcollect -srdc ORA-00600 Enter the time of the ORA-00600 [YYYY-MM-DD HH24:MI:SS,=ALL] : Enter the Database Name [=ALL] : 1. Oct/18/2018 02:38:37 : [ogg11204] ORA-00600: internal error code, arguments: [ktfbtgex-7], [1015817], [1024], [1015816], [], [], [], [], [], [], [], [] 2. Oct/18/2018 02:38:25 : [ogg11204] ORA-00600: internal error code, arguments: [ksprcvsp2], [1596993584], [], [], [], [], [], [], [], [], [], [] Please choose the event : 1-2 [1] 38
- 41. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 41 OS Watcher (Support Tools Bundle) Collect & Archive OS Metrics • Executes standard UNIX utilities (e.g. vmstat, iostat, ps, etc) on regular intervals • Built in Analyzer functionality to summarize, graph and report upon collected metrics • Output is Required for node reboot and performance issues • Simple to install, extremely lightweight • Runs on ALL platforms (Except Windows) • MOS Note: 301137.1 – OS Watcher Users Guide
- 42. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 42 Procwatcher (Support Tools Bundle) Monitor & Examine Database Processes • Single instance & RAC • Generates session wait, lock and latch reports as well as call stacks from any problem process(s) • Ability to collect stack traces of specific processes using Oracle Tools and OS Debuggers • Typically reduces SR resolution for performance related issues • Runs on ALL major UNIX Platforms • MOS Note: 459694.1 – Procwatcher Install Guide
- 43. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 43 oratop (Support Tools Bundle) Near Real-Time Database Monitoring • Single instance & RAC • Monitoring current database activities • Database performance • Identifying contentions and bottleneck
- 44. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Automatic Database Log Purge • TFA can automatically purge database logs – OFF by default – Except on a Domain Service Cluster (DSC), which it is ON by default • Turn auto purging on or off: • Will remove logs older than 30 days – configurable with: • Purging runs every 60 minutes – configurable with: 44 tfactl set manageLogsAutoPurge=<ON|OFF> tfactl set manageLogsAutoPurgePolicyAge=<n><d|h> tfactl set manageLogsAutoPurgeInterval=<minutes>
- 45. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Manual Database Log Purge • TFA can manage ADR log and trace files – Show disk space usage of individual diagnostic destinations – Purge these file types based on diagnostic location and or age: • "ALERT“, "INCIDENT“, "TRACE“, "CDUMP“, "HM“, "UTSCDMP“, "LOG“ tfactl managelogs <options> Runs as the ADR home owner. So will only be able to purge files this owner has permission to delete Option Description –show usage Shows disk space usage per diagnostic directory for both GI and database logs -show variation –older <n><m|h|d> Use to determine per directory disk space growth. Shows the disk usage variation for the specified period per directory. -purge –older <n><m|h|d> Remove all ADR files under the GI_BASE directory, which are older than the time specified –gi Restrict command to only diagnostic files under the GI_BASE –database [all | dbname] Restrict command to only diagnostic files under the database directory. Defaults to all, alternatively specify a database name -dryrun Use with –purge to estimate how many files will be affected and how much disk space will be freed by a potential purge command. May take a while for a large number of files 45
- 46. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 46 Manual Database Log Purge tfactl managelogs –show usage tfactl managelogs –show variation –older <n><m|h|d> Use -gi to only show grid infrastructure Use –database to only show database
- 47. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 47 Manual Database Log Purge tfactl managelogs –purge –older n<m|h|d> -dryrun tfactl managelogs –purge –older n<m|h|d> Use –dryrun for a “what if”
- 48. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 48 Generates Diagnostic Metrics View of Cluster and Databases Cluster Health Monitor (CHM) GIMR ologgerd (master) osysmond osysmond osysmond osysmond 12c Grid Infrastructure Management Repository • Always on - Enabled by default • Provides Detailed OS Resource Metrics • Assists Node eviction analysis • Locally logs all process data • User can define pinned processes • Listens to CSS and GIPC events • Categorizes processes by type • Supports plug-in collectors (ex. traceroute, netstat, ping, etc.) • New CSV output for ease of analysis OS Data OS Data OS Data OS Data Confidential – Oracle Internal/Restricted/Highly RestrictedConfidential – Oracle Restricted
- 49. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 49 Oclumon CLI or Full Integration with EM Cloud Control Cluster Health Monitor (CHM) Confidential – Oracle Internal/Restricted/Highly RestrictedConfidential – Oracle Restricted
- 50. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Cluster Health Advisor (CHA)* Discovers Potential Cluster & DB Problems - Notifies with Corrective Actions 50 OS Data GIMR ochad • Always on - Enabled by default • Detects node and database performance problems • Provides early-warning alerts and corrective action • Supports on-site calibration to improve sensitivity • Integrated into EMCC Incident Manager and notifications • Standalone Interactive GUI Tool DB Data CHM Node Health Prognostics Engine Database Health Prognostics Engine * Requires and Included with RAC or R1N License Confidential – Oracle Restricted
- 51. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Calibrating CHA to your RAC deployment Confidential – Oracle Restricted 51 Choosing a Data Set for Calibration – Defining “normal” $ chactl query calibration –cluster –timeranges ‘start=2016-10-28 07:00:00,end=2016-10-28 13:00:00’ Cluster name : mycluster Start time : 2016-10-28 07:00:00 End time : 2016-10-28 13:00:00 Total Samples : 11524 Percentage of filtered data : 100% 1) Disk read (ASM) (Mbyte/sec) MEAN MEDIAN STDDEV MIN MAX 0.11 0.00 2.62 0.00 114.66 <25 <50 <75 <100 >=100 99.87% 0.08% 0.00% 0.02% 0.03% 2) Disk write (ASM) (Mbyte/sec) MEAN MEDIAN STDDEV MIN MAX 0.01 0.00 0.15 0.00 6.77 <50 <100 <150 <200 >=200 100.00% 0.00% 0.00% 0.00% 0.00% 3) Disk throughput (ASM) (IO/sec) MEAN MEDIAN STDDEV MIN MAX 2.20 0.00 31.17 0.00 1100.00 <5000 <10000 <15000 <20000 >=20000 100.00% 0.00% 0.00% 0.00% 0.00% 4) CPU utilization (total) (%) MEAN MEDIAN STDDEV MIN MAX 9.62 9.30 7.95 1.80 77.90 <20 <40 <60 <80 >=80 92.67% 6.17% 1.11% 0.05% 0.00%
- 52. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Calibrating CHA to your RAC deployment • Create and store the new model $ chactl query calibrate cluster –model daytime –timeranges ‘start=2018-10-28 07:00:00, end=2018-10-28 13:00:00’ • Begin using the new model $ chactl monitor cluster –model daytime • Confirm the new model is being used $ chactl status –verbose monitoring nodes svr01, svr02 using model daytime monitoring database qoltpacdb, instances oltpacdb_1, oltpacdb_2 using model DEFAULT_DB Confidential – Oracle Restricted 52 Creating a new CHA Model with CHACTL
- 53. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Cluster Health Advisor – Command Line Operations Confidential – Oracle Restricted 53 Monitoring Your Databases and Nodes with CHACTL Enable CHA monitoring on RAC database with optional model $ chactl monitor database –db oltpacdb [-model model_name] Enable CHA monitoring on RAC database with optional verbose $ chactl status –verbose monitoring nodes svr01, svr02 using model DEFAULT_CLUSTER monitoring database oltpacdb, instances oltpacdb_1, oltpacdb_2 using model DEFAULT_DB
- 54. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | CHA Command Line Operations Confidential – Oracle Restricted 54 Checking for Health Issues and Corrective Actions with CHACTL QUERY DIAGNOSIS $ chactl query diagnosis -db oltpacdb -start "2016-10-28 01:52:50" -end "2016-10-28 03:19:15" 2016-10-28 01:47:10.0 Database oltpacdb DB Control File IO Performance (oltpacdb_1) [detected] 2016-10-28 01:47:10.0 Database oltpacdb DB Control File IO Performance (oltpacdb_2) [detected] 2016-10-28 02:59:35.0 Database oltpacdb DB Log File Switch (oltpacdb_1) [detected] 2016-10-28 02:59:45.0 Database oltpacdb DB Log File Switch (oltpacdb_2) [detected] Problem: DB Control File IO Performance Description: CHA has detected that reads or writes to the control files are slower than expected. Cause: The Cluster Health Advisor (CHA) detected that reads or writes to the control files were slow because of an increase in disk IO. The slow control file reads and writes may have an impact on checkpoint and Log Writer (LGWR) performance. Action: Separate the control files from other database files and move them to faster disks or Solid State Devices. Problem: DB Log File Switch Description: CHA detected that database sessions are waiting longer than expected for log switch completions. Cause: The Cluster Health Advisor (CHA) detected high contention during log switches because the redo log files were small and the redo logs switched frequently. Action: Increase the size of the redo logs.
- 55. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Cluster Health Advisor – Command Line Operations Confidential – Oracle Restricted 55 HTML Diagnostic Health Output Available (-html <file_name>)
- 58. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | ORAchk/EXAchk email Notification • Automatically started & configured to run Critical Health Checks • You only need to configure your email for notification 58 tfactl orachk/exachk -set “NOTIFICATION_EMAIL=SOME.BODY@COMPANY.COM
- 60. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Configure Diagnostic Collection email Notification • Set notification email for any problem detected: • To set notification email for specific ORACLE_HOMEs include the OS home owner: 60 tfactl set notificationAddress=john.doe@oracle.com tfactl set notificationAddress=oracle:another.person@oracle.com
- 76. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 76 Deploy with Minimum Footprint and Maximum Manageability Oracle 18c Domain Services Cluster Application Member Cluster Database Member Cluster Database Member Cluster Oracle Domain Services Cluster Database Member Cluster Application Member Cluster Database Member Cluster ORACLE CLUSTER DOMAIN Management Repository Service Trace File Analyzer Service Grid Names Service Storage Services QoS Management Service Rapid Home Provisioning Service Confidential – Oracle Internal/Restricted/Highly Restricted • Hosts Framework as Services • Reduces local resource footprint • Centralizes management • Speeds deployment and patching • Optional Shared Storage • Supports multiple versions and platforms going forward
- 77. 77
- 78. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Domain Services Cluster Already Has TFA User Interface • Central TFA Repository utilizing ACFS Storage • Member Clusters Send TFA Collections to the TFA Service on DSC • TFA Service indexes the Collection and runs Analysers. • New UI will be shipped in 19 78
- 79. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Standalone User Interface • TFA Collector will upload to central repository • TFA UI analyses files and generates – Events TimeLine – Anomaly TimeLine using Applied Machine Learning – Root Cause Analysis and Recommendations where available. – Interface to easily access all files and analyser reports. • Already used in Oracle Database Cloud. • Coming On Prem in 19 79
- 82. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Database Writes Error to Alert Log Errors in file /cdb1810_1/trace/cdb1810_1_lmhb_16476.trc (incident=11791) (PDBNAME=CDB$ROOT): ORA-29770: global enqueue process LMS0 (OSID 16458_16462) is hung for more than 70 seconds Incident details in: /cdb1810_1/incident/incdir_11791/cdb1810_1_lmhb_16476_i11791.trc LMHB (ospid: 16476): terminating the instance due to ORA error 29770 Cause - 'ERROR: Some process(s) is not making progress. LMHB (ospid: 16476) is terminating the instance. ERROR: Some process(s) is not making progress. 'System state dump requested by (instance=1, osid=16476 (LMHB)), summary=[abnormal instance termination]. error - 'Instance is terminating.' System State dumped to trace file /cdb1810_1/trace/cdb1810_1_diag_16435_20181016124352.trc Instance terminated by USER, pid = 30173 Starting ORACLE instance (normal) (OS id: 30489) 82
- 83. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | TFA Detects and Generates a Collection [INFO] [Thread-328-4437] addCriticalEvent: Thu Oct 18 14:37:13 PDT 2018:/cdb1810_1/trace/alert_cdb1810_1.log ORA-29770: global enqueue process LCK0 (OSID 26445) is hung for more than 70 seconds [INFO] [Thread-2801-9133] Sleeping for 5 minutes to collect events [INFO] [Thread-328-4437] addCriticalEvent: Thu Oct 18 14:37:29 PDT 2018:/cdb1810_1/trace/alert_cdb1810_1.log System State dumped to trace file /cdb1810_1/trace/cdb1810_1_diag_26239_20181018143729.trc [INFO] [Thread-2801-9133] Processing event ORA-29770: global enqueue process LCK0 (OSID 26445) is hung for more than 70 seconds at .... Thu Oct 18 14:37:13 PDT 2018 [INFO] [Thread-2801-9133] Processing event System State dumped to trace file /cdb1810_1/trace/cdb1810_1_diag_26239_20181018143729.trc at .... Thu Oct 18 14:37:29 PDT 2018 [INFO] [Thread-2801-9133] Creating a composite event at Thu Oct 18 14:37:29 PDT 2018 [INFO] [Thread-2801-9133] compositeEvents: Thu Oct 18 14:37:29 PDT 2018 [INFO] [Thread-2801-9133] Done processing events 83
- 86. 86
- 87. 87
- 90. 90
- 91. 91
- 92. 92
- 94. 94
- 97. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 97 Trace File Analyzer – High Level Anomaly Detection Flow Log Cleansing 1 2 3 4 5 6 Entry Feature Creation Entry Clustering Model Generation Expert Input Knowledge Base Creation Knowledge Base Indexing Feedback Training Real-time Log File Processing Timestamp Correlation & Ranking 8 9 7 Batch Feedback
- 98. 98
- 100. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Feed Machine Learning Model Back into TFA collector 100 TFA & EXAchk SRs Expert Input Feedback & Improvement SRs Model Generation Model Knowledge Extraction Machine Learning Training Cleansing, metadata creation & clustering 1 Model generation with expert scrubbing2 Deployed as part of cloud image, running from the start, using model to detect issues 3 • Identify Individual/Sequence of Log entries around which Automatic Collection should be performed using new additions to knowledgebase & provide needed collection context. • Proactive pre-emptive actions based on analysis of multiple time series metrics can be performed. • Reactive corrective action can be optionally automated via TFA Service Confidential – Oracle Internal