Sandesh_Rao_Navigating Oracle Troubleshooting- AHF Insights for Database 23ai - AIOUG July 2024.pdf
0 likes127 views
The document discusses AHF Insights, a unified reporting tool for the AHF stack, aimed at enhancing Oracle database troubleshooting through effective issue resolution and insights generation. It covers various scenarios of issues such as inconsistent RPMs, software version discrepancies, and recent system changes, which can cause node evictions and instability. Additionally, it emphasizes the importance of maintaining current software for better security and compatibility.
![§ For troubleshooting one needs to know :
§ What type of system does user have ?
§ What’s going on around the time of issue ?
§ Can I get a full picture across all nodes ?
§ Can I zoom into specific timeframe ?
§ Can I look at the data from various perspectives ?
Customer Complains of “Grid failure - CRS-8503 []” in SR](https://image.slidesharecdn.com/sandeshraonavigatingoracletroubleshooting-ahfinsightsfordatabase23ai-aiougjuly2024-241119052650-596b7836/85/Sandesh_Rao_Navigating-Oracle-Troubleshooting-AHF-Insights-for-Database-23ai-AIOUG-July-2024-pdf-32-320.jpg)
Sandesh_Rao_Navigating Oracle Troubleshooting- AHF Insights for Database 23ai - AIOUG July 2024.pdf
- 1. A unified reporting & correlation tool for AHF Stack Navigating Oracle Troubleshooting: AHF Insights for Database 23ai Sandesh Rao VP AIOps and Machine Learning , Autonomous Database @sandeshr https://www.linkedin.com/in/raosandesh/ https://www.slideshare.net/SandeshRao4
- 2. AHF Stack
- 3. § Purpose § Provides bird's eye view of the system from diagnostic perspective. § Offers insights for effective issue resolution with guidance & co-relation. § Unifies the AHF stack under a single user interface § Usage § Reactive, Proactive. § Target Users § Customers, Operations, Support, Development § To generate AHF Insights, run : § Collected by Default in AHF Collections from AHF 24.2 AHF Insights
- 4. AHF Insights (over command-line)
- 5. AHF Insights as part of AHF collection
- 6. AHF Insights as part of AHF collection
- 7. § All Manual Collections § This is included in all AHF manual collections since 24.2 § SRDCs (Subset) Automatic Insights Report with AHF collection § dbinstancecrash § dbunixresources § crsdbhangperf § dbhangperflite § dbperf § dbrac § dbracperf § exadata § dbdataguard § dbhangperflite_auto
- 10. Need configurations details while troubleshooting an issue
- 11. § BareMetal system (DomU has access to query storage server and fabric switch details) § DomU has limited access Configuration Details
- 12. Cluster Details
- 13. ASM Details
- 14. Database Details
- 19. Troubleshoot : Issue due to inconsistent RPMs
- 20. § Example Scenario - Inconsistent glibc Package Versions § Issue - Node Eviction and Clusterware Instability § Example § In a four-node Oracle RAC cluster, nodes 1 and 2 have glibc version 2.17-307 installed, while nodes 3 and 4 have glibc version 2.17-307.el7.1 installed. This discrepancy can cause several problems. § Impact § Node Eviction - Due to the different versions of glibc, nodes 3 and 4 might face eviction as the clusterware detects inconsistencies in the environment. § Clusterware Instability - The inconsistency in glibc can cause instability in Oracle Clusterware, leading to startup failures and communication errors. Issue due to inconsistent RPMs
- 22. Troubleshoot : Issue due to software version lower than MAA Software Recommendations
- 23. Recommended Software All software should be updated regularly. Maintaining software at current or recent releases provides the following benefits: § Better software security § More stable maintenance releases § Continued compatibility with newer related software § Better support and faster resolution of issues § Ability to receive fixes for newly discovered issues.
- 24. Troubleshoot : Issue due to recent changes on the system
- 25. § Issue due to Application of new Patch § Issue due to Changes on ASM / Database parameter § Issue due to New OS package installed § Issue due to New Oracle Software installed Issue due to recent changes on the system
- 26. System Changes
- 27. Troubleshoot : Space Usage Issues
- 29. Troubleshoot : Issue due to Best Practice Violations
- 31. Troubleshoot : Major Events happening across the cluster
- 32. § For troubleshooting one needs to know : § What type of system does user have ? § What’s going on around the time of issue ? § Can I get a full picture across all nodes ? § Can I zoom into specific timeframe ? § Can I look at the data from various perspectives ? Customer Complains of “Grid failure - CRS-8503 []” in SR
- 33. Customer’s System around the time of Issue
- 34. Major Events around the time of issue
- 35. Major Events around the time of issue
- 36. Major Events around the time of issue
- 37. Troubleshoot : Operating System Issues
- 38. Customer’s System undergoes Node eviction
- 39. Customer’s System undergoes Node eviction
- 40. Customer’s System undergoes Node eviction High Memory Pressure Increase in RSS consumption by ‘extract’ process
- 41. Customer’s System undergoes Node eviction 50GB RSS hogged by extract process
- 43. Database Anomalies as observed in the Database Anomaly Advisor
- 44. Database Anomalies as observed in the Database Anomaly Advisor
- 45. Troubleshoot : Node Eviction Problems
- 46. Node eviction due to Huge Page over-allocation
- 47. Node eviction due to Huge Page over-allocation
- 48. Node eviction due to Huge Page over-allocation
- 49. Node eviction due to Huge Page over-allocation
- 50. Node eviction due to Huge Page over-allocation
- 51. Node eviction due to Huge Page over-allocation
- 52. Node eviction due to Huge Page over-allocation
- 53. How customers can log SRs using diagnostics
- 57. Demo
- 58. Thank you 46 Confidential – © 2019 Oracle Internal/Restricted/Highly Restricted