Oracle Fuson Middleware Diagnostics, Performance and Troubleshoot
0 likes957 views
The document outlines performance and diagnostics solutions for Oracle Fusion Middleware, focusing on the WebLogic Server. It covers troubleshooting methods, JVM behavior, and specific error handling techniques, providing insights into server hangs, JDBC issues, and JMS problems. It emphasizes the importance of structured diagnostics, using various tools for effective performance monitoring and issue resolution.
![•
Java Mission Control
-XX:+UnlockCommercialFeatures-XX:+UnlockCommercialFeatures
jcmd [PID] VM.unlock_commercial_features](https://image.slidesharecdn.com/oraclefmwdiagperftunetroubleshoot-180607184510/85/Oracle-Fuson-Middleware-Diagnostics-Performance-and-Troubleshoot-25-320.jpg)
Oracle Fuson Middleware Diagnostics, Performance and Troubleshoot
- 1. ©2018 Michel Schildmeijer©2018 Michel Schildmeijer Oracle Fusion Middleware: Performance & Diagnostics solutions on WebLogic nlOUG Tech Experience ‘The Cloud is next’ Amersfoort, 7 June 2018 By Michel Schildmeijer,
- 2. Me….. • Michel Schildmeijer, almost 52 years • Solutions Architect • Author of 2 books • Specialties: • WebLogic 6 to 12, OSB, Tuxedo, SOA, • Architect and design
- 3. Contents • Troubleshoot Patterns & Methods • A slight view under the hood • Hints, tips & tricks
- 5. • System errors, messages lost, bad performance • Where do I begin?? • What method should I follow? Troubleshoot overview
- 6. • Using these troubleshooting methods helps you to: • A structured way of detecting issues • Get behind the cause of the problems that occur • Methods to follow: • Use of Operating system tools • Use of JDK tools • Use of WebLogic Server tools • Use of FMW Diagnostic tools • Use of Enterprise Manager ( Cloud Control )(?) • Use of OMC : Application Performance Management, Log Analytics(?) Troubleshoot Methods
- 7. • Patterns to identify a “Hang” • Hangs related to the WebLogic Server, Java or some subset of resources • Generic Server Hang Pattern • JDBC Hang Pattern • EJB & RMI Hang Pattern • JMS Hang Pattern Troubleshoot Patterns
- 8. • Server doesn’t respond to new requests • Requests time out. • Requests take longer and longer to process (may be on the way to a hang). • A server crash is not usually a symptom of a hung server but final state Server Hang
- 9. • DriverManager.getConnection() in code. • SQL takes unexpectedly long time • JDBC connection pool hangs • Slow network causes to slow down or hang. • A deadlock all execute threads to hang & wait • RefreshMinutes or TestFrequencySeconds in conn pool causes hang • Long response JDBC conn pool shrinking and • No Sufficient DB resources JDBC Hang
- 10. • Issues over RMI with remote JVM • Serialization • Cluster issues • JNDI lookup fails at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java: 341) EJB RMI Hang
- 11. • Increasing pending messages • Sent by a producer • Received by a consumer • Lagging async client • Durable subscribers (to avoid missing placed messages in a topic) JMS Hang
- 12. • O/S tools: • sar: system activity • mpstat: per-processor statistics • vmstat: virtual memory statistics • netstat: network statistics • iostat: Input/Output statistics • top WebLogic Troubleshoot overview
- 13. • Start with a simple check – it seems so obvious but: • Is there enough storage • Is there enough RAM available • Does the system do a lot of paging WebLogic Troubleshoot overview
- 14. JVM Troubleshoot
- 15. The JavaVirtualMachine Java Source code .java Java bytecodeJVM RuntimeCompiler Java Runtime JDK Compiler
- 16. • A Java application runs a piece of code • Code enters the JVM • It enters several stages: • Operations • Data structure • Transformation ( to bytecode) JVM from code to machine Java code JVMJVM Operat ons Data re Data Structu re Transformati on 110110 100
- 17. • JVM process consists Java and JVM internal threads • JVM threads: Garbage Collect, Optimizer and Finalizer • Each thread has a stack to store runtime data • Overflow will happen if this stack will store more than its size • Each O/S has its default stacksize • Can be extended with -Xss Understanding Java threads
- 18. • Runtime behaviour: o JVM runs as an OS process o Execute work through threading model (concurrency) o Garbage Collector • Each Weblogic Server (instance) runs inside a separate JVM (instance). JVM from code to machine
- 19. • Crash can occur during • Compile process • Cause: invalid native code (C, C++, Libs) • Internal Code – Garbage Collect ( Segmentation Fault) • Cause: GC Fault leading into bad memory access • Long running GC can result in paging or CPU overload • Application code • Like 3rd Party drivers • (JDBC, MQ libraries) JVM from code to machine
- 20. • What to look for in java thread dumps: • Low Memory Detector • (Daemon thread to detect and report low memory conditions) • CompilerThreadn • (Daemon thread/s to compile & optimize byte code) • Signal Dispatcher • (Daemon thread to respond OS signals) JVM troubleshoot
- 21. • Finalizer • Daemon thread to handle objects in finalizer queue • Reference Handler • Daemon thread to handle objects in reference queue • VM Thread • Main thread from JVM • GC task thread#n • Thread/s to perform garbage collection JVM troubleshoot
- 22. • Liveset and fragmentation • Heap gets fragmented when small & large objects are mixed store with various lifetimes. May cause longer GC • Garbage collect time and behavior JVM troubleshoot
- 23. • Java HotSpot • Java Mission Control • JConsole • JVisualVM JVM tools
- 24. • Java Mission Control provides: • Console and command line (jcmd) • Live JVM statistics • Graphs and metrics about local or remote systems • Basic JMX interaction with MBeans • Runtime data • An in-depth look at live metrics, like memory, CPU and stack trace on specific threads • Profile methods or exceptions & execute commands • Remote or local Java Mission Control -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=3614 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
- 25. • Java Mission Control -XX:+UnlockCommercialFeatures-XX:+UnlockCommercialFeatures jcmd [PID] VM.unlock_commercial_features
- 26. • Diagnostic Volume must be set higher than the default (Low) Java Mission Control
- 27. Java Troubleshoot • Some well known errors • Class not found – missing or incorrect Classpath set during startup (server start params or setDomainEnv) or adding domain libs • NullPointers – A value is expected but it’s not there (null): • Poorly written • Bad error handling • Missing config • Stack Overflow – Recursive procedures, a method with no exit • Too Many Open Files – Ulimit settings
- 29. • Server applies to: Managed server or AdminServer • Partial or total (OutOfMemory) • Lack of resources/deadlocks Server Hangs
- 30. • The server does not respond to new requests. • Requests time out. • Requests take longer and longer to process (may be on the way to a hang). • A server crash is not usually a symptom of a hung server but may follow. Symptoms
- 31. Deadlock Thread 1 Thread 2 Resource A Resource B Lock Wait LockWait Deadlock
- 32. Possible Causes • All threads waiting for RJVM, RMI responses. • Garbage Collection taking too much time. • Long Running JDBC calls or , connection leaks, deadlocks • JVM hang during code optimization • JSP compilation under heavy load.
- 33. • WebLogic Server generic: • WebLogic Diagnostics Framework • Logs • WLST • JVM specific : • Java Mission Control, • JStat, • JConsole Tools/Methods
- 34. • Force a JVM dump by: • kill -3 <pid> … stack in STDOUT • Usually in <domain>/servers/logs/<serverinstance>/<serverinstance.out> • See more in the next slides about WebLogic Threads Tools/Methods
- 35. WebLogic Threads
- 36. • A threaddump is a snapshot of the state of a WebLogic Server Instance • Different ways to take a threaddump: • kill -3 <pid> the O/S way • WebLogic Console • WLST(WebLogic Scripting Tool) cd ('Servers') ls() cd ('AdminServer') ls() threadDump() WebLogic taking threaddumps
- 38. WebLogic Threads • Active threads: • Are either processing or waiting for requests • Can be tagged with additional attributes such as Idle or Hogging • Standby threads are not used to process requests unless throughput is needed. • Workmanagers can control threads of specific parts
- 39. WebLogic Stuck Threads Running for a long time • Deadlocked • Infinite loop • Overloaded server • WebLogic: • Periodically checked how long threads are running • Logs a warning if a thread becomes stuck • Creates additional threads to handle new requests • Sets its state to Failed after specified nr of threads become stuck • Individual work managers can detect stuck threads & reject new requests.
- 40. Stuck Thread possible actions
- 41. • Hanging threads will be diagnosed as STUCK • Default time to get STUCK is 600 secs • Thread has not completed it’s work The sense of taking threaddumps
- 42. • Thread Dumps can give you: • View of the state of application server threads at that instant in time • Information about potential issues such as • Hot spots within code which seem to be called often • Portions of code where the application seems to be hung • Locking and thread synchronization issues in an application The sense of taking threaddumps
- 43. • Take several dumps to investigate various states • Threads high in stack can be treated as idle, mostly waiting for some work to do • Blocked or Wait on socketAccept() listener thread waiting for socket connections The sense of taking threaddumps
- 44. • WebLogic specific: • Ignore threads in ExecuteThread.waitForRequest() • Muxer and processSocket threads are for housekeeping and listening • A Muxer which is blocked should not be waiting for a lock by a non Muxer The sense of taking threaddumps
- 45. • In the AdminConsole WebLogic taking threaddumps
- 46. • Are meant to synchronize between threads • Lock contention is a thread waiting for another to be released • Fat locks: Have a history of contention (several threads try to take the lock simultaneously), or waiting for notification • Thin locks: Without any contention. • Recursive locks: A lock on a thread several times without having been released. Understanding Locks
- 48. • Wrong or missing drivers in Classpath • Connection errors by: • Usernames/passwords • Connection leaks by bad apps • Set Inactive timeout to a value (default is 0) • Insufficient connection parameters • Max Capacity, Reserve Time Out(-1) • Invalid JDBC URLS weblogic.common.ResourceException: Could not create pool connection. The DBMS driver exception wa… JDBC additonal errors
- 49. WebLogic JDBC mechanism Application JDBC Driver Database1. Connection 2. Statement 3. SQL 4. Results 5. Close Pool Conn1 Conn2 JNDI Lookup
- 50. • Using dumpPool and WLDF capture JDBC Profiling
- 51. • DebugJDBCConn - Trace all connections. • DebugJDBCSQL - Trace all JDBC API calls • DebugJDBCInternal - low-level internal data source activities. • DebugJTAJDBC - Trace transaction management Debug Flags
- 52. • Bad implemented applications can starve connections and data • Cache or “hog” connections instead of releasing them. • Do not explicitly release connections using close(). • Automatically reclaim with “Inactive Connection Timeout” Connection Leaks
- 54. • Incorrect targeting • Connection factories to subdeployments instead of to entire servers • Targeting distributed destinations to entire servers instead to subdeployments • Using multiple subdeployments within the same module • Configure JMS migration without a cluster • Convert a single-server domain to a cluster Common JMS issues
- 55. • OOM in case of large amount of pending messages • JMS Quota can prevent that • Message paging – messages in virtual memory(file or db) • Flow control Common JMS issues - OOM
- 56. • Expired messages • Set a TimeToLive & Exp policy • JMS system failure and nonpersistent messages • Set a delivery and time to delivery mode • Insufficient retry settings • Message delay settings • Nondurable topic subscriptions Common JMS issues – Lost messages
- 57. • Applications used to automatically consume JMS • Often used in SOA Suite applications ( SOA Suite, OSB ) • Manages itself ( pooling ) • A few debug flags or use WLDF to diagnose MDBs
- 58. WebLogic Log and debug methods
- 59. • Logging services are important for diagnostics • WebLogic produces output by using a Message Catalog • Using the standard java.util.logging • Can be extended using log4j modules • All kinds of events, like startup/shutdown, appl.deployments, subsystem failures are written to logs WebLogic Log and Debug options
- 61. • Several severity levels like: • TRACE, DEBUG,INFO .. • Important to set the proper level • For FMW specific components, ODL-HANDLER can be used WebLogic Log and Debug options
- 62. • Subsystem logs can be from: • HTTP • Transaction Manager • JDBC • JMS server • WebLogic Auditing provider • FMW Logs WebLogic Log and Debug options
- 63. • Debug • Setting debug options in the Admin Console • Or use –D options in startup script -Dweblogic.debug.DebugJDBCSQL=true Prints information about all JDBC methods WebLogic debug options
- 64. • Or use WLST serverConfig() or domainConfig() debug = getMBean('/Servers/mserver/ServerDebug/mserver') debug.setDebugJDBCInternal(true) WebLogic debug options
- 65. • OpenSource Log engine from Apache • Enable the LogMBean.isLog4jLoggingEnabled attribute • In WLST: cmo.setLog4jLoggingEnabled(true) • Add Log4j classes, • WL_HOME/server/lib/wllog4j.jar & log4j.jar to server CLASSPATH WebLogic Log4j
- 66. • Enable log4j in AdminConsole WebLogic Log4j
- 67. • As an extension on the default logging handlers • Comes with FMW installation, not WebLogic default • Logging view and config in Fusion Middleware Control • Can be set to several trace levels • Output in <servername>-diagnostic.log Oracle diagnostic Logging (ODL)
- 68. Oracle diagnostic Logging Runtime
- 69. Oracle diagnostic Logging • Edit or create your own handler
- 70. Oracle Diagnostic Logging • Edit or create your own handler
- 72. WebLogic Diagnostic Framework •Included with WebLogic •Define Watches to monitor runtime conditions and notify in case of failure. •Collect configuration and runtime data for analysis. •Capture images to preserve system state in failure situation. Watch & Notification Instrumentation Data Collectors Image Capture Data Providers Archive Accessor Monitoring Dashboard
- 73. • Framework for monitoring and diagnostics • Consolidated access for MBean data, instrumentation, logging, debugging • Source for monitoring data • Server & application instrumentation notifications based on the output : log, email, SNMP, JMX, JMS • Request dyeing and tracing • Data archiving for later access • WLDF Dashboard WLDF
- 74. • Key Features • Gathering input from JMX MBeans • WebLogic subsystems and application code can be instrumented to provide events and watches • Notifications can be defined for events and watches • Console extension for monitoring of current and historical data • Data archiving WLDF
- 75. • Can be configured on filesystem or Database • Configure with Data retirement WLDF
- 76. • Create Diagnostic Module • Collect Metrics. A collection includes: • The MBean type to query • The specific MBean instance names to query (all instances, • by default) • The MBean attributes to collect (all attributes, by default) • How often to gather data • Create Watches & Notifications WLDF
- 77. Diagnostic Frameworks & Diagnostics Tools in FMW • WLDF – WebLogic Diagnostic Framework • DFW – Diagnostic Framework o Dynamic Monitoring Service (DMS) o Oracle Diagnostic Logging (ODL) o Incident Packaging System (IPS or ADRCI) • Additional Tools o Selective Tracing
- 78. WebLogic Diagnostic Framework Oracle Fusion Middleware
- 79. Diagnostic Tools for SOA and FMW • Predefined Watches and Diagnostic Dumps • Detect, diagnose and resolve problems • Use WLS and SOA MBeans and DMS Metrics. • Detect critical failures and collect diagnostic dumps containing relevant diagnostic information like logs, metrics, server images, • Create incidents in the ADR (Automatic Diagnostic Repository)
- 80. Integration of WLDF, DFW and Diagnostic Tools for SOA and FMW Monitor System • Predefined Watches • Predefined Diagnostic Dumps Critical Failure • Collect Diagnostic Dumps • Create Incident Collect • Collect Data using EM or Email
- 81. Dynamic Monitoring Service • DMS for runtime metrics applications and components • Can be monitored through WLDF Watches to trigger data collections through DFW.
- 82. Dynamic Monitoring Service (DMS) • Direct accessible through http://<wlshost>:7001/dms • Give pretty much raw unstructured results • Better option is to use FMW Console
- 83. DMS in FMW Console
- 84. Diagnostic Framework (DFW) • collect a pre-configured set of information at exactly the right time through a triggering mechanism. • Triggered from Oracle Diagnostic Logging events or WebLogic Diagnostic Framework Watches. • Information can also be collected manually using WLST
- 85. Oracle Diagnostic Logging (ODL) • ODL is the primary logging for FMW • The ODL log entries are in XML format
- 86. Selective Tracing (ST) • Selective Tracing is a facility available through Enterprise Manager to limit the scope of trace logging. • This allows for debugging of a production system without overloading the system with logging activity.
- 89. Performance
- 90. • Performance : How system’s & applications response time and throughput are affected by adding load. • Capacity : Maximum threshold a system is under a given set of conditions. • Scalability : how does a system responds to increasing load by adding additional resources. • Key indicators are : • Response time • Throughput Performance Terms and Definitions
- 91. • Locate & minimize bottlenecks • Gather info about Application(s) • Concurrent number of users and interfaces. • Number and size of app requests. • Amount of data and its consistency. • Determining your target CPU utilization • Tune your Database(s) (customers or repos) • Tune WebLogic Server Performance Parameters • Tune Your JVM • Tune the Operating System • Tuning WebLogic Persistent Store Performance objectives
- 92. • Investigate and analyze! • Always consider before implementing : • What am I changing • Has the bottleneck gone away? • Is there a new bottleneck? Tune WebLogic Server
- 93. • Enable by default • Will use a O/S threadpool instead from “borrowing” from WebLogic default Exec.Q • Native muxers use platform-specific native binaries to read data from sockets • Non native or Java sockets : • Uses pure Java to read data from sockets. • Blocks until there is data to be read from a socket(can cause issues on server level) Native I/O and muxers
- 94. • Represents nr of simultaneous operations performed by apps that use the execute Q WebLogic Thread Pool WebLogic Socket Muxer Application WebLogic Kernel assigns Execute Req to ExecuteThread 1 ExecuteThread 1 handles the request (for example executes a JDBC request) WebLogic Kernel handles self tuning ThreadPool, Workmanagers, allocations The app now handles the thread itself and gives back result Execute Request
- 95. • Different execute Q’s in 11g for applications and administration work • weblogic.kernel.default, RMI, HTTP • 1 default Queue • In 12c Single threadpool • Self tuning • Work managers • Prioritize by rules • Different constraints regarding min max response capacity -Dweblogic.threadpool.MinPoolSize=5 -Dweblogic.threadpool.MaxPoolSize=5 WebLogic Thread Count
- 96. • Default All apps use all execute queues • Configuring multiple execute queues can provide additional control for applications. • By using multiple execute queues, selected applications have access to a fixed number of execute threads, regardless of the load on WebLogic Applications and Execute Queues
- 97. • ThreadPoolPercentSocketReaders • Default set to 33 ( range is 1-99) • Allocating threads as readers can increase accepting requests Threads as SocketReaders
- 98. • Piece of memory for read and write data to sockets (network layer) • A pool of chunks is maintained • For applications with large amounts of data • Parameters are: • weblogic.Chunksize • Size of a chunk (in bytes). Set to network's maximum transfer unit (MTU), after subtract it from any Ethernet or TCP header sizes • weblogic.utils.io.chunkpoolsize • Sets the maximum size of the chunk pool. Monitor the CPU profile or use a memory/ heap profiler for call stacks invoking the constructor weblogic.utils.io.Chunk. • weblogic.PartitionSize • Sets the number of pool partitions used (default is 4). Partitioning the thread pool spreads the potential for contention over more than one partition To be set in the setDomainEnv.sh WebLogic ChunkSize
- 99. • Specifies how many TCP connections can be buffered. These are on the TCP stack but not yet accepted by an app. • Default is 50 • Maximum depends on O/S WebLogic Backlog
- 100. • When to tune? • When getting “connection refused” in high throughput apps (SOA composites) • If tuning, increase by 25% of it’s default until errors dissapear WebLogic Backlog
- 101. • Prepared Statement Cache • Keeps compiled SQL statements in memory • Set to LRU • A data source with 10 connections deployed on 2 servers, with the Statement Cache to 10 (default), 200 cursors can be used on the database for these cached statements • MaxCapacity • Max of physical connections • Db drivers can limit connections • Number should equal number of client sessions • LLR instead of XA • Decrease 2PC overhead • Disable wrapping datatypes (debug, track, transaction control of blb, clob) JDBC Connectionpool tuning
- 102. • In some cases it takes ages after startup to continue • Generate a stackup Tuning Startup
- 103. • This happens when using default seed generator: • Use of dev/urandom during startup in stead of random • Specifiy in startup properties - Djava.security.egd=file:///dev/urandom • Or in $JAVA_HOME/jre/lib/security/java.security securerandom.source=file:/dev/./urandom Tuning Startup
- 104. • Keep MDS and Dehydration store clean • MDS delete older label deployments • Dehydration store purge strategy Tuning Startup SOA
- 105. • Configure Transaction settings: • BPEL SyncMaxWaitTime • BPEL EJBS • JTA • DB Tuning other SOA
- 106. : • Discovery cache age to 28800000 • discovery wait time to 30000 • Set the cache results to true • Dumpinterval to 10800 with a maximum of 75 MB • Enable Large Repository • Increase the collector configuration settings for DMS: • Use an interval of 300 secs with a remove cycle of 3 • And use an interval of 120 secs with a remove cycle of 2, set this to default • Restrict the amount of displayed instance information to 12 hours • Tuning AdminServer EM SOA
- 107. • Most aspects of JVM tuning relate to: • Sufficient memory heap • An efficient garbage collection scheme • Classloading • Compile/interpret Tuning the JVM
- 108. • Most important to focus on: • Garbage collection scheme • Compaction/Defrag • Free up memory • Memory heap • Is a runtime repository for live objects, dead objects and free memory. • No more than 75% of total RAM • The JVM heap sets frequency & duration GC • Large heap size : • full garbage collection slower and less frequent • Small heap size • full garbage collection is faster & higher frequency Tuning the JVM
- 109. • Goal is: • Minimize Garbage Collect • Set heapsize that GC does no happen all the time Tuning the JVM
- 110. • Investigate with –verbosegc option • How often does GC run • How long does it take? • Full garbage collection should not take longer than 3 to 5 seconds. Lower heap if major GC time is greater. Tuning the JVM
- 111. • Set Xms and Xmx parameters to specify initial and Max HeapSize • Xmx is the Java Object Heap, not the System Heap –> which is larger • Values equal will cause no overhead of heap management • PermSize and MaxPermSize For stored classes, methods Be aware: • Java methods, thread stacks, native handles are in memory separate from heap Tuning the JVM
- 112. “The Cloud is next”
- 114. Any Questions? Let’s keep in touch!! The ACE Community mschildmeijer@qualogy.com https://community.oracle.com/blogs/mnemonic https://www.qualogy.com/nl/techblog/author/michel- schildmeijer
- 115. Any Questions? Let’s keep in touch!! @MNEMONIC01 @Qualogy_news nl.linkedin.com/in/mschldmr