Web Sphere Problem Determination Ext
6 likes2,398 views
This document provides an overview and agenda for the "Busy Java Developer's Guide to WebSphere Debugging & Troubleshooting" presentation. The presentation covers various WebSphere Application Server components, troubleshooting tools like IBM Support Assistant, JVM troubleshooting tools, problem determination tools, common problem scenarios, how customers run into trouble, and includes a demo and Q&A section. It provides an in-depth look at debugging and resolving issues with WebSphere Application Server.
![Why am I getting Out Of Memory
Errors ?
16
Java 1.4.* -look for heap fragmentation (rare for Java 5/6)
Set –Xp and –Xk for pinned and dosed objects
JVM Heap sized incorrectly
IBM JDK Team does NOT recommend Xms == Xmx
GC adapts heap size to keep occupancy [40, 70]%
Determine heap occupancy of the app. under load
Xmx = 43% larger than max. occupancy of app.
For 700MB occupancy , 1000MB Max. heap is reqd. (700 +43% of 700)
Analyze heapdumps & system dumps with dump tools
Lack of Java heap or Native heap
Finding which methods allocated large objects
Prints stacktrace for all objects above 1K
-Xdump:stack:events=allocation,filter=#1k
Enable Java Heap and Native heap monitoring
JMX and PMI metrics output by WAS JVM
Classloader exhaustion in pre Java 5 SR3](https://image.slidesharecdn.com/websphereproblemdetermination-ext-100401135150-phpapp01/85/Web-Sphere-Problem-Determination-Ext-16-320.jpg)
Web Sphere Problem Determination Ext
- 1. THE BUSY JAVA DEVELOPER'S GUIDE TO WEBSPHERE DEBUGGING & TROUBLESHOOTING ROHIT KELAPURE IBM ADVISORY SOFTWARE ENGINEER HTTP://WWW.LINKEDIN.COM/IN/ROHITKELAPURE HTTP://TWITTER.COM/RKELA HTTP://WASDYNACACHE.BLOGSPOT.COM/ May 1, 2010 IBM WebSphere Application Server
- 2. Agenda 2 WebSphere Application Server component overview IBM Support Assistant JVM Troubleshooting Tools WebSphere Problem Determination Tools Scenario based problem resolution How customers get in trouble BadApp Demo Q&A
- 4. IBM Support Assistant Workbench to help with Problem Determination 4
- 5. ISA V 4.1.1 Data Collection Remote Collection using ISA lite 5 2 1 3 4
- 6. ISA Tools Single Repository, up-to-date, scripts with IGAA 6
- 7. Tools for information generated by JVM Problem 7 Artifact Tools Memory leaks Verbose Garbage collection • Garbage Collection & Out of Memory errors log (native_stdout.log) Memory Visualizer Application Unresponsive • Pattern Modeling & Analysis tool High CPU, Crash, Hang, Javadump, Javacore • Thread Monitor & Dump Performance bottleneck, (javacore*.txt) Analyzer (TMDA) Unexpected termination Lock Contention Threads (Connection to running • Monitoring & Diagnostic Tools Low CPU at high load JVM) for Java – Health Center Memory Leak Heapdump (*.phd, *.txt, • Memory Analyzer Tool Out of Memory errors *.hprof) • HeapAnalyzer • MDD4J Native Memory Leak System or core dump • Dump Analyzer Anomalies (core.dmp, user.dmp), Files must • Modules for Dump Unexpected Crash be processed with jextract tool Analyzer
- 8. WAS Runtime Serviceability aids 8 Troubleshooting panels in the admin console Performance Monitoring Infrastructure metrics Tivoli Performance Viewer Monitoring Diagnostic Provider Mbeans Dump Configuration, State and run self-test Application Response Measurement/Request Metrics Follow transaction end-to-end and find bottlenecks Trace logs & First Failure Data Capture Runtime Performance Advisors Memory leak detection, session size, … Specialized tracing and Runtime checks Session crossover, Connection leak, WsByteBuffer leak detection … zOS runaway CPU thread protection Diagnostic Toolkit & Framework for Java API for accessing information in a system dump in an OS agnostic way
- 9. WebSphere Troubleshooting Problem Tools, Runtime capabilities 9 Thread correlation, Trace log analysis WAS Trace Analyzer Delays in system response WAS & Plugin Trace & Request Analyzer Connection Leaks • Database Connection Pool Analyzer • Connection Leak Trace facility • Enabled using trace string WAS.j2c=finest Server hang Thread Monitor - Hung Thread Detection Policy • Every 3 minutes detects threads active for > 10 min Classloader issues Classloader viewer in the WAS Admin console Corrupt Install Install Verification Utility (installver, ivt.bat/sh) Runtime network status Port Scanning tool Comparing & Validating • Visual Configuration Explorer Configuration files • WebServices validation tool for WSDL & SOAP HTTP Plugin & Proxy (No response, • WebServer Plugin Analyzer Incorrect load balancing) • Dump ODC tree script Issues with Security LDAP Server test connection, SSL Certificate & Key Management, Security Configuration Report
- 10. One more tool and I am going to scream
- 11. WebSphere Monitoring tools 11 Sun origin JVM Tools - visualVM & jConsole, SIBus – Explorer & Destination Handler Dynacache - Cache Monitor & Extended Cache Monitor HTTPSession - Session tracker debug Servlet* WebServices Request/Response tracking TCPMonitor & Fiddler dumpNameSpace tool osgiConsole.sh/bat Coregroup Visualization and analysis tool* Sun JVM ONLY Tools- jps, jstatd, jstack, jmap, jinfo, jhat LDAP tools- ldapsearch, apache LDAP browser JVM Diagnostics Collector
- 12. Most common JVM Problem Scenarios 12 Functional Problems • Unexpected Exceptions, Compatibility OOM Errors • Java Heap ,Native Heap Classloaders Hangs • Synchronized resources, GC Pause times Crash • JVM errors, JIT errors, JNI errors High CPU • Spin loops
- 13. Have you ever pondered … 13 Why does my app. run slow every time I do <…> ? Why does my app. have erratic response times ? Why am I getting Out of Memory Errors ? What is my applications memory footprint ? Which parts of my app. are CPU intensive ? How did my JVM vanish without a trace ? Why is my application unresponsive ? $%@ is wrong with my application ? What monitoring do I put in place for my app. ?
- 14. Why does my app run slow when I do xxx ? 14 Understand impact of activity on WAS components Look at the thread & method profiles in the IBM Java Health Center Set IBM JVM method & dump trace - pinpoint performance problems. Shows entry & exit times of any Java method Method to trace to file for all methods in tests.mytest.package -Xtrace:maximal=mt,output=trace.out,methods={tests/mytest/*.*} Allows taking javadump, heapdump, etc when a method is hit Dump javacore when method testInnerMethod in an inner class TestInnerClass of a class TestClass is called -Xtrace:trigger=method{com/ibm/TestClass$TestInnerClass.testInnerMethod,javadump,,,5} -Xtrace and -Xdump can be triggered on a huge range of events … with filters and ranges gpf, user, abort, fullgc, slow, allocation, thrstop, throw … Stack traces, tool launching
- 15. Why does my application have erratic response times ? 15 Verbose gc should be enabled by default (<2%) impact on performance GCMV &PMAT : Visualize gc output & look @ recos. In use space after GC Positive gradient indicates memory leak Increased load (use for capacity plan) OR memory leak (take HDs for PD.) Chose the right GC policy -Xgcpolicy:optthruput - optimized for “batch” type applications, consistent allocation profile -Xgcpolicy:optavgpause - tight responsiveness criteria, allocations of large objects -Xgcpolicy:gencon - high rates of object “burn”, large # of transitional objects -Xgcpolicy:subpools - 12, 16 core SMP systems with allocation contention (AIX only) GC overhead > 10% wrong policy or more tuning Enable compressed References enabled for 64 bit JVM ?
- 16. Why am I getting Out Of Memory Errors ? 16 Java 1.4.* -look for heap fragmentation (rare for Java 5/6) Set –Xp and –Xk for pinned and dosed objects JVM Heap sized incorrectly IBM JDK Team does NOT recommend Xms == Xmx GC adapts heap size to keep occupancy [40, 70]% Determine heap occupancy of the app. under load Xmx = 43% larger than max. occupancy of app. For 700MB occupancy , 1000MB Max. heap is reqd. (700 +43% of 700) Analyze heapdumps & system dumps with dump tools Lack of Java heap or Native heap Finding which methods allocated large objects Prints stacktrace for all objects above 1K -Xdump:stack:events=allocation,filter=#1k Enable Java Heap and Native heap monitoring JMX and PMI metrics output by WAS JVM Classloader exhaustion in pre Java 5 SR3
- 17. What is my applications memory footprint ? 17 HPROF – profiler shipped with JDK – uses JVMTI Analysis of memory usage -Xrunhprof:heap=all Performance Inspector tools - JPROF Java Profiling Agent Capture state of the Java Heap later processed by HDUMP Use MAT to investigate heapdumps & system dumps Find large clumps, Inspect those objects, What retains them ? Why is this object not being garbage collected List Objects > incoming refs, Path to GC roots, Immediate dominators Limit analysis to a single application in a JEE environment Dominator tree grouped by Class Loader Set of objects that can be reclaimed if we could delete X Retained Size Graphs Traditional memory hogs like Session, Dynacache Use Object Query Language (OQL)
- 18. Which parts of my application are CPU intensive ? 18 HPROF - CPU spends most of its time -Xrunhrof:cpu=samples, -Xrunhprof:cpu=time JPROF – method level execution times, who calls whom, etc. Generate WAS startup script & set the JVM argument "-agentlib:jprof=rtarcf,callflow,logpath=./jprof" "-Xjit:disableInlining“ Output visualized using VPA Poor mans profiler - Periodic javacores -Thread analysis - TMDA IBM – Health Center Low overhead Gets data from JIT
- 19. How did my JVM vanish without trace ? 19 JVM Process Crash Usual Suspects Bad JNI calls, Segmentation violations, Call Stack Overflow Native memory leaks - Object allocation fails with sufficient space in the JVM heap Unexpected OS exceptions (out of disk space, file handles), JIT failures Monitor the OS process size Runtime check of JVM memory allocations -Xcheck:memory Native memory usage - Create a core or svc dump on an OOM -Xdump:system:events=systhrow,filter=java/lang/OutOfMemoryError JNI code static analysis -Xcheck:jni (errors, warnings, advice) GCMV provides scripts and graphing for native memory Windows “perfmon“, Linux “ps” & AIX “svmon” IBM Dump Analyzer & TMDA to find the last stack of native code executing on the thread during the crash
- 20. Why is my application unresponsive ? 20 Usual Suspects (25-50% overhead of an app.) Thread Contention & sync. issues Deadlocks and Spin loops Take three javadumps spaced 1 minute apart Native stacks in javadumps in Java 6, SR7 Monitor hold time accounting & Contention statistics Java Lock Monitor, IBM Health Center (always on profiling) Monitor hung thread messages WSVR0605W (Thread hung), WSVR0606W (Thread completed), WSVR0607W (threshold adjusted) Cause javacore on thread hang - Set“com.ibm.websphere.threadmonitor.dump” custom property to “true” Spawning new threads is an anti-pattern since these threads are outside the hang detection monitoring Is traffic flowing into the application server from the Plugin or the Proxy ? Increasing OS level TCP/IP connection backlog at the webserver/proxy IBM Health Center starts up on port 1972 for WAS and needs a generic JVM argument “-agentlib:healthcenter” Overhead < 3%
- 21. What is wrong with my application ? General problem determination steps 21 Finding the Bottleneck CPU, Memory, I/O, Network Systems Resource Usage top, perfmon, svmon, tasklist, ps, <add your favorite utility here …> JVM Heap Sizing Setting Correct GC Policy, Heap occupancy JIT Compilation & Performance Application Profiling Inbuilt hprof, Health Center Free - Performance Inspector tools (JPROF, TPROF, JLM, Jinsight …) Commercial- RAD, Jprofiler, Yourkit … OS specific tuning Setting TCP TIME_WAIT
- 22. What do I monitor ? 22 OS Metrics PMI
- 23. 11 Malpractices 23 no arch. plan No migration plan No change records No Capacity plan No Production traffic profile Changes put directly in Prod. No load & Stress testing Communication breakdown No education Application Error Test environment != Production
- 24. Demo, Q&A 24
- 25. I would like to thank … Please google & read their blogs & articles 25 Daniel Julin (IBM Lab) John Pape (SWAT) Kevin Grigorenko (SWAT) Art Jolin (ISSW) Chris Bailey (JDK) Katie Barett (Serviceability team) Russell Wright (ISA) Thomas Alcott (Tech. Sales)
- 26. 26