Debugging Your Production JVM
- 1. Debugging Your Production JVM Machine Ken Sipe Perficient (PRFT) kensipe@gmail.com
- 2. Abstract > Learn the tools and techniques used to monitor, trace and debugging running Java applications. 2
- 3. Agenda > Java Memory Management > Memory Management Tools • Command-line Tools • VisualVM > Btrace > Summary > Resources 3
- 4. Agenda > Java Memory Management > Memory Management Tools • Command-line Tools • VisualVM > Btrace > Summary > Resources 4
- 5. Donʼt Worry… > C (malloc / free) > C++ (new / delete) > Java (new / gc) > Memory Allocation / Deallocation in Java is automatic 5
- 6. Object Lifetimes > Most objects live very short lives • 80-98% of all newly allocated objects die • within a few million instructions • Before another megabyte is allocated > Old objects tend to live a long…. time
- 7. Memory Spaces > New Space • Where all objects are new (sort of…) > Old Space • Where all objects are old New Space Old Space
- 8. Memory Spaces > New Space Division • Eden • Where all objects are created • Survivor Space 0 • Provide object aging • Survivor Space 1 Eden Old SpaceSS0 SS1
- 9. Perm Spaces > Permanent Space • class information • static information Eden Old SpaceSS0 SS1 Perm Space
- 10. GC Responsibility > Heap Walking from GC Roots > Mark / Sweep • Garbage detection (mark) • Sort out the live ones from the dead ones • Reference counting • Garbage reclamation (sweep) • Make space available
- 11. Minor Garbage Collection > Minor gc (scavenge) • When eden is “full” a minor gc is invoked • Sweeps through eden and the current survivor space, removing the dead and moving the living to survivor space or old • Ss0 and ss1 switch which is “current” • A new tenuring age is calculated
- 12. Major Garbage Collection > Major gc • When old is “full” • All spaces are garbage collected including perm space • All other activities in the jvm are suspended
- 14. Agenda > Java Memory Management > Memory Management Tools • Command-line Tools • VisualVM > Btrace > Summary > Resources 14
- 15. Java Memory Tools > JPS • Getting the Process ID (PID) > Jstat • jstat -gcutil <pid> 250 7
- 16. Looking at the Heap > %JAVA_HOME%/bin/jmap – histo:live <pid> • Looking at all the “live” objects > %JAVA_HOME%/bin/jmap – histo <pid> • Looking at all objects > The difference between is the list of unreachable objects
- 17. Taking a Heap Dump > %JAVA_HOME%/bin/jmap – dump:live,file=heap.out,format=b <pid> • Dumps the Heap to a file > JConsole
- 18. JHat > %JAVA_HOME%/bin/jhat <filename> • Starts a web server to investigate the heap > Queries • Show instance count for all classes • Show Heap Histogram • Show Finalizer • Use the Execute Object Query Language (OQL) • select s from java.lang.String s where s.count >=100
- 19. JMX – Looking at Flags
- 20. MAT – Memory Analyzer Tool
- 21. VisualVM > Open Source All-in-One Java Troubleshooting tool > https://visualvm.dev.java.net/ 21
- 22. Visualgc > visualgc <pid> > Visual Garbage Collection Monitoring • a graphical tool for monitoring the HotSpot Garbage Collector, Compiler, and class loader. It can monitor both local and remote JVMs.
- 23. 23 DEMO
- 24. Agenda > Java Memory Management > Memory Management Tools • Command-line Tools • VisualVM > Btrace > Summary > Resources 24
- 25. BTrace > dynamically bytecode instrumenting (BCI) • read / not write • probe-based • observe running Java applications > Integration with DTrace on Solaris > http://btrace.dev.java.net > visualvm plugin 25
- 26. BTrace Tools 26 Target JVM BTrace Agent VisualVM + BTrace Plugin BTrace command-line
- 27. BTrace Terminology > Probe Point • “location” or “event” at which tracing statements are executed > Trace Actions • statements which are executed whenever a probe fires > Action Methods • static methods which define a trace 27
- 28. Probes and Actions > Probe Targets • method entry / exit • line number • exceptions • return from method • exception throw (before) • synchronization entry / exit > Actions • static methods in trace class 28
- 29. BTrace Restrictions > no new objects > no new arrays > no exceptions > no outer, inner, nested or local classes > no synchronization blocks > no loops > no interfaces 29
- 30. Tracing > Annotations • com.sun.btrace.annotations • @BTrace • denotes a btrace class > Probe Points • @OnMethod • @OnTimer • @OnEvent • @OnExit 30
- 31. Simple Example: Looking for Object Size 31
- 32. Simple Example: Looking for Object Size 32
- 33. BTrace Run Options > BTrace Command-Line • btrace • runs btrace tool (and compiles btrace script if necessary) • btracec • btrace script compiler • btracer • convenience script to start java project with tracing enabled at application startup > VisualVM + BTrace Plugin 33
- 34. Testing the Probe > Start Target Application • java -jar java2demo.jar > Get the PID • jps > Inject Probe • btrace <pid> Sizeof.java 34
- 35. BTrace + JMX Script 35
- 37. ThreadLocal 37
- 38. BTrace Events 38
- 39. Checking Synchronization Entry / Exits 39
- 40. Tracing Opportunities > Thread Monitors > Socket / Web Services > Object Creation and Size > File Access 40
- 41. 41 DEMO
- 42. Known Solutions with BTrace > Terracotta • concurrency issue > Hibernate / Atlassian Issue • thread / session management issue • http://jira.atlassian.com/browse/CONF-12201 42
- 43. Summary > jmap, jhat, jconsole, jstat • tools already in the JDK bin directory > VisualVM • swiss army knife for JVM process monitoring and tracing > BTrace • dynamic tracing tool 43
- 44. Resources > Performance and Troubleshooting • http://java.sun.com/performance/reference/ whitepapers/6_performance.html • http://java.sun.com/javase/6/webnotes/trouble/TSG- VM/html/docinfo.html > VisualVM • https://visualvm.dev.java.net/ > BTrace • https://btrace.dev.java.net/ 44