![2016-06-26 17:13:23
Full thread dump Java HotSpot(TM) 64-Bit Server VM (23.7-b01 mixed mode):
"Hashed wheel timer #9261" prio=10 tid=0x00007f0442e17800 nid=0x112b waiting on condition
[0x00007f042e709000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:483)
at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:392)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at java.lang.Thread.run(Thread.java:722)
"Reconnection-1" prio=10 tid=0x00007f0442e10800 nid=0x112a waiting on condition [0x00007f042f719000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x007b3953a98> (a java.util.concurrent.locks.AbstractQueuedSynchr)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.lang.Thread.run(Thread.java:722)
:
:
1 2
3
Anatomy of Thread Dump
1 Timestamp at which thread dump was triggered - 2016-06-26 17:13:23
2 JVM Version info
3 Thread Details - <<details in following slides>>](https://image.slidesharecdn.com/dontdumpthreaddumps-160921114332/85/Don-t-dump-thread-dumps-7-320.jpg)
![Thread Details
"InvoiceThread-A996" prio=10 tid=0x00002b7cfc6fb000 nid=0x4479 runnable [0x00002b7d17ab8000]
java.lang.Thread.State: RUNNABLE
at com.buggycompany.rt.util.ItinerarySegmentProcessor.setConnectingFlight(ItinerarySegmentProcessor.java:380)
at com.buggycompany.rt.util.ItinerarySegmentProcessor.processTripType0(ItinerarySegmentProcessor.java:366)
at com.buggycompany.rt.util.ItinerarySegmentProcessor.processItineraryByTripType(ItinerarySegmentProcessor.java:254)
at com.buggycompany.rt.util.ItinerarySegmentProcessor.templateMethod(ItinerarySegmentProcessor.java:399)
at com.buggycompany.qc.gds.InvoiceGeneratedFacade.readTicketImage(InvoiceGeneratedFacade.java:252)
at com.buggycompany.qc.gds.InvoiceGeneratedFacade.doOrchestrate(InvoiceGeneratedFacade.java:151)
at com.buggycompany.framework.gdstask.BaseGDSFacade.orchestrate(BaseGDSFacade.java:32)
at com.buggycompany.framework.gdstask.BaseGDSFacade.doWork(BaseGDSFacade.java:22)
at com.buggycompany.framework.concurrent.BuggycompanyCallable.call(buggycompanyCallable.java:80)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
1 2 3 4 5
6
7
1 Thread Name - InvoiceThread-A996
2 Priority - Can have values from 1 to 10
3
Thread Id - 0x00002b7cfc6fb000 - In Oracle/Sun's JDK implementation, this ID is simply a auto-incrementing long, starting from 1. The thread ID is unique
during its lifetime. When a thread is terminated, this thread ID may be reused. It's platform independent. It's returned by calling the Thread.getId() method.
4 Native Id - 0x4479 - This ID is highly platform dependent. On Linux, it's the pid of the thread. On Windows, it's simply the OS-level thread ID within
a process. On Mac OS X, it is said to be the native pthread_t value.
5 Address space - 0x00002b7d17ab8000 -
6 Thread State - RUNNABLE
7 Stack trace -](https://image.slidesharecdn.com/dontdumpthreaddumps-160921114332/85/Don-t-dump-thread-dumps-10-320.jpg)
Don't dump thread dumps
- 1. Don’t Dump Thread Dumps Ram Lakshmanan Founder – GCEasy.io & FastThread.io
- 2. Agenda • How to take Thread Dumps? • Anatomy of Thread Dump • Troubleshooting – Real world problems through Patterns • Sudden CPU spikes • Out Of Memory Error • Unresponsive DB • Unresponsive NoSQL • Unresponsive Application
- 3. jstack -l <pid> > /tmp/threadDump.txt jcmd <pid> Thread.print > /tmp/threadDump.txt Jstack (since Java 5) kill -3 jVisualVM JMC Windows (Ctrl + Break) ThreadMXBean APM Tools Jcmd (since Java 7) 1. 2. 3. 4. 5. 6. 7. 8. 1 4 7 2 5 8 3 6 How to take thread dumps? – 8 options
- 4. jstack -l <pid> > /tmp/threadDump.txt jcmd <pid> Thread.print > /tmp/threadDump.txt Jstack (since Java 5) kill -3 jVisualVM JMC Windows (Ctrl + Break) ThreadMXBean APM Tools JCMD (since Java 7) 1. 2. 3. 4. 5. 6. 7. 8. How to take thread dumps? – 8 options How to take thread dumps? Click here for read the full blog
- 5. Best Practice • 3 thread dumps in a gap of 10 seconds
- 6. Agenda • How to take Thread Dumps? • Anatomy of a Thread Dump • Troubleshooting – Real world problems through Patterns • Sudden CPU spikes • Out Of Memory Error • Unresponsive DB • Unresponsive NoSQL • Unresponsive Application
- 7. 2016-06-26 17:13:23 Full thread dump Java HotSpot(TM) 64-Bit Server VM (23.7-b01 mixed mode): "Hashed wheel timer #9261" prio=10 tid=0x00007f0442e17800 nid=0x112b waiting on condition [0x00007f042e709000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:483) at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:392) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at java.lang.Thread.run(Thread.java:722) "Reconnection-1" prio=10 tid=0x00007f0442e10800 nid=0x112a waiting on condition [0x00007f042f719000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x007b3953a98> (a java.util.concurrent.locks.AbstractQueuedSynchr) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.lang.Thread.run(Thread.java:722) : : 1 2 3 Anatomy of Thread Dump 1 Timestamp at which thread dump was triggered - 2016-06-26 17:13:23 2 JVM Version info 3 Thread Details - <<details in following slides>>
- 8. Found one Java-level deadlock: ============================= "Thread-1": waiting to lock monitor 0x0000000007252140 (object 0x00000007ac3b1970, a java.lang.Object), which is held by "Thread-0" "Thread-0": waiting to lock monitor 0x0000000007250d90 (object 0x00000007ac3b1980, a java.lang.Object), which is held by "Thread-1" Java stack information for the threads listed above: =================================================== "Thread-1": at com.tier1app.DeadLockSimulator$SecondThread.run(DeadLockSimulator.java:29) - waiting to lock <0x00000007ac3b1970> (a java.lang.Object) - locked <0x00000007ac3b1980> (a java.lang.Object) "Thread-0": at com.tier1app.DeadLockSimulator$FirstThread.run(DeadLockSimulator.java:16) - waiting to lock <0x00000007ac3b1980> (a java.lang.Object) - locked <0x00000007ac3b1970> (a java.lang.Object) Found 1 deadlock. 4 4 Optional – if deadlock is detected
- 9. Heap def new generation total 458752K, used 292671K [0x62c00000, 0x82c00000, 0x82c00000) eden space 393216K, 57% used [0x62c00000, 0x709cfe50, 0x7ac00000) from space 65536K, 100% used [0x7ec00000, 0x82c00000, 0x82c00000) to space 65536K, 0% used [0x7ac00000, 0x7ac00000, 0x7ec00000) tenured generation total 1523712K, used 743702K [0x82c00000, 0xdfc00000, 0xdfc00000) the space 1523712K, 48% used [0x82c00000, 0xb0245a40, 0xb0245b00, 0xdfc00000) compacting perm gen total 262144K, used 70366K [0xdfc00000, 0xefc00000, 0xefc00000) the space 262144K, 26% used [0xdfc00000, 0xe40b7a60, 0xe40b7b00, 0xefc00000) 5 5 Optional - Java Heap usage statistics
- 10. Thread Details "InvoiceThread-A996" prio=10 tid=0x00002b7cfc6fb000 nid=0x4479 runnable [0x00002b7d17ab8000] java.lang.Thread.State: RUNNABLE at com.buggycompany.rt.util.ItinerarySegmentProcessor.setConnectingFlight(ItinerarySegmentProcessor.java:380) at com.buggycompany.rt.util.ItinerarySegmentProcessor.processTripType0(ItinerarySegmentProcessor.java:366) at com.buggycompany.rt.util.ItinerarySegmentProcessor.processItineraryByTripType(ItinerarySegmentProcessor.java:254) at com.buggycompany.rt.util.ItinerarySegmentProcessor.templateMethod(ItinerarySegmentProcessor.java:399) at com.buggycompany.qc.gds.InvoiceGeneratedFacade.readTicketImage(InvoiceGeneratedFacade.java:252) at com.buggycompany.qc.gds.InvoiceGeneratedFacade.doOrchestrate(InvoiceGeneratedFacade.java:151) at com.buggycompany.framework.gdstask.BaseGDSFacade.orchestrate(BaseGDSFacade.java:32) at com.buggycompany.framework.gdstask.BaseGDSFacade.doWork(BaseGDSFacade.java:22) at com.buggycompany.framework.concurrent.BuggycompanyCallable.call(buggycompanyCallable.java:80) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) 1 2 3 4 5 6 7 1 Thread Name - InvoiceThread-A996 2 Priority - Can have values from 1 to 10 3 Thread Id - 0x00002b7cfc6fb000 - In Oracle/Sun's JDK implementation, this ID is simply a auto-incrementing long, starting from 1. The thread ID is unique during its lifetime. When a thread is terminated, this thread ID may be reused. It's platform independent. It's returned by calling the Thread.getId() method. 4 Native Id - 0x4479 - This ID is highly platform dependent. On Linux, it's the pid of the thread. On Windows, it's simply the OS-level thread ID within a process. On Mac OS X, it is said to be the native pthread_t value. 5 Address space - 0x00002b7d17ab8000 - 6 Thread State - RUNNABLE 7 Stack trace -
- 11. Agenda • How to take Thread Dumps? • Anatomy of Thread Dump • Troubleshooting – Real world problems through Patterns • Sudden CPU spikes • Out Of Memory Error • Unresponsive DB • Unresponsive NoSQL • Unresponsive Application
- 13. 2. Consecutive Full GCs Diagnose sudden CPU spikes 1. Infinite looping while (aCondition()) { doSomething(); }
- 14. Thread Mill Pattern Thread dump analysis pattern thread mill Read the full blog
- 15. Diagnose OutOfMemoryError: Java Heap Space public class MyObject { @Override public void finalize() { } } java.lang.ref.Finalizer#ReferenceQueue My Object Finalizer Thread java.lang.ref.Finalizer#ReferenceQueue MyObject Finalizer Thread public class MyObject { @Override public void finalize() { SomeObject.doSomething(); } } public class SomeObject { @Override public static synchronized void doSomething() { } }
- 16. Leprechaun Trap Pattern THREAD DUMP ANALYSIS PATTERN – LEPRECHAUN TRAP Read the full blog
- 17. Diagnose Unresponsive DB • RSI Pattern
- 18. Diagnose OutOfMemoryError: Unable to Create Native Threads • RSI – Repetitive Strain Injury THREAD DUMP ANALYSIS PATTERN – REPETITIVE STRAIN INJURY (RSI)
- 19. Diagnose unresponsive App • Traffic Jam Pattern THREAD DUMP ANALYSIS PATTERN – TRAFFIC JAM
- 20. More Patterns For more blogs on Thread Dump Patterns , please
- 21. On-site Training SPA – Scalability, Performance, Availability Consulting Services Offered ram@tier1app.com
Editor's Notes
- #11: public Thread(Runnable target, String name); public Thread(ThreadGroup group, String name); public Thread(ThreadGroup group, Runnable target, String name); public Thread(ThreadGroup group, Runnable target, String name, long stackSize); OR java.util.concurrent.ThreadFactory
- #15: C:\workspace\javaOne\dontDumpThreadDumps\threadDumps\demo-1-thread-mill\
- #18: https://www.youtube.com/watch?v=nnpHLxlj3fQ