How to Troubleshoot 9 Types of OutOfMemoryError Session
- 1. 1
- 2. Does Java process memory utilization go beyond –Xmx? 2 –XX:MaxMetaspaceSize -Xmx Young Old Metaspac e Threads JNI misc GC Direc t Buff Code Cach e Heap Memory Native Memory Java Process Memory -XX:MaxDirectMemorySize
- 4. 4 How to know the type of OutOfMemoryError? java.lang.OutOfMemoryError: <type>
- 5. 5 Java Heap Space 1 Metaspac e Threads JNI misc GC Direc t Buff Code Cach e Heap Memory Native Memory JVM Memory Most common type -Xmx
- 6. 6 GC Behavior of a Healthy Application - Full Garbage Collection Event
- 7. 7 GC Behavior of Acute Memory Leak - Full Garbage Collection Event
- 8. 8 GC Behavior when there is a Memory Leak - Full Garbage Collection Event
- 9. 9 How to study GC Behavior? -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:<file-path> Till Java 8 -Xlog:gc*:file=<file-path> From Java 9 Enable GC logs (always) Almost zero overhead
- 10. 10 GCeasy GC Log Analysis Demo
- 11. 11 How to capture Heap Dumps? 8 options: https://blog.heaphero.io/2017/10/13/how-to-capture-java-heap-dumps-7-options/
- 12. 1. GC Log 10. netstat 12. vmstat 2. Thread Dump 9. dmesg 3. Heap Dump 6. ps 8. Disk Usage 5. top 13. iostat 11. ping 14. Kernel Params 15. App Logs 16. metadata 4. Heap Substitute 7. top -H 12 Open-source script: https://github.com/ycrash/yc-data-script 360° Troubleshooting artifacts ./yc –p <PROCESS_ID>
- 13. 13 HeapHero How to analyze Heap Dump?
- 14. 14 OutOfMemoryError: Java Heap Space Causes 1. Memory leak due to buggy code 2. Increase in Traffic Volume Solutions 1. Fix Memory Leak in the code 2. Increase heap size (-Xmx) Artifacts 1. Garbage Collection Log 2. Heap Dump Tools 1. GCeasy (GC log analysis) 2. HeapHero 3. Eclipse MAT 4. JVisualVM
- 15. 15 GC overhead limit exceeded 2 Metaspac e Threads JNI misc GC Direc t Buff Code Cach e Heap Memory Native Memory JVM Memory -Xmx
- 16. 16 OutOfMemoryError: GC overhead limit exceeded Diagnosis: Same as ‘Java Heap Space’ Java process is spending more than 98% of its time doing garbage collection and recovering less than 2% of the heap and has been doing so far the last 5 consecutive garbage collections
- 17. 17 Requested array size exceeds VM limit 3 Young Old Metaspac e Threads JNI misc GC Direc t Buff Code Cach e Heap Memory Native Memory JVM Memory
- 18. 18 Demo OutOfMemoryError: Requested array size exceeds VM limit
- 19. 19 OutOfMemoryError: Requested array size exceeds VM limit Causes 1. Allocated array larger than heap size Solutions 1. Fix code which allocates large array size 2. Increase heap size (-Xmx) Artifacts 1. Application Log or std error Tools
- 20. 20 Metaspace 4 Young Old Threads JNI misc GC Direc t Buff Code Cach e Heap Memory Native Memory JVM Memory
- 21. 21 Metaspace Memory Leak - Full Garbage Collection Event
- 23. 23 How to study Metaspace Behavior? java {app_name} -verbose:class Till Java 8 java {app_name} - Xlog:class+load=info:<filename> From Java 9
- 24. 24 OutOfMemoryError: Metaspace Causes 1. Creating large number of Dynamic classes (Java Reflection, Groovy type of scripting languages) 2. Loading large number of Classes (3rd party libraries/frameworks) 3. Loading large number of Classloaders Solutions 1. Fix Memory Leak code 2. Increase -XX:MetaspaceSize and -XX:MaxMetaspaceSize. Artifacts 1. Java 8 & below: java {app_name} -verbose:class 2. Java 9 & above: java {app_name} - Xlog:class+load=info:<filena me> 3. jcmd {pid} GC.class_histogram 4. Heap Dump Tools 1. GCeasy (GC log analysis) 2. HeapHero 3. Eclipse MAT
- 25. 25 Permgen space 5 Young Old Threads JNI misc GC Direc t Buff Code Cach e Heap Memory Native Memory JVM Memory Happens only in Java 7 & below
- 26. 26 OutOfMemoryError: Permgen Space Diagnosis: Same as ‘Metaspace’
- 27. 27 Unable to create new native threads 6 Young Old Metaspac e JNI misc GC Direc t Buff Code Cach e Heap Memory Native Memory JVM Memory
- 28. 28 How to capture Thread Dump? 9 options:https://blog.fastthread.io/how-to-take-thread-dumps-7-options/
- 29. 1. GC Log 10. netstat 12. vmstat 2. Thread Dump 9. dmesg 3. Heap Dump 6. ps 8. Disk Usage 5. top 13. iostat 11. ping 14. Kernel Params 15. App Logs 16. metadata 4. Heap Substitute 7. top -H 29 Open-source script: https://github.com/ycrash/yc-data-script 360° Troubleshooting artifacts
- 30. 30 Real Case Study Slowdown in a Major Financial Institution’s Middleware in USA http://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTcvMDMvMTQvLS10aHJlYWREdW1wLTIudHh0LS0xMi0yOC0zMw==&s=t
- 31. 31 OutOfMemoryError: Unable to create new native threads Causes 1. Threads are leaking Solutions 1. Fix thread leak 2. Increase the Thread Limits Set at Operating System(ulimit –u) 3. Kill other processes 4. Increase RAM capacity 5. Reduce Java Heap Size 6. Reduce thread stack size (-Xss). Note: can cause Artifacts 1. Thread Dump Tools 1. FastThread 2. Text Editor
- 32. 32 Direct buffer memory 7 Young Old Metaspac e Threads JNI misc GC Code Cach e Heap Memory Native Memory JVM Memory Becoming Pervasive in modern Spring Boot Frameworks
- 33. 33 Demo OutOfMemoryError: Direct buffer Memory
- 34. 34 OutOfMemoryError: Direct buffer memory Causes 1. Increase(or Leak) in direct buffers usage a. java.nio package b. Moving from Spring RestTemplate to WebClient c. Image Processing Libraries d. Networking Libraries e. Some JDBC Drivers Solutions 1. Fix Memory Leak code 2. Increase - XX:MaxDirectMemorySize 3. Upgrade to Java 17 (There some issues in Java 11) Artifacts 1. App Log or Std error 2. Native Memory Tracking Tools
- 35. 35 Kill process or sacrifice child 8 JVM Kernel X
- 36. 1. GC Log 10. netstat 12. vmstat 2. Thread Dump 9. dmesg 3. Heap Dump 6. ps 8. Disk Usage 5. top 13. iostat 11. ping 14. Kernel Params 15. App Logs 16. metadata 4. Heap Substitute 7. top -H 36 Open-source script: https://github.com/ycrash/yc-data-script 360° Troubleshooting artifacts
- 37. 37
- 38. 38 Real Case Study Intermittent HTTP 502 errors in AWS EBS Service
- 41. 41
- 42. 42 OutOfMemoryError: Kill process or sacrifice child Causes 1. Lack of RAM capacity 2. High Memory utilization by other processes in the environment 3. Memory Leak in the code Solutions 1. Kill other processes 2. Increase RAM capacity 3. Fix Memory Leak in the code Artifacts 1. dmesg 2. GC Log 3. Heap Dump 4. Native Memory Tracking Tools 1. HeapHero 2. Eclipse MAT 3. yCrash
- 43. 43 Reason stack_trace_with_native_method 9 Young Old Metaspac e Threads misc GC Direc t Buff Code Cach e Heap Memory Native Memory JVM Memory
- 44. 44 OutOfMemoryError: Reason stack_trace_with_native_method Don’t worry, if you are not using JNI
- 45. 45 OutOfMemoryError: Reason stack_trace_with_native_method Causes 1. Heavy Usage of Native Methods 2. Recursive Native Method Calls Solutions 1. Fix the problem in the native app Tools 1. OS native tools a. Dtrace b. pmap c. pstack Artifacts 1. Application Log or std error
- 46. 46 You need data GC Log Heap Dump 1. Java Heap Space 2. GC overhead limit exceeded 3. Requested array size exceeds VM limit 7. Direct buffer Memory App Logs Verbose Class Log 4. Metaspace 5. PermGen Space Thread Dump 6. Unable to create native threads dmesg 8. Kill process or sacrifice child GC Log
- 47. 1. GC Log 10. netstat 12. vmstat 2. Thread Dump 9. dmesg 3. Heap Dump 6. ps 8. Disk Usage 5. top 13. iostat 11. ping 14. Kernel Params 15. App Logs 16. metadata 4. Heap Substitute 7. top -H 47 Open-source script: https://github.com/ycrash/yc-data-script You need 360° data
- 48. Ram Lakshmanan ram@tier1app.com @tier1app https://www.linkedin.com/company/ycrash This deck will be published in: https://blog.heaphero.io If you want to learn more … 48 THANK YOU FRIENDS
Editor's Notes
- #3: Eclipse: MemoryLeakDemo, LaunchPad_oom
- #10: Gceasy - C:\workspace\ycrash\speeches\9-types-oom\gc-memory-leak.zip
- #11: https://blog.heaphero.io/2017/10/13/how-to-capture-java-heap-dumps-7-options/
- #13: http://localhost:8080/heap-index.jsp C:\workspace\tmp\dev-nexus\dump-2024-04-09.hprof http://localhost:8080/heap-report-wc.jsp?p=ZDI0ZHIxRGJtaUhHQXN3eUFWa3htT2ZQVmt2NHFiUWM1WWR3TWRLamhTWTV5ME9oVGNMVlJmZkdaR2FBRnk2OEg0UFgyTFg2N2VTOUdPRmlCUHJEYUtrOWFRMEZCSkV4VEIxYjRsemc2V2g2dFFSeVBLVmFOYW1zYTNwNUdoZzY= http://localhost:8080/heap-report-wc.jsp?p=ZDI0ZHIxRGJtaUhHQXN3eUFWa3htQW9QeERicTZpZjNUczlXTFVVTE9jcWpMeHBsRkpSZEV4Vkk5cDZqZHJIK2NzditQeWx0UDZDbUhmNEIySTBZMkZ6NUhjMkh3SFdOSUdPZU5WTHZHZlk2dEdnQnNCMmp2MkltMEdoUnFxcUY=
- #17: OOMRequestedArraySizeExceedsVMLimit
- #18: OOMRequestedArraySizeExceedsVMLimit
- #21: https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjIvMDcvMTYvbWV0YXNwYWNlLW9vbWVycm9yLmdjLS0xOC0xNy0zMg==&s=t&channel=WEB
- #22: OOMMetaspace
- #24: OOMMetaspace_verbose
- #28: https://blog.fastthread.io/how-to-take-thread-dumps-7-options/
- #30: C:\workspace\ycrash\speeches\9-types-oom\thread-leak.txt Backup 1: http://localhost:8080/my-thread-report.jsp?p=Qzpcd29ya3NwYWNlXHRtcFx5Yy0yXzEyLWJlcmxpblxzaGFyZWRcMjAyNC01LTI1XHRocmVhZC1sZWFrLnR4dC0xNy0xMy01Mjs7& Backup 2: http://localhost:8080/yc-report.jsp?ou=SAP&de=127.0.0.1&app=yc&ts=2024-04-09T11-09-51
- #33: OOMDirectBuffer
- #37: https://test.ycrash.io/yc-report-kernel.jsp?ou=czlWbG0rUko0UXAxazlSbjZrSUIwUT09&de=172.31.7.106&app=yc&ts=2023-09-01T10-25-39
- #41: https://test.ycrash.io/yc-report-kernel.jsp?ou=czlWbG0rUko0UXAxazlSbjZrSUIwUT09&de=172.31.7.106&app=yc&ts=2023-09-01T10-25-39