Troubleshooting Memory Problems in Java Applications

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Troubleshooting Memory
Problems
in Java Applications
Poonam Parhar
Consulting Member of Technical Staff
JVM Sustaining Engineer, Oracle

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.
2

About me
• Poonam Parhar
• JVM Sustaining Engineer at Oracle
• Co-author of the book ‘Java Performance Companion’
• JavaOne RockStar
• Speaker at Java Conferences
• Blog: https://blogs.oracle.com/poonam/
• Published articles
3

OutOfMemoryError
4
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Unknown Source)
at java.lang.String.<init>(Unknown Source)
at java.io.BufferedReader.readLine(Unknown Source)
at java.io.BufferedReader.readLine(Unknown Source)
at com.abc.ABCParser.dump(ABCParser.java:23)
at com.abc.ABCParser.mainABCParser.java:59)

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 5
• Misconfiguration of Memory Pools
• Excessive use of Finalizers
• Explicit GC Invocations
• Memory Leak
• Long GC Pauses
• Excessive GCs
• Application Failure with OutOfMemoryError
• Unexpected Memory Growth
• Poor Application Performance
Symptoms of
Caused by

Agenda
6
 Java Heap Memory Issues
 PermGen/Metaspace Memory Issues
 OutOfMemoryError due to Finalization
 CodeCache Issues
 Native Memory Issues

Java Heap Memory Issues
7

Confirm Memory Leak
• Monitor Java Heap usage over time
• If the Full GCs are not able to claim any space in the Old Gen of the heap
then it could be a configuration issue
• Heap might be sized too small
• Increase the heap size and test the application again
• If there is continuous memory growth and the failure persists at the
increased heap size too, there could be a memory leak
8

Appropriate Heap Size
-Xmx
11

Java Heap: Diagnostic Data
12

Java Heap: Diagnostic Data
• GC Logs
• Heap Dumps
• Heap Histograms
13

GC Logs
• Very helpful in determining the heap requirements
• Excessive GCs
• Long GC pauses
14

GC Logging Options
• Java 9:
 G1: -Xlog:gc*,gc+phases=debug:file=gc.log
 Non-G1: -Xlog:gc*:file=gc.log
• Prior Java Versions
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintGCDateStamps
-Xloggc:<gc log file>
15

2017-03-21T13:21:39.595+0000: 289634.716: [Full GC [PSYoungGen: 182784K-
>182660K(364544K)] [ParOldGen: 1091964K->1091964K(1092096K)] 1274748K-
>1274625K(1456640K) [PSPermGen: 493573K->493573K(494080K)], 1.2891230 secs]
[Times: user=6.01 sys=0.00, real=1.29 secs]
GC Logs: Heap Usage

4.381: [Full GC (Ergonomics) [PSYoungGen: 6656K->6656K(7680K)]
[ParOldGen: 16823K->16823K(17920K)] 23479K->23479K(25600K), [Metaspace: 2724K-
>2724K(1056768K)], 0.0720605 secs] [Times: user=0.23 sys=0.00, real=0.07 secs]
GC Logs: Excessive GCs

2017-04-11T14:49:31.875+0000: 322836.828: [Full GC (Allocation
Failure) 31G->24G(31G), 50.8369614 secs] [Eden: 0.0B(1632.0M)-
>0.0B(2048.0M) Survivors: 0.0B->0.0B Heap: 31.8G(31.9G)-
>24.5G(31.9G)], [Metaspace: 29930K->29564K(1077248K)] [Times:
user=83.83 sys=0.00,real=50.84 secs]
GC Logs: Long GC Pauses

Heap Dumps
• Most important diagnostic data for troubleshooting memory issues
• Can be collected using:
• jcmd <pid/main class> GC.heap_dump heapdump.dmp
• jmap -dump:format=b,file=snapshot.jmap <pid>
• JConsole utility, using Mbean HotSpotDiagnostic
• Java Mission Control, using HotSpotDiagnostic or DiagnosticCommand Mbeans
• -XX:+HeapDumpOnOutOfMemoryError
19

…<several Full GCs>…
[Full GC (Ergonomics) [PSYoungGen: 12799K->12799K(14848K)] [ParOldGen: 33957K->33957K(34304K)] 46757K-
>46757K(49152K), [Metaspace: 2723K->2723K(1056768K)], 0.1026523 secs] [Times: user=0.25 sys=0.00,
real=0.09 secs]
real=0.11 secs]
real=0.11 secs]
real=0.09 secs]
real=0.10 secs]
real=0.10 secs]
>46753K(49152K), [Metaspace: 2723K->2723K(1056768K)], 0.1001630 secs] [Times: user=0.39 sys=0
.00, real=0.09 secs][Full GC (Ergonomics) [PSYoungGen: 12799K->12799K(14848K)] [ParOldGen: 33954K-
>33954K(34304K)] 46754K->46754K(49152K), [Metaspace: 2723K->2723K(1056768K)], 0.0988169 secs] [Times:
user=0.39 sys=0.00, real=0.08 secs]
real=0.10 secs]
real=0.10 secs]
java.lang.OutOfMemoryError: GC overhead limit exceeded
Dumping heap to java_pid18904.hprof ...
• GC can continuously attempt to
free up room on the heap by
invoking frequent back-to-back Full
GCs
• Even when the gains of the efforts
are very little
• This impacts the performance of
the application and delays the re-
start of the process
• Delays the collection of heap dump
22
-XX:+HeapDumpOnOutOfMemoryError

-XX:GCTimeLimit and -XX:GCHeapFreeLimit
• GCTimeLimit sets an upper limit on the amount of time that GCs can spend in percent of
the total time
– Its default value is 98%
– Decreasing this value reduces the amount of time allowed that can be spent in the garbage collections
• GCHeapFreeLimit sets a lower limit on the amount of space that should be free after
the garbage collections, represented as percent of the maximum heap
– Its default value is 2%
– Increasing this value means that more heap space should get reclaimed by the GCs.
• An OutOfMemoryError is thrown after a Full GC if the previous 5 consecutive GCs (could
be minor or full) were not able to keep the GC cost below GCTimeLimit and were not
able to free up GCHeapFreeLimit space.
23

Heap Histograms
• Quick glimpse of objects in Java heap
• Can be collected using:
• -XX:+PrintClassHistogram, and SIGQUIT on Posix platforms and SIGBREAK on
Windows
• jcmd <process id/main class> GC.class_histogram filename=Myheaphistogram
• jmap -histo pid
• jhsdb jmap (in JDK 9)
• Java Mission Control (Diagnostic Commands)
24

Histogram

Java Heap: Analysis of Diagnostic Data
26

GC Logs Analysis
27

GC Logs Analysis
• What do we want to look for:
– Are there too many Full GCs?
– Are there GCs with long pauses?
– Are there GCs happening too frequently?
• Manual inspection
• Automatic Analysis tools
– Examples: GCHisto, GCViewer, gceasy.io etc.
28

Explicit Full GCs
164638.058: [Full GC (System) [PSYoungGen: 22789K->0K(992448K)] [PSOldGen:
1645508K->1666990K(2097152K)] 1668298K->1666990K(3089600K) [PSPermGen:
164914K->164914K(166720K)], 5.7499132 secs] [Times: user=5.69 sys=0.06, real=5.75
secs]
• -Dsun.rmi.dgc.server.gcInterval=n -Dsun.rmi.dgc.client.gcInterval=n
• -XX:+PrintClassHistogram and SIGBREAK /SIGQUIT signal
• Solution: -XX:+DisableExplicitGC
30

Heap Dump Analysis
31

Eclipse MAT - Memory Analyzer Tool
• Community developed tool for analyzing heap dumps
• Some of the amazing features that it offers are:
– Leak Suspects
– Histograms
– Unreachable objects
– Duplicate Classes
– Path to GC roots
– OQL
32

PermGen/MetaSpace Memory Issues
36

Confirm Memory Leak
• Monitor PermGen/Metaspace usage over time
• If the Full GCs are not able to claim any space in the PermGen/Metaspace
then it could be a configuration issue
• PermGen/Metaspace might be sized too small
• Increase the PermGen/Metaspace size and test the application again
• If there is continuous memory growth and the failure persists at the
increased PermGen/Metaspace size too, there could be a memory leak
37

Configure PermGen Size
–XX:PermSize=m –XX:MaxPermSize=n
38

Configure Metaspace Size
-XX:MetaspaceSize=m -XX:MaxMetaspaceSize=n
39

OutOfMemoryError: Compressed class space
• If UseCompressedClassPointers is enabled, then two separate areas of native memory
are used for class metadata
– By default UseCompressedClassPointers is ON if UseCompressedOops is ON
• 64-bit class pointers are represented by 32-bit offsets
• 32-bit offsets can be used to reference class-metadata stored in the ‘compressed class
space’
• By default, 1GB of address space is reserved for the compressed class space. This can be
configured using CompressedClassSpaceSize.
• MaxMetaspaceSize sets an upper limit on the total committed size of both of these
regions
• committed size of compressed class space +committed size of Metaspace<= MaxMetaspaceSize
40

_mark
_klass
<Object Fields>
_mark
_klass
<Object Fields>
Klass
Klass
Other
metadata
Other
metadata

_mark
_klass
<Object Fields>
_mark
_klass
<Object Fields>
Klass
Klass
Other
metadata
Other
metadata
+UseCompressedClassPointers

Metaspace used 2921K, capacity 4486K, committed 4864K, reserved 1056768K
class space used 288K, capacity 386K, committed 512K, reserved 1048576K
GC log with +UseCompressedClassPointers

PermGen/MetaSpace: Diagnostic Data
collection and Analysis
44

PermGen/MetaSpace: Diagnostic Data collection and
Analysis
• GC logs including options:
 –verbose:class
or
 -XX:+TraceClassLoading –XX:+TraceClassUnloading
• Data collected with:
• jmap –permstat (up to JDK 7)
• jmap –clstats (JDK 8 onwards)
• Heap Dumps
• JDK 8: class statistics information with ‘jcmd <pid> GC.class_stats’
45

Make sure that classes get unloaded
• Ensure –Xnoclassgc is not in use
• Ensure that –XX:+CMSClassUnloadingEnabled is used when using CMS on
Java 6 or 7
46

[Loading weblogic.i18n.logging.MessageDispatcher from
file:/opt/weblogic1213/wlserver/modules/features/weblogic.server.merged.jar]
[Loaded weblogic.i18n.logging.MessageDispatcher from
[Loaded weblogic.i18n.logging.CoreEnginePrimordialLoggerWrapper from
[Loading weblogic.logging.WLMessageLogger from
…
[Loading sun.reflect.GeneratedMethodAccessor486 from __JVM_DefineClass__]
[Loaded sun.reflect.GeneratedMethodAccessor486 from __JVM_DefineClass__]
…
[Unloading class sun.reflect.GeneratedMethodAccessor489 0x000000010128fc30]
[Unloading class sun.reflect.GeneratedMethodAccessor488 0x000000010128f830]
[Unloading class sun.reflect.GeneratedMethodAccessor482 0x000000010128e030]
[Unloading class sun.reflect.GeneratedMethodAccessor481 0x000000010128dc30]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor297 0x0000000101274c30]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor296 0x0000000101274830]
-verbose:class

74062.764: [Full GC (Last ditch collection) 74062.764: [CMS: 1444990K-
>1444951K(2599396K), 6.0356492 secs] 1444990K->1444951K(3046692K), [Metaspace:
4050878K->4050878K(6356992K)], 6.0366164 secs] [Times: user=6.04 sys=0.01, real=6.04
secs]
74068.804: [GC (CMS Initial Mark) [1 CMS-initial-mark: 1444951K(2599396K)]
1445248K(3046692K), 0.9821073 secs] [Times: user=1.23 sys=0.00, real=0.98 secs]
74069.787: [CMS-concurrent-mark-start]
74069.818: [Full GC (Metadata GC Threshold) 74069.818: [CMS74071.433: [CMS-concurrent-
mark: 1.642/1.647 secs] [Times: user=3.33 sys=0.00, real=1.65 secs] (concurrent mode
failure): 1444951K->1444879K(2599396K), 7.5785637 secs] 1445513K->1444879K(3046692K),
[Metaspace: 4050878K->4050878K(6356992K)], 7.5795618 secs] [Times: user=9.19 sys=0.00,
real=7.58 secs]
java.lang.OutOfMemoryError: Compressed class space
GC Logs

$ jmap -permstat 29620
Attaching to process ID 29620, please wait...
Debugger attached successfully. Client compiler detected.
JVM version is 24.85-b06
12674 intern Strings occupying 1082616 bytes. finding class loader instances ..
done. computing per loader stat ..done. please wait.. computing liveness.........................................done.
class_loader classes bytes parent_loader alive? type
<bootstrap> 1846 5321080 null live <internal>
0xd0bf3828 0 0 null live sun/misc/Launcher$ExtClassLoader@0xd8c98c78
0xd0d2f370 1 904 null dead sun/reflect/DelegatingClassLoader@0xd8c22f50
0xd0c99280 1 1440 null dead sun/reflect/DelegatingClassLoader@0xd8c22f50
0xd0b71d90 0 0 0xd0b5b9c0 live java/util/ResourceBundle$RBClassLoader@0xd8d042e8
0xd0d2f4c0 1 904 null dead sun/reflect/DelegatingClassLoader@0xd8c22f50
0xd0b5bf98 1 920 0xd0b5bf38 dead sun/reflect/DelegatingClassLoader@0xd8c22f50
0xd0b5bf38 6 11832 0xd0b5b9c0 dead sun/reflect/misc/MethodUtil@0xd8e8e560
0xd0d2f3a8 1 904 null dead sun/reflect/DelegatingClassLoader@0xd8c22f50
0xd0b5b9c0 317 1397448 0xd0bf3828 live sun/misc/Launcher$AppClassLoader@0xd8cb83d8
0xd0d2f3e0 1 904 null dead sun/reflect/DelegatingClassLoader@0xd8c22f50
0xd0ec3968 1 1440 null dead sun/reflect/DelegatingClassLoader@0xd8c22f50
0xd0e0a248 1 904 null dead sun/reflect/DelegatingClassLoader@0xd8c22f50
0xd0d2f4f8 1 904 null dead sun/reflect/DelegatingClassLoader@0xd8c22f50
0xd0e0a280 1 904 null dead sun/reflect/DelegatingClassLoader@0xd8c22f50
total = 22 2186 6746816 N/A alive=4, dead=18 N/A
jmap -permstat

jmap -clstats 26240
Attaching to process ID 26240, please wait...
Debugger attached successfully. Server compiler detected. JVM version is 25.66-b00
finding class loader instances ..done. computing per loader stat ..done. please wait..
computing liveness.liveness analysis may be inaccurate ...
class_loader classes bytes parent_loader alive? type
<bootstrap> 513 950353 null live <internal>
0x0000000084e066d0 8 24416 0x0000000084e06740 live
sun/misc/Launcher$AppClassLoader@0x0000000016bef6a0
0x0000000084e06740 0 0 null live
sun/misc/Launcher$ExtClassLoader@0x0000000016befa48
0x0000000084ea18f0 0 0 0x0000000084e066d0 dead
java/util/ResourceBundle$RBClassLoader@0x0000000016c33930
jmap -clstats

Heap Dumps
• Heap Dumps help here too
• Look for classes that should have been unloaded
• Eclipse MAT offers a very nice feature called ‘Duplicate Classes’
– Displays classes that were loaded multiple times by different classloader instances
– If duplicate classes keep growing over time/redeployments, it’s a red flag
51

OutOfMemoryError due to Finalization
53

Finalization
• OutOfMemoryError can also be caused due to excessive use of finalizers
• Objects with a finalizer (i.e. a finalize() method) may delay the reclamation
of the space occupied by them
• Finalizer thread needs to invoke the finalize() method of the instances
before those instances can be reclaimed
• If the Finalizer thread does not keep up with the rate at which the objects
become eligible for finalization, JVM might fail with an OutOfMemoryError
• Deprecated in Java 9
• https://bugs.openjdk.java.net/browse/JDK-8165641
54

Finalization: Diagnostic Data and Tools
55

Finalization: Diagnostic Data and Tools
• JConsole
• jmap -finalizerinfo
• Heap Dumps
56

CodeCache is full. Compiler has been
disabled
59

CodeCache is full. Compiler has been disabled
• CodeCache is the memory pool to store the compiled code generated by
the JIT compilers
• There is no specific OutOfMemoryError thrown when CodeCache is full
• Managed by Sweeper
• An emergency CodeCache cleanup is done when it becomes full
– This may discard the compiled code that might be needed shortly after
– Compiler needs to work again to generate the compiled code for those methods
• Ensure CodeCache size is sufficient
– Increase CodeCache maximum size with ReservedCodeCacheSize
60

OutOfMemoryError: Native Memory
61

# A fatal error has been detected by the Java
Runtime Environment:
#
# java.lang.OutOfMemoryError : unable to create
new native Thread
# A fatal error has been detected by the Java
Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 32756
bytes for ChunkPool::allocate. Out of swap
space?
#
# Internal Error (allocation.cpp:166),
pid=2290, tid=27 # Error: ChunkPool::allocate
62
Native OutOfMemoryError

Native OutOfMemoryError
• JVM is not able to allocate from native memory
– Not managed by the JVM
• This process or other processes on the system are eating up the native
memory
• Can make more room for native heap by:
– Reducing the Java Heap, PermGen/Metaspace, number of threads and/or their stack
sizes etc.
– Reducing the number of processes running on the system
• If the above don’t help, we might be facing a native heap memory leak
– Example: JNI code allocating native buffers
63

Native Memory: Diagnostic Data
64

Native Memory: Diagnostic Data
• Native memory leaks in the JVM
• Native Memory Tracker output
• Native Memory Leaks outside JVM
• Process map output with tools like pmap
• Results from native memory leak tools such as libumem, valgrind etc.
• Core file
65

Native Memory Tracker
• The Native Memory Tracker (NMT) can be used to track native memory
that is used internally by the JVM
• It cannot track memory allocated outside the JVM or by native libraries
66

NMT
• Start the process with NMT enabled using NativeMemoryTracking
• The output level can be set to a ‘summary’ or ‘detail’ level:
• -XX:NativeMemoryTracking=summary
• -XX:NativeMemoryTracking=detail
• Use jcmd to get the native memory usage details:
• jcmd <pid> VM.native_memory
67

jcmd 90172 VM.native_memory 90172:
Native Memory Tracking:
Total: reserved=3431296KB, committed=2132244KB
- Java Heap (reserved=2017280KB, committed=2017280KB)
(mmap: reserved=2017280KB, committed=2017280KB)
- Class (reserved=1062088KB, committed=10184KB)
(classes #411)
(malloc=5320KB #190)
- Thread (reserved=15423KB, committed=15423KB)
(thread #16)
(stack: reserved=15360KB, committed=15360KB)
(malloc=45KB #81)
(arena=18KB #30)
- Code (reserved=249658KB, committed=2594KB)
(malloc=58KB #348)
- GC (reserved=79628KB, committed=79544KB)
(malloc=5772KB #118)
- Compiler (reserved=138KB, committed=138KB)
(malloc=8KB #41)
(arena=131KB #3)
- Internal (reserved=5380KB, committed=5380KB)
(malloc=5316KB #1357)
- Symbol (reserved=1367KB, committed=1367KB)
(malloc=911KB #112)
(arena=456KB #1)
- Native Memory Tracking (reserved=118KB, committed=118KB)
(malloc=66KB #1040)
(tracking overhead=52KB)
- Arena Chunk (reserved=217KB, committed=217KB)
(malloc=217KB)

Native Memory Leaks Outside JVM
• For the native memory leaks stemming from outside the JVM, we need to
rely on the native memory leak tools for their detection and
troubleshooting
• Native Memory Leak Detection Tools
– dbx
– libumem
– valgrind
– purify
– and so on
69

Summary
• Causes of memory related problems:
– Misconfiguration of Memory Pools
– Excessive use of Finalizers
– Explicit GC Invocations
– Memory Leaks
• Size memory pools appropriately
• Tools
• HeapDumpOnOutOfMemoryError and PrintClassHistogram JVM Options
• JConsole
• jcmd
• jmap
• GC Logs
• Eclipse MAT
• NMT
• Native Memory Leak Detection Tools such as dbx, libumem, valgrind, purify etc.
70

References
• Troubleshooting Guide:
– https://docs.oracle.com/javase/9/troubleshoot/toc.htm
– https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/index.html
• Blog:
– https://blogs.oracle.com/poonam/
• Free Online Course: Java Virtual Machine Troubleshooting
– http://www.oracle.com/goto/jvm
• Article: https://www.infoq.com/articles/Troubleshooting-Java-Memory-
Issues
71

Stay connected
• Join us: DevOps Corner (Developer Lounge – Moscone West)
• Learn more: openjdk.java.net | wercker.com/java
• Follow: @OpenJDK, @wercker #JavaOne #DevOps
72

Troubleshooting Memory Problems in Java Applications

Troubleshooting Memory Problems in Java Applications

More Related Content

What's hot (20)

Similar to Troubleshooting Memory Problems in Java Applications (20)

Recently uploaded (20)

Troubleshooting Memory Problems in Java Applications