Java profiling Do It Yourself

Java profiling
Do It Yourself
Alexey Ragozin
alexey.ragozin@gmail.com

JVM diagnostic interfaces
• JMX
• JVMTI – native API only
• Attach API
 Ad hoc instrumentation
 and more
• Perf counters
• Heap dump
• Flight recorder

MBeans: threading
 CPU usage per thread (user / sys)
 Memory allocation per thread
 Block / wait times
 Should be enabled
 Stack traces
Invaluable

SJK: ttop
2014-10-01T19:27:22.825+0400 Process summary
process cpu=101.80%
application cpu=100.50% (user=86.21% sys=14.29%)
other: cpu=1.30%
GC cpu=0.00% (young=0.00%, old=0.00%)
heap allocation rate 123mb/s
[000037] user=83.66% sys=14.02% alloc= 121mb/s - Proxy:ExtendTcpProxyService1:TcpAcceptor:TcpProcessor
[000075] user= 0.97% sys= 0.08% alloc= 411kb/s - RMI TCP Connection(35)-10.139.200.51
[000029] user= 0.61% sys=-0.00% alloc= 697kb/s - Invocation:Management
[000073] user= 0.49% sys=-0.01% alloc= 343kb/s - RMI TCP Connection(33)-10.128.46.114
[000023] user= 0.24% sys=-0.01% alloc= 10kb/s - PacketPublisher
[000022] user= 0.00% sys= 0.10% alloc= 11kb/s - PacketReceiver
[000026] user= 0.12% sys=-0.07% alloc= 2217b/s - Cluster|Member(Id=18, Timestamp=2014-10-01 15:58:3...
[000076] user= 0.00% sys= 0.04% alloc= 6657b/s - JMX server connection timeout 76
[000021] user= 0.00% sys= 0.03% alloc= 526b/s - PacketListener1P
[000034] user= 0.00% sys= 0.02% alloc= 1537b/s - Proxy:ExtendTcpProxyService1
[000049] user= 0.00% sys= 0.02% alloc= 6011b/s - JMX server connection timeout 49
[000032] user= 0.00% sys= 0.01% alloc= 0b/s - DistributedCache
https://github.com/aragozin/jvm-tools/blob/master/sjk-core/COMMANDS.md#ttop-command
Available via PerfCounters

MBeans: memory
• Memory geometry information
• Collection count
• Last collection details
 for each collector
• GC events available as notifications
since Java 7

SJK: GC
[GC: Copy#1806 time: 7ms interval: 332ms mem: Eden Space: 108032k-108032k->0k[max:191168k,rate:-325397.59kb/s] Tenured
Gen: 162185k+14k->162199k[max:477888k,rate:42.22kb/s] Survivor Space: 235k-13k->222k[max:23872k,rate:-41.93kb/s]]
Gen: 162418k+0k->162418k[max:477888k,rate:0.00kb/s] Survivor Space: 1k+0k->1k[max:23872k,rate:0.24kb/s]]
Total
Copy[ collections: 28 | avg: 0.0065 secs | total: 0.2 secs ]
MarkSweepCompact[ collections: 0 | avg: NaN secs | total: 0.0 secs ]
https://github.com/aragozin/jvm-tools/blob/master/sjk-core/COMMANDS.md#gc-command

MBeans: diagnostic commands
com.sun.management:type=DiagnosticCommand
com.sun.management:type=HotSpotDiagnostic
• Forcing GC / GC log rotation
• Head dump
• Flight recoder
• Changing --XX options
• etc
Java 8

JVM Attach API
• List JVM processes
• Attach to JVM by PID
• Send control commands
 heap dump / histogram
 stack dump
• Inspect system properties and VM options
• Launch instrumentation agents
https://github.com/gridkit/jvm-attach

SJK: hh --dead
Dead object histogram
 Similar to jmap –histo
 Invoke jmap –histo two time
 all heap objects
 live heap object
 calculates difference
 Can show top N rows
https://github.com/aragozin/jvm-tools/blob/master/sjk-core/COMMANDS.md#hh-command

SJK: hh --dead
1: 19117456 2038375696 [C
2: 9543865 441272568 [Ljava.lang.Object;
3: 13519356 432619392 java.util.HashMap$Entry
4: 12558262 301398288 java.lang.String
5: 7193066 287722640 org.hibernate.engine.spi.CollectionKey
6: 619253 160678888 [I
7: 4710497 113051928 org.jboss.seam.international.Messages$1$1
8: 571327 100876880 [Ljava.util.HashMap$Entry;
9: 1436183 57447320 org.hibernate.event.spi.FlushEntityEvent
10: 1661932 53181824 java.util.Stack
11: 209899 52047904 [B
12: 1624200 51974400 org.hibernate.engine.internal.Cascade
13: 929354 44608992 java.util.HashMap
14: 1812762 43506288 org.hibernate.i.u.c.IdentityMap$IdentityMapEntry
15: 850157 34006280 java.util.TreeMap$Entry
16: 1044636 25071264 java.util.ArrayList
17: 1340986 23423328 [Ljava.lang.Class;
18: 710973 22751136 java.io.ObjectStreamClass$WeakClassKey
19: 885164 21243936 org.hibernate.event.internal.WrapVisitor
20: 885126 21243024 org.hibernate.event.internal.FlushVisitor
...
Total 95197823 4793878008
https://github.com/aragozin/jvm-tools/blob/master/sjk-core/COMMANDS.md#hh-command

SJK: jps
JDK’s jps on steroid
 Uses attach API
 Lists VMs
 Filtering by JVM system properties
 Prints property values
 Prints effective –XX options
https://github.com/aragozin/jvm-tools/blob/master/sjk-core/COMMANDS.md#jps-command

SJK: jps
My favorite command
> sjk jps -pd PID MAIN duser.dir XMaxHeapSize
90543 sjk-0.3.1-SNAPSHOT.jar /var/vas_sdk_test_server -XX:MaxHeapSize=32126271488
5315 WrapperSimpleApp /var/vas_sdk_test_server/vas-sdk-test-13030 -XX:MaxHeapSize=4294967296
11094 WrapperSimpleApp /var/vas_sdk_test_server/vas-sdk-test-13020 -XX:MaxHeapSize=4294967296
993 Main /var/gedoms-uat/private/rtdb_1 -XX:MaxHeapSize=12884901888
56603 AxiomApplication /var/gedoms-uat/private/gedoms_1 -XX:MaxHeapSize=2147483648
24046 WrapperSimpleApp /var/sonar/sonar-3.6.2/bin/linux-x86-64 -XX:MaxHeapSize=536870912
https://github.com/aragozin/jvm-tools/blob/master/sjk-core/COMMANDS.md#jps-command

Perf counters
 Based on shared memory
 safe for target JVM
 Flat data model
 misc JVM counters
 true GC CPU usage data
 you can add own counter programmatically

Stack Trace Sampling
Capture
• Dump stack traces via local connection
• Store in highly compressed dump
10-30 bytes per trace
Analysis
• Frame frequency
• Conditional frame frequency
• Traces classification histogram
https://github.com/aragozin/jvm-tools/blob/master/sjk-core/COMMANDS.md#stcap-command

Stack Trace Sampling
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Base
other
DefaultServlet.doGet
LifecycleImpl.render
LifecycleImpl.execute
WorkItemController.doselect
Seam bean interceptor - lock contention
Seam bean interceptor - inject/disinject/outject
ResourceBundle - getObject
ResourceBundle - missing resource
Faceletscompile
Hibernate (rest)
Hibernate (autoFlush)
JDBC

Working with heap dumps
Java API to traverse heap dump object graph
Available at https://github.com/aragozin/jvm-tools/tree/master/hprof-heap
 Based on NetBeans profiler library
 No temporary files used
 Fixed generic method signatures
 Improved performance
Useful for
 In-place processing of large heap dumps
150 GiB is my personal record
 Write domain specific heap usage reports

HeapPath
 Convenient way to extract value from dump
 Error proof
 Handles String, primitives/boxed and arrays
myfield1.myfield2.myfield3
myarrayfield[0].myfield
myarrayfield[*].myfield
myarrayfield[*][*]
myfield1.*.myfield3
[*].value(MyClass)
myhashmap?entrySet[key=description].value

See also
https://github.com/vlsi/mat-calcite-plugin
Heap dump meets SQL

SJK Summary
Visit https://github.com/aragozin/jvm-tools
 Single executable JAR
 Command line interface
 Exploits JMX / Attach API / PerfCounters
 Sampling profiler included
 Extensible commands
 Write commands for your own application

Sigar
System Information Gatherer And Reporter
https://github.com/hyperic/sigar
• Cross platform
• Common system metrics
 CPU, Context switches, IO, etc
• Java bindings
 Self extracting JAR: org.gridkit.lab:sigar-lib:1.6.4

BTrace
Visit https://kenai.com/projects/btrace
Instrumentation profiling
 Inject code snippets written in Java
 CLI or Java API to use
 Extendible

BTrace
@OnMethod(clazz = "org.jboss.seam.Component",
method = "/(inject)/")
void entryByMethod2(@ProbeClassName String className, @ProbeMethodName String methodName,
@Self Object component) {
if (component != null) {
Field nameField = field(classOf(component), "name", true);
if (nameField != null) {
String name = (String)get(nameField, component);
Profiling.recordEntry(bench,
concat("org.jboss.seam.Component.", concat(methodName, concat(":", name))));
}
}
}
@OnMethod(clazz = "org.jboss.seam.Component",
method = "/(inject)/",
location = @Location(value = Kind.RETURN))
void exitByMthd2(@ProbeClassName String className, @ProbeMethodName String methodName,
@Self Object component, @Duration long duration) {
if (component != null) {
Field nameField = field(classOf(component), "name", true);
if (nameField != null) {
String name = (String)get(nameField, component);
Profiling.recordExit(bench,
concat("org.jboss.seam.Component.", concat(methodName, concat(":", name))), duration);
}
}
}

Flight Recorder
+ Accessible via JMX
+ Targeting JVM internals
+ Low overhead
‐ Non-compact file format
‐ Biased profiling
‐ Weak support for thread sampling

Self profiling benchmarks
Memory allocation regression tests
 assert for memory consumption using thread’s
allocation counter
Microbenchmarks
 MonitorGCeventstoexcluderesultaffectedbyGCpause
 Track CPU usage
 Thread sampling during benchmark

Performance tests
Nimble – framework for automated distributed performance testing
https://github.com/gridkit/nimble
https://code.google.com/p/gridkit/source/browse/grid-lab/trunk/examples/zk-benchmark-sample/
End – to – End automation
 Setup environment
 Run test scenarios
 Capture application KPI
 Capture metrics for OS and Java processes
 Integrated profiling with BTrace
 Single output data file

Profiling in production
In-house continuous query engine
 20+ applications
 Different environments, support teams, etc
 A lot of performance challenges
Built-in self profiling – currently in pilot
 Thread sampling
 CPU / allocation tracing
 Application specific diagnostics
 Fully encapsulated in application itself
Threading MBean performance
depredate for multithreaded access

Heap analyzer
In-house continuous query engine
 Relational graph up to few thousand nodes
 High memory consumption
Heap dump reporter automatically generates
 Memory consumption by operators
 Row count per operator
 Graph topology
 In Excel friendly format 

StackViewer
Visual tools
for thread dump
analysis
https://github.com/aragozin/stackviewer

Heap dump API + scripting
Coding in Java works well, but
 Reports are not interactive
 Slow trial and error turn around
Howaboutinteractive consoleforheapanalysis?
 bean shell / groovy ?
 how to do code completion in console?

Big Brother 4 J
We have already done it for performance testing
 Project detection
 Metrics capturing
 Integrated profiling sampling + instrumentation
 All metric in one place
How about ?
 Standalone tool for production
 Detecting specific Java processing
 Creating flight recording automatically
 In hyper dense file format

Thank you
Alexey Ragozin
alexey.ragozin@gmail.com
http://blog.ragozin.info
- my technical blog
http://github.com/aragozin
http://github.com/gridkit
- my open source projects

Java profiling Do It Yourself

More Related Content

What's hot (17)

Viewers also liked (16)

Similar to Java profiling Do It Yourself (20)

More from aragozin (20)

Recently uploaded (20)

Java profiling Do It Yourself