ContextualContinuous Profilng

Contextual Continuous Profiling
Getting deeper insights into your application
Jaroslav Bachorik
Staff Software Engineer
Datadog

1. Introduction
2. Continuous profiling
3. Contextual continuous profiling
4. Takeaways
5. Q&A
Agenda

Introduction
- Active in JVM performance area since 2006
- NetBeans Profiler
- VisualVM [1]
- BTrace [2]
- JMX, Serviceability
- OpenJDK member and Reviewer
- Currently at Datadog in charge of JVM profiling
- In-house profiling agent
- Heavily based on async-profiler [3]
- Participating in OpenJDK
- JFR fixes, backports
- Proposing/implementing new features
Disclaimer
- Examples will be shown in Datadog UI
- Yet, not a Datadog pitch!
[1] https://visualvm.github.io
[2] https://github.com/btraceio/btrace
[3] https://github.com/async-profiler/async-profiler

Continuous Profiling
- Single execution profiling
- Traditional profilers - JProfiler, VisualVM, YourKit etc.
- Used in development phase
- Overhead not a big concern
- Results restricted by the development environment
- Continuous profiling
- Cloud deployments, Continuous delivery etc.
- Profiling in development environment not sufficient
- Profiler is ‘always on’
- Past performance can be inspected and analyzed
- Very overhead sensitive

JDK Flight Recorder
- Available in JDK 9+ and JDK 8 after update 272
- Capture profiling data on demand
- Jcmd
- JMX
- Streaming available since JDK 14 [1]
[1] https://openjdk.org/jeps/349
[2] https://docs.oracle.com/en/java/javase/11/tools/java.html
[3]
https://docs.oracle.com/javacomponents/jmc-5-5/jfr-command-referenc
e/diagnostic-command-reference.htm
> jcmd myapp.jar JFR.dump name=rec1 filename=dump1.jfr
…
> jcmd myapp.jar JFR.dump name=rec1 filename=dump2.jfr
Start JFR at JVM startup
> java -XX:StartFlightRecording=name=rec1,filename=my_recording.jfr -jar myapp.jar
More JFR options are described in Java Tools Reference [2]
Capture profiling data via JCMD
More JCMD related options are described in JCMD Tool Reference [3]

JVMTI Agent
- AsyncGetCallTrace
- ‘Unofficial’ API to get non-biased stacktraces
- Not really maintained
- Lurking bugs can crash your JVM
- async-profiler [1]
- Widely used fully functional profiler
- Exports to multiple formats
- CSV
- Flamegraph [2]
- JFR binary format
[2] https://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html

Datadog Continuous Profiler
[1] https://openjdk.org/jeps/349
[3] https://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html
- Agent, Backend, UI
- Backend and UI are proprietary, closed source
- Agent is open source
- https://github.com/dataDog/dd-trace-java
- Opportunistic Agent
- Use any available datasource
- Combines JFR and in-house profiling agent
- Subject to availability
- AsyncGetCallTrace can be crashy on older JVMs
- JFR is not available from all Java vendors
- J9, Zing
- Integrated Agent
- Distributed as a part of the tracer agent
- Integrates with the tracer
- ! Important for context !

Introducing Context
- ‘Context’ is:
- A set of simple values describing current workload
- Can be thought of as tags
- User specific meaning
- ‘Context’ allows:
- Mapping performance data back to
- HTTP requests
- REST API calls
- GRPC calls
- etc.
- Slice’n’Dice analysis of the performance data
- ‘Context’ is difficult because it must be:
- Of acceptable cardinality
- Fully propagated between threads
- Executors
- Fork-Join
- Reactive frameworks
- Loom!

Implementing Context
- Labels in PPROF
- Ready to use
- Profile size implications
- Go runtime has native support
- Nothing in JVM
- ‘Thread-coloring’ approach was considered in JRockit
- Never implemented
- JFR is still not aware of context
- Custom implementation is needed

Datadog Profiling Context
- Context propagation
- Implemented in Java tracer
- Context associated with a unit of work
- Independent of executing thread
- Context persistence
- Implemented in the profiler agent
- Store context in JFR events
- Easy and fast Java<->Native interop is mandatory
- No JNI calls, please!
- Shared memory buffer
- Relying on Java and native side being tightly coupled
- Semi-custom context
- Capped at ten custom tags
- Custom tag types/names
- Must be defined before profiler is started
- Stored in the JFR recording

Shared Memory Context
- One context per thread
- Sparse thread-page map
- Static size
- Efficient memory layout
- 64 bytes to match the common x64 cache line size
- Checksum
- Used to detect tearing, partial writer
- 64 bit/8 bytes
- Context Content
- Provides 10 slots (currently)
- Each slot is 4 bytes
- Possibly up to 14 slots (56 bytes)

Shared Memory Context
Thread 1
Thread 2
…
Thread N
1 2 3 4 5 6 7 8 9 10
chksum
64b
Context data (10 slots, 40 bytes
64 bytes (eg. cache line)
1 2 3 4 5 6 7 8 9 10
chksum
64b
Context data (10 slots, 40 bytes)
64 bytes (eg. cache line)
Thread
page
map

JFR Event Context
- Contextual JFR events
- Used context slots as event attributes
- Event scheme generated at startup
- Store context slot names/ids
- Use Settings event
- Dictionarized context contents
- Strings mapped to unique IDs
- Context content is the ID
- Strings stored as JFR constant pool
- Standard JFR binary format feature
- Custom context is fully restorable
- Context slot name
- Context slot value

Java API
- ContextSetter
- Register context before profiling is started
- Count
- Names
- Set context values
- Register dictionarized strings

Context Propagation
- Context is bound to a work item
- Work item can be processed by multiple threads
- Manual threads
- Thread pools
- Reactive frameworks
- Context must be carried from thread to thread
- Concept of activate/deactivate
- Piggy-back on distributed tracing
- Datadog Tracer already does propagation
- Context propagation ‘for free’
- Profiling context needs more detailed propagation
- Tracer needs to be aware of profiler needs

Wrap-up
- Benefits of using profiling context
- Captures additional information about the environment during
profiling
- Enables slicing and dicing of profiling data to identify
performance issues
- Associates profiling data with specific users or inputs for
informed decisions
- Provides connection between parallel code execution to identify
interactions
- Overall, helps developers optimize code more effectively
- Next steps
- Bring the benefits to JDK/JFR
- Many built-in events would benefit from this
- Locks, I/O, etc.
- Standardized implementation

ContextualContinuous Profilng

More Related Content

Similar to ContextualContinuous Profilng (20)

Recently uploaded (20)

ContextualContinuous Profilng