Understand the Trade-offs Using Compilers for Java Applications

Understand the trade-offs using
compilers for Java applications
(From AOT to JIT and Beyond!)
Mark Stoodley
Eclipse OpenJ9 & OMR project co-lead
Senior Software Developer @ IBM Canada
mstoodle@ca.ibm.com
@mstoodle

InfoQ.com: News & Community Site
• Over 1,000,000 software developers, architects and CTOs read the site world-
wide every month
• 250,000 senior developers subscribe to our weekly newsletter
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• 2 dedicated podcast channels: The InfoQ Podcast, with a focus on
Architecture and The Engineering Culture Podcast, with a focus on building
• 96 deep dives on innovative topics packed as downloadable emags and
minibooks
• Over 40 new content items per week
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
tradeoffs-java-compilers/

Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

Java ecosystem has a rich history exploring native code compilation!
• JIT
• 1999: Hotspot JVM ( https://en.wikipedia.org/wiki/HotSpot ) released by Sun Microsystems
• 1999: IBM SDK for Java included productized JIT compiler originally built by IBM Tokyo Research Lab, used until Java 5.0
• 2000: jRockit released by Appeal Virtual Machines (https://en.wikipedia.org/wiki/Appeal_Virtual_Machines)
• 2006: IBM SDK for Java 5.0 includes J9 JVM with “Testarossa” JIT, now open source as Eclipse OpenJ9
• 2017: Azul released Falcon JIT based on LLVM
• 2018: Graal compiler available as experimental high opt compiler in Java 10
• AOT
• 1997: IBM High Performance Compiler for Java (https://link.springer.com/chapter/10.1007/978-1-4615-4873-7_60)
• Statically compiled Java primarily for scientific/high performance computing on mainframes
• 1998: GNU Compiler for Java (gcj) (https://en.wikipedia.org/wiki/GNU_Compiler_for_Java)
• Statically compile Java used GCC compiler project
• 2000: Excelsior JET (https://en.wikipedia.org/wiki/Excelsior_JET)
• Commercial AOT compiler
• 2017: Experimental jaotc compiler available in OpenJDK9 uses Graal compiler
• 2018: GraalVM project introduces native images supporting a subset of Java on SubstrateVM
• “Caching” JIT code
• 2003: jRockit JIT introduces experimental support for cached (but not optimized) code generation
• https://docs.oracle.com/cd/E13188_01/jrockit/docs142/userguide/codecach.html
• 2007: IBM ”dynamic AOT” production support introduced in IBM SDK for Java 6
• 2019: Azul Zing introduces “code stashing” as part of ReadyNow
2

Native compilers in today’s Java ecosystem
• Hotspot JITS
• C1 “client” and C2 “server” JIT compilers
• Default a.k.a. reference native compilers used in OpenJDK
• Eclipse OpenJ9’s JIT
• JIT compiler with multiple adaptive optimization levels (cold through scorching)
• Historically offered Java compliant AOT compilation for embedded and real-time systems
• Today caches JIT compilations (a.k.a “dynamic AOT”) alongside classes in shared classes cache
• Azul Zing’s Falcon JIT based on LLVM
• Alternative “high opt” compiler to C2
• Can stash JIT compilations to disk and reload in subsequent runs
• Oracle Graal compiler
• Written in Java
• Since Java 9: experimental AOT compiler jaotc
• Since Java 10: experimental alternative to C2 JIT compiler
• Create native images using SubstrateVM (under “closed world” assumption and other limitations) 3

Outline
• Let’s compare:
• JIT
• AOT
• Caching JIT code (== both AOT and JIT!)
• Taking JITs to the cloud
• Wrap Up
4

JIT = Just In Time
• JITs compile code at same time program runs
• Adapt to whatever the program does “this time”
• Adapt even to the platform the program is running on
• After more than two decades of sustained effort:
• JIT is the leader for Java application performance
• Despite multiple significant parallel efforts aimed at AOT performance
• Why is that? At least 2 reasons you may already know…
5

1. JITs speculate on class hierarchy
• Calls are virtual by specification
• But many calls only have a single target (monomorphic) in a particular program run
• JITs speculate that this one target will continue to be the only target
• Optimize aggressively and keep going deeper (calls to calls to calls….)
• Speculation can greatly expand ability to inline call targets
• Which expands optimization scope
• Compiling too early, though, can fool compiler to speculate wrongly
6

2. JITs use profile data collected as program runs
• Not all code paths execute as frequently
• Profile data tells compiler which paths are worth optimizing
• Not all calls have a single possible target
• Profile data can prioritize to enable method inlining most profitable target(s)
• Efficient substitute for some kinds of larger scope compiler analyses
• Takes too long to analyze entire scope but low overhead profile data still identifies constants
• Contributes to practical compile time
• BUT accumulating good profile data takes time
• JIT compilers work very well if the profile data is high quality
7

But JIT performance advantage isn’t free
• Collecting profile data is an overhead
• Cost usually paid while code is interpreted : slows start-up and ramp-up
• Quality data means profiling for a while: slows ramp-up
• JIT compilers consume transient resources (CPU cycles and memory)
• From under a millisecond to seconds of compile time, can allocate 100s MBs
• Cost paid when compiling : slows start-up and ramp-up
• Takes time to get to “full speed” because there may be 1000s of methods to compile
• Also some persistent resource consumption (memory)
• Profile data, class hierarchy data, runtime assumptions, compiler meta data
8

Strengths and Weaknesses
JIT
Code Performance (steady state)
Runtime: adapt to changes
Ease of use
Platform neutral deployment
Start up (ready to handle load)
Ramp up (until steady state)
Runtime: CPU & Memory
9

JIT
Ease of use
Everyone hopes:
Maybe AOT helps here?
10

AOT = Ahead of Time
• Introduce an ”extra” step to generate native code before deploying application
• e.g. run jaotc command to convert class files to a platform specific “shared object”
• Akin to approach taken by less dynamic languages: C, C++, Rust, go, Swift, etc.
• Still considered “experimental” (JDK9+) and works on x86-64 and AArch64 platforms
• Two deployment options (decided at build time):
• No JIT at runtime: statically compiled code runs, anything else interpreted
• With JIT at runtime: runtime JIT (re)compiles via triggers or heuristics
• AOT has some runtime advantages over a JIT compiler
• Compiled code performance “immediately” (no wait to compile)
• Start-up performance can be 20-50% better especially if combined with AppCDS
• Reduces CPU & memory impact of JIT compiler 11

BUT there are a few big BUTs
• No longer platform neutral
• Different AOT code needed for each deployment platform (Linux, Mac, Windows)
• Other usability issues
• Some deployment options decided at build time, e.g. GC policy, ability to re-JIT, etc.
• Different platforms: different classes load and methods to compile?
• Ongoing curation for list of classes/modules, methods to compile as your application
and its dependencies evolve
• What about classes that aren’t available until the run starts?
• How about those reasons for excellent JIT performance?
1. Speculate on class hierarchy? Not as easy as for JIT
2. Profile data? Not as easy as for JIT
• AOT compilers (in pure form) can only reason about what happens at runtime
12

Sidebar: Life of a running Java application
”Big bang”
(java process
created)
Time
13

”Big bang”
(java process
created)
JVM loaded,
initialized &
about to
load first
class to run
main()
Size and Complexity
of
Class Hierarchy
14
Time

”Big bang”
(java process
created)
Finally ready
to run main()
~ 750 classes
loaded,
handful of
class loader
objects active
App class loading and
init, can be 100s active
class loaders, 1000s
classes
15
JVM loaded,
initialized &
about to
load first
class to run
main()
Size and Complexity
of
Class Hierarchy
Time

”Big bang”
(java process
created)
App class loading and
initialization phase, up
to 100s active class
loaders, 10,000s classes
Size and Complexity
of
Class Hierarchy
16
Finally ready
to run main()
~ 750 classes
loaded,
handful of
class loader
objects active
JVM loaded,
initialized &
about to
load first
class to run
main()
Time

”Big bang”
(java process
created)
Ready to do
application work:
begin exercising
code paths
May load more
classes, may
invalidate early
assumptions
Size and Complexity
of
Class Hierarchy
17
Finally ready
to run main()
~ 750 classes
loaded,
handful of
class loader
objects active
JVM loaded,
initialized &
about to
load first
class to run
main()
Startup
Time

”Big bang”
(java process
created)
Ready to do
application work:
begin exercising
code paths
May load more
classes, may
invalidate early
assumptions
Code paths
& profile
stabilizes
Size and Complexity
of
Class Hierarchy
18
Finally ready
to run main()
~ 750 classes
loaded,
handful of
class loader
objects active
JVM loaded,
initialized &
about to
load first
class to run
main()
RampupStartup
Time

JIT compiler’s view is inside the process
”Big bang”
(java process
created)
Ready to do
application work:
begin exercising
code paths
May load more
classes, may
invalidate early
assumptions
Code paths
& profile
stabilizes
JIT
Size and Complexity
of
Class Hierarchy
19
Finally ready
to run main()
~ 750 classes
loaded,
handful of
class loader
objects active
JVM loaded,
initialized &
about to
load first
class to run
main()
RampupStartup
Time

AOT compiler’s view is through the “big bang”
”Big bang”
(java process
created)
Ready to do
application work:
begin exercising
code paths
May load more
classes, may
invalidate early
assumptions
Code paths
& profile
stabilizes
AOT
Size and Complexity
of
Class Hierarchy
20
Finally ready
to run main()
~ 750 classes
loaded,
handful of
class loader
objects active
JVM loaded,
initialized &
about to
load first
class to run
main()
RampupStartup
Time

Imagine two classes B,C: C.foo() calls B.bar()
Simple opportunity to inline call to b.bar()?
class C {
public void foo() {
B b = get_a_b();
= b.bar();
…
}
}
class B {
public int bar() {
return 5;
}
}
22

Imagine two classes B,C: C.foo() calls B.bar()
Can now optimize C.foo() using ‘5’
class C {
public void foo() {
B b = get_a_b();
= 5; //b.bar();
…
}
}
class B {
public int bar() {
return 5;
}
}
23

But C’s notion of B is decided by C’s class loader
class C {
public void foo() {
B b = get_a_b();
= 5; //b.bar();
…
}
}
ClassLoader CL1
class B {
public int bar() {
return 5;
}
}
C B
24

C’s ClassLoader is a Java object created on heap
class C {
public void foo() {
B b = get_a_b();
= 5; //b.bar();
…
}
}
ClassLoader CL1
class B {
public int bar() {
return 5;
}
}
C B
Java heap
25

Class loader objects can invalidate the inlining…
class C {
public void foo() {
B b = get_a_b();
b.bar(); // 5 or -5?
…
}
}
ClassLoader CL1
ClassLoader CL2
class B {
public int bar() {
return 5;
}
}
class B {
public int bar() {
return -5;
}
}
C
C B
B
Java heap
26

… and C.foo() may be what resolves B !
class C {
public void foo() {
B b = get_a_b();
b.bar(); // 5 or -5?
…
}
}
ClassLoader CL1
ClassLoader CL2
class B {
public int bar() {
return 5;
}
}
class B {
public int bar() {
return -5;
}
}
C
C B
B
Java heap
27

In each run, maybe only CL1 or only CL2 or
could be both: AOT probably has to hedge
class C {
public void foo() {
B b = get_a_b();
b.bar(); // 5 or -5?
…
}
}
ClassLoader CL1
ClassLoader CL2
class B {
public int bar() {
return 5;
}
}
class B {
public int bar() {
return -5;
}
}
C
C B
B
Java heap
28

Contrived example?
• Modelled on OSGi modules enabling two different versions of the same library
to be loaded at the same time (i.e. jar file hell)
• But ask yourself: what prevents this scenario if classes can be loaded
dynamically and even created on the fly?
• AOT must completely understand how class loaders will operate at runtime
• JIT acts at runtime and easily deals even with both cases coexisting
• Each “C” loads as a different j/l/Class so each C.foo() compiled independently
• i.e. inline b.bar() returning 5 in one case and returning -5 in the other
• For AOT compiler, every inlining hedge reduces optimization scope
• Increasing gap to JIT performance levels
29

Profile Directed Feedback (PDF) may help?
• BUT: AOT code must run all possible user executions
• No longer compiling for “this” user on “this” run
• Really important to use representative input data when collecting profile for AOT
• Risk: can be misleading to use only a few input data sets
• AOT compiler can specialize to one data set and then run well on it
• But PDF can lead compiler astray if data isn’t properly representative
• Monomorphic in one runtime instance ≠ Monomorphic across all runtime instances
• Benchmarks may not stress AOT compilers properly (not many input sets)
• Cross training critically important
• Input data sets need to be curated and maintained as application and users evolve
• Profile data collection and curation responsibility is on the application provider
• Observation: PDF has not really been a huge success for static languages 30

JIT AOT
Ease of use
31

JIT AOT AOT
+JIT
Ease of use
32

Is that as good as it gets?
33

Caching JIT Compiles
• Basic idea:
• Store JIT compiled code (JIT) in a cache for loading by other JVMs (“AOT”)
• Goal: JIT compiled code performance levels earlier
• Also: reduce JIT compiler’s transient CPU and memory overheads
• Really different than AOT ? No and Yes
• From perspective of second+ JVM: code loads as if it was AOT compiled
• First JVM: JIT compiles while app runs but generates code that can be cached
• Need meta data to validate later runs match first (i.e. same classes loaded same way)
• If invalid, don’t use cached code: instead do JIT or even more AOT recompilations
• Return to platform neutrality!
• Different users still get compiled code tailored for their environment
34

Two implementations
1. ”Dynamic AOT” in Eclipse OpenJ9 open source JVM1
• Originally introduced in 2007 (IBM SDK for Java 6), currently in JDK8 and later
• Stores (warm) compiled JIT code to shared memory cache (also persisted on disk)
• Performance for loaded code within 5%-10% of peak JIT performance (getting better)
• Resilient to application changes
2. “Compile Stashing”in Azul’s proprietary Falcon JIT2
• Introduced in 2018 for JDK8 and later
• Stores compiled code to disk
• Stashed code typically reuseable in another run for 60-80% of methods
• JIT recompilations recover remaining performance
• Resilient to application changes
1 https://blog.openj9.org/2018/10/10/intro-to-ahead-of-time-compilation/
2 https://www.slideshare.net/dougqh/readynow-azuls-unconventional-aot
35

OpenJ9: Caching JIT code accelerates start-up
• OpenJ9 Shared Class Cache (SCC)
• Memory mapped file for caching:
• Class files*
• AOT compiled code
• Profile data, hints
• Population of the cache happens
naturally and transparently at runtime
• Also -Xtune:virtualized
• Caches JIT code even more aggressively
to accelerate ramp-up (under load)
• Maybe slight (5-10%) performance drop
36
0
20
40
60
80
100
120
OpenJ9 no
SCC
OpenJ9
default
SCC
OpenJ9
full SCC
HotSpot
Normalizedstart-uptime
Apache Tomcat 8 Start-up Time
28%
43%
19%
• SCC for JCL bootstrap classes enabled by default
• Use -Xshareclasses option for full sharing
* Technically an internal format that can load faster than a .class file

JIT AOT AOT
+JIT
Cache
JIT
Ease of use
Start up (ready to handle load) *
Ramp up (until steady state) *
Runtime: CPU & Memory *
* After first run 37

Still some “not green” boxes there
…even for caching JITs…
L
38

Outline
• Let’s compare:
• JIT
• AOT
• Caching JIT code
• Taking JITs to the cloud
• Wrap Up
39

What if the JIT became a JIT Server
JIT
JVM
JIT
Server
JIT
JVM
JIT
JVM
Orchestrator
load balancing,
affinity, scaling,
reliability
JIT
Server
JVM client identifies methods to compile, but asks server to do the actual compilation
• JIT server asks questions to the client JVM (about classes, environment, etc.)
• Sends generated code & meta data back to be installed in client’s code cache 40

Benefits of an independent JIT server
• Move much of JIT induced CPU and memory spikes away from client
• Client CPU and memory consumption dictated by application
• JIT server connected to client JVM at runtime, so:
• Theoretically no loss in performance using same profile and class hierarchy info
• Still adaptable to changing conditions
• JVM client still platform neutral
41

AcmeAir rampup with JIT Server using
–Xshareclasses
All JVMs run in containers, client and server on different machines with direct cable connection
Note: Hotspot takes twice as long as OpenJ9 to ramp up to about the same performance level
0
1000
2000
3000
4000
5000
6000
0 100 200 300 400 500 600
Throughput(pages/sec)
Time (sec)
AcmeAir with -Xshareclasses (Cold Run)
Container limits: 1P, 150M
JITServer-cold OpenJ9-cold
0
1000
2000
3000
4000
5000
6000
0 100 200 300 400 500 600
Time (sec)
AcmeAir with -Xshareclasses (Warm Run)
Container limits: 1P, 150M
JITServer-warm OpenJ9-warm
43

JITServer Performance – Daytrader 7 Throughput
Throughput benefits grow in constrained environments
0
200
400
600
800
1000
1200
1400
0 100 200 300 400 500 600
Time (sec)
--cpus=1, -m=300m
JITServer OpenJ9
0
200
400
600
800
1000
1200
1400
0 100 200 300 400 500 600
Time (sec)
--cpus=1, -m=256m
JITServer OpenJ9
0
200
400
600
800
1000
1200
1400
0 100 200 300 400 500 600
Time (sec)
--cpus=1, -m=200m
JITServer OpenJ9
Smaller memory limit
44

What about network latency?
Won’t that hurt start up and ramp up?
Will it be practical in the cloud?
45

JIT Server works well on Amazon AWS!
46
* JITaaS == JIT Server

JIT AOT AOT
+JIT
Cache
JIT
JIT
Server
Ease of use
Platform Neutral deployment
Start up (ready to handle load) * **
Ramp up (until steady state) * **
Runtime: CPU & Memory *
* After first run ** After first run across cluster47

JIT Server Current Status
• Code is fully open source at Eclipse Open J9 and Eclipse OMR
• Has now been merged into our master branch but not yet built-in by default
• Simple options lend well to all kinds of Java workload deployments
• Server: java -XX:+StartAsJITServer –XX:JITServerPort=<port>
• Client: java -XX:+UseJITServer -XX:JITServerPort=<port>
-XX:JITServerAddress=<host> YourJavaApp
• Current focus is ensuring stability so it can be built into OpenJ9 by default
• Targeting early 2020 (OpenJ9 0.18 release) to be included in our release
binaries (JDK8 and up) at AdoptOpenJDK
48

We are really just at the beginning…
• Primary focus has been on mechanics to move JIT compilation to a server
• Once compilation work is redirected to server :
• Do that work more efficiently across a cluster of JVMS (think microservices)
• Classify and categorize JVM clients using machine learning
• Optimize groups of microservices together
• …
49

Wrapping up
• JITs continue to provide the best peak performance
• AOT compilers can improve start-up by 20-50% but expect steady-state
performance to be less than JIT performance
• Some serious usability issues; I think caching JITs are easier to use
• Caching JIT compilers are within ~5-10% of JIT with excellent start-up and ramp-
up even for large complex JakartaEE applications
• Still room to improve both throughput and start up without sacrificing compliance
• JIT Servers are coming with Eclipse OpenJ9!
• Hopefully built into AdoptOpenJDK binaries in early 2020!
50

51
https://adoptopenjdk.net
Select “OpenJ9”
Button!!

Important disclaimers
• THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY.
• WHILST EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION
CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED.
• ALL PERFORMANCE DATA INCLUDED IN THIS PRESENTATION HAVE BEEN GATHERED IN A CONTROLLED
ENVIRONMENT. YOUR OWN TEST RESULTS MAY VARY BASED ON HARDWARE, SOFTWARE OR
INFRASTRUCTURE DIFFERENCES.
• ALL DATA INCLUDED IN THIS PRESENTATION ARE MEANT TO BE USED ONLY AS A GUIDE.
• IN ADDITION, THE INFORMATION CONTAINED IN THIS PRESENTATION IS BASED ON IBM’S CURRENT
PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM, WITHOUT NOTICE.
• IBM AND ITS AFFILIATED COMPANIES SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT
OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.
• NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:
• CREATING ANY WARRANT OR REPRESENTATION FROM IBM, ITS AFFILIATED COMPANIES OR ITS
OR THEIR SUPPLIERS AND/OR LICENSORS
53

Legal Notice
IBM and the IBM logo are trademarks or registered trademarks of IBM Corporation, in the United States, other
countries or both.
Java and all Java-based marks, among others, are trademarks or registered trademarks of Oracle in the United
States, other countries or both.
Other company, product and service names may be trademarks or service marks of others.
THE INFORMATION DISCUSSED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES
ONLY. WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION,
IT IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, AND IBM SHALL NOT BE
RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, SUCH
INFORMATION. ANY INFORMATION CONCERNING IBM'S PRODUCT PLANS OR STRATEGY IS SUBJECT TO
CHANGE BY IBM WITHOUT NOTICE.
54

You can prepopulate Docker containers
with Shared Caches (SCCs)
• Prepopulating Docker containers with shared
caches very efficient with new SCC layers
• Working in synergy with Docker layers
• Each Docker layer can prepopulate its own SCC
layer that is independent of lower SCC layers
• Each SCC layer can be trimmed-to-fit because
upper layers won’t add to it
• Layers are transparent at runtime
• Classes and code will load from correct layer
• Significant reduction in disk footprint of Docker
images that package a SCC
• Faster pushing/pulling of Docker images from a
Docker registry
56
84
8
17
6
84
18
21
8
84
26
25
10
0
50
100
150
200
250
On Disk Over Net On Disk Over Net
Single-Layer SCC Multi-Layer SCC
TotalSCCImageSize(MiB)
Single- vs Multi-Layer SCC
App Layer SCC
Liberty Layer SCC
Java Layer SCC

Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
tradeoffs-java-compilers/

Understand the Trade-offs Using Compilers for Java Applications

More Related Content

What's hot (20)

Similar to Understand the Trade-offs Using Compilers for Java Applications (20)

More from C4Media (20)

Recently uploaded (20)

Understand the Trade-offs Using Compilers for Java Applications