JITServerTalk-OSS-2023.pdf
- 1. The Next Frontier in Open Source Java Compilers: Just-In-Time Compilation as a Service 1
- 2. Rich Hagarty IBM Software Developer / Advocate rich.hagarty@ibm.com @rhagarty8 https://www.linkedin.com/in/rhagarty/ 2
- 3. Agenda REASON: JVM and JIT compiler – the good and the bad 3 JIT-as-a-Service SOLUTION: JIT-as-a-Service to the rescue PROBLEM: Java on Cloud - a bad fit for microservices
- 4. Java on Cloud A Good Fit? 4
- 5. Legacy Java Apps 5 •Java monolith on dedicated server •Plenty of CPU power and memory •Never went down •6 month upgrade/refresh schedule
- 6. Moving to the Cloud 6 -Running in containers -Managed by Cloud Provider (and K8s) -Auto-scaling to meet demand Cloud native App talking to Microservices
- 7. Main Motivators 7 -Flexible & scalable -Easier to roll-out new releases more frequently -Take advantage of latest-greatest Cloud technologies -Less infrastructure to maintain and manage -Saving money
- 8. Performance vs Cost 8 Performance Cost Variables – Container Size and Container Instances Why is this so hard to get right?
- 9. JVM and JIT compiler: The Good and The Bad 9
- 10. Java Virtual Machine (JVM) 10 The Good • Device independent – write once, run anywhere • > 25 years of improvements • JIT produces optimized machine code through use of Profilers • Efficient garbage collection • Longer it runs, the better it runs (JVM collects more profile data, JIT compiles more methods)
- 11. Java Virtual Machine (JVM) 11 The Bad • Initial execution run is “interpreted”, which is relatively slow • “Hot Spot” methods compiled by JIT can create CPU and memory spikes • CPU spikes cause lower QoS • Memory spikes cause OOM issues, including crashes • Slow start-up time • Slow ramp-up time
- 12. Java Virtual Machine (JVM) 12 0 50 100 150 200 250 300 350 400 0 30 60 90 CPU utilization (%) Time (sec) Daytrader7 CPU consumption CPU spikes caused by JIT compilation 0 100000 200000 300000 400000 500000 600000 0 30 60 90 Resident set size (KB) Time (sec) Daytrader7 memory footprint Footprint spikes caused by JIT compilation
- 13. Cost vs Performance - Revisited 13 Finding the ”Sweet Spot” • Right-sized provisioned containers • Well-behaved efficient auto-scaling Let’s revisit our 2 variables
- 14. Container Size 14 Main issues: •Need to over-provision to avoid OOM •Very hard to do – JVMs have a non-deterministic behavior 0 100000 200000 300000 400000 500000 600000 0 30 60 90 Resident set size (KB) Time (sec) Daytrader7 memory footprint Footprint spikes caused by JIT compilation
- 15. Container Instances (Auto-Scaling) 15 Main issues: •Slow start-up and ramp-up times •CPU spikes can cause auto-scaler to incorrectly launch additional instances
- 16. Cost vs Performance - Revisited 16 Solution •Minimize/eliminate CPU and memory spikes •Improve start-up and ramp-up times
- 18. JIT-as-a-Service Decouple the JIT compiler from the JVM and let it run as an independent process Offload JIT compilation to remote process Remote JIT Remote JIT JVM JIT JVM JIT Kubernetes Control Plane Treat JIT compilation as a cloud service • Auto-managed by orchestrator • A mono-to-micro solution • Local JIT still available 18
- 19. Eclipse OpenJ9 JITServer • JITServer feature is available in the Eclipse OpenJ9 JVM • “Semeru Cloud Compiler” when used with Semeru Runtimes • OpenJ9 combines with OpenJDK to form a full JDK Link to GitHub repo: https://github.com/eclipse-openj9/openj9 19
- 20. Overview of Eclipse OpenJ9 Designed from the start to span all the operating systems needed by IBM products This JVM can go from small to large Can handle constrained environments or memory rich ones Renowned for its small footprint, fast start-up and ramp-up time Is used by the largest enterprises on the planet 20
- 22. IBM Semeru Runtimes “The part of Java that’s really in the clouds” IBM-built OpenJDK runtimes powered by the Eclipse OpenJ9 JVM No cost, stable, secure, high performance, cloud optimized, multi- platform, ready for development and production use Open Edition • Open source license (GPLv2+CE) • Available for Java 8, 11, 17, 18 (soon 19) Certified Edition • IBM license • Java SE TCK certified. • Available for Java 11, 17 22
- 23. JITServer advantages for JVM Clients 23 Provisioning Easier to size; only consider the needs of the application Performance Improved ramp-up time due to JITServer supplying extra CPU power when the JVM needs it the most. Reduced CPU consumption with JITServer AOT cache Cost Reduced memory consumption means increased application density and reduced operational cost. Efficient auto-scaling – only pay for what you need/use. Resiliency If the JITServer crashes, the JVM can continue to run and compile with its local JIT
- 25. JITServer value in Kubernetes • https://blog.openj9.org/2021/10/20/save-money-with-jitserver-on-the- cloud-an-aws-experiment/ • Experimental test bed • ROSA (RedHat OpenShift Service on AWS) • Demonstrate that JITServer is not tied to IBM HW or SW • OCP cluster: 3 master nodes, 2 infra nodes, 3 worker nodes • Worker nodes have 8 vCPUs and 16 GB RAM (only ~12.3 GB available) • Four different applications • AcmeAir Microservices • AcmeAir Monolithic • Petclinic (Springboot framework) • Quarkus • Low amount of load to simulate conditions seen in practice • OpenShift Scheduler to manage pod and node deployments/placement 25
- 26. JITServer improves container density and cost Default config AM 500 B 550 C 550 F 450 P 450 P 450 B 550 F 450 AM 500 A 350 AM 500 M 200 Q 350 P 450 Q 350 D 600 D 1000 F 450 B 550 Q 350 AM 500 AM 500 AM 500 B 550 B 550 A 350 C 550 F 450 M 200 P 450 P 450 P 450 Q 350 Q 350 D 1000 AM 500 B 550 P 450 AM 500 B 550 B 550 C 550 C 550 F 450 F 450 P 450 Q 350 Q 350 D 1000 D 1000 Q 350 AM 250 AM 250 P 250 P 250 F 250 F 250 B 400 C 350 Q 150 Q 150 M 150 AM 250 AM 250 P 250 P 250 F 250 B 400 Q 150 Q 150 J 1200 A 250 B 400 B 400 C 350 D 1000 D 1000 D 600 AM 250 AM 250 P 250 P 250 F 250 F 250 B 400 C 350 Q 150 Q 150 M 150 AM 250 AM 250 P 250 P 250 F 250 B 400 Q 150 Q 150 J 1200 A 250 B 400 B 400 C 350 D 1000 D 1000 JITServer config Legend: AM: AcmeAir monolithic A: Auth service B: Booking service C: Customer service D: Database (mongo/postgres) F: Flight service J: JITServer M: Main service P: Petclinic Q: Quarkus Total=8250 MB Total=8550 MB Total=8600 MB Total=9250 MB Total=9850 MB 6.3 GB less 26
- 27. Throughput comparison 0 200 400 600 800 1000 1200 0 240 480 720 960 1200 1440 1680 1920 Throughput (pages/sec) Time (sec) AcmeAir Microservices 0 200 400 600 800 1000 1200 0 240 480 720 960 1200 1440 1680 1920 Throughput (pages/sec) Time (sec) AcmeAir Monolithic 0 20 40 60 80 100 120 140 160 180 0 240 480 720 960 1200 1440 1680 1920 Throughput (pages/sec) Time (sec) Petclinic 0 500 1000 1500 2000 2500 3000 3500 0 240 480 720 960 1200 1440 1680 1920 Throughput (pages/sec) Time (sec) Quarkus Machine load: 17.5% from apps 7% from OpenShift 27 è JITServer and default configuration achieve the same level of throughput at steady-state JITServer Baseline
- 28. Conclusions from high density experiments • JITServer can improve container density and reduce operational costs of Java applications running in the cloud by 20-30% • Steady-state throughput is the same despite using fewer nodes 28
- 29. Horizontal Pod Autoscaling in Kubernetes • Better autoscaling behavior with JITServer due to faster ramp-up • Less risk to trick the HPA due to transient JIT compilation overhead 29 Setup: Single node Microk8s cluster (16 vCPUs, 16 GB RAM) JVMs limited to 1 CPU, 500MB JITServer limited to 8 CPUs and has AOT cache enabled Load applied with JMeter, 100 threads, 10 ms think-time, 60s ramp-up time Autoscaler: scales up when average CPU utilization exceeds 0.5P. Up to 15 AcmeAir instances 0 1000 2000 3000 4000 5000 6000 7000 8000 9000 0 60 120 180 240 300 360 420 480 Throughput (pages/sec) Time (sec) AcmeAir throughput when using Kubernetes autoscaling Baseline JITServer+AOTcache
- 30. DEMO 30
- 31. Improve ramp-up time with JITServer • Experiment in docker containers • Show that JITServer improves ramp-up • Show that JITServer allows a lower memory limit for JVM containers 31 OpenLiberty+ AcmeAir 1P, 400 MB limit OpenLiberty+ AcmeAir 1P, 200 MB limit OpenLiberty+ AcmeAir 1P, 200 MB limit MongoDB JITServer 4P, 1GB limit JMeter JMeter JMeter InfluxDB Grafana Collect throughput data from JMeter Display throughput data Apply load to AcmeAir instances Provide data persistence services Run the AcmeAir application Provide JIT compilation services Grafana Prometheus Scrape metrics Display JITServer metrics
- 32. 32
- 33. 33
- 34. How to use it 34
- 35. JITServer usage basics • One JDK, three different personas • Normal JVM: $JAVA_HOME/bin/java MyApp • JITServer: $JAVA_HOME/bin/jitserver • Client JVM: $JAVA_HOME/bin/java -XX:+UseJITServer MyApp • Optional further configuration through JVM command line options • At the server: -XX:JITServerPort=… default: 38400 • At the client: -XX:JITServerAddress=… default: ‘localhost’ -XX:JITServerPort=… default: 38400 • Full list of options: https://www.eclipse.org/openj9/docs/jitserver/ • Note: Java version and OpenJ9 release at client and server must match 35
- 36. JITServer usage in Kubernetes • Typically we create/configure • JITServer deployment • JITServer service (clients interact with service) • Use • Yaml files • Helm charts: repo https://raw.githubusercontent.com/eclipse/openj9-utils/master/helm-chart/ • Certified Operators • Tutorial: https://developer.ibm.com/tutorials/using-openj9-jitserver-in- kubernetes/ 36
- 37. JITServer encryption/authentication through TLS • Needs additional JVM options • Server: -XX:JITServerSSLKey=key.pem -XX:JITServerSSLCert=cert.pem • Client: -XX:JITServerSSLRootCerts=cert.pem • Certificates and keys can be provided using Kubernetes TLS Secrets • Create TLS secret: • kubectl create secret tls my-tls-secret --key <private-key-filename> --cert <certificate-filename> • Use a volume to map “pem” files 37 apiVersion: v1 kind: Pod metadata: name: my-pod spec: containers: - name: my-container-name image: my-image volumeMounts: - name: secret-volume mountPath: /etc/secret-volume volumes: - name: secret-volume secret: secretName: my-tls-secret
- 38. Monitoring • Support for custom metrics for Prometheus • Metrics scrapping: GET request to http://<jitserveraddress>:<port>/metrics • Command line options: -XX:+JITServerMetrics -XX:JITServerMetricsPort=<port> • Metrics available • jitserver_cpu_utilization • jitserver_available_memory • jitserver_connected_clients • jitserver_active_threads • Verbose logging • Print client/server connections -XX:+JITServerLogConnections • Heart-beat: periodically print to verbose log some JITServer stats • -Xjit:statisticsFrequency=<period-in-ms> • Print detailed information about client/server behavior -Xjit:verbose={JITServer},verbose={compilePerformance},vlog=… 38
- 39. JITServer usage recommendations • When to use it • JVM needs to compile many methods in a relatively short time • JVM is running in a CPU/memory constrained environment, which can worsen interference from the JIT compiler • The network latency between JITServer and client VM is relatively low (<1ms) • To keep network latency low, use “latency-performance” profile for tuned and configure your VM with SR-IOV • Recommendations • 10-20 client JVMs connected to a single JITServer instance • JITServer needs 1-2 GB of RAM • Better performance if the compilation phases from different JVM clients do not overlap (stagger) • Encryption adds to the communication overhead; avoid if possible • In K8s use “sessionAffinity” to ensure a client always connects to the same server • Enable JITServer AOT cache: -XX:+JITServerUseAOTCache (client needs to have shared class cache enabled) 39
- 40. Open Liberty InstantOn • InstantOn improves start-up times 10-20x • InstantOn improves Semeru Cloud Compiler scaling 40
- 41. Final thoughts • JIT provides advantage, but compilation adds overhead • Disaggregate JIT from JVM è JIT compilation as a service • Eclipse OpenJ9 JITServer (a.k.a Semeru Cloud Compiler) • Available now on Linux for Java 8, Java 11 and Java 17 (IBM Semeru Runtimes) • Especially good for constrained environments (micro-containers) • Kubernetes ready (Helm chart available, Prometheus integration) • Can improve ramp-up, autoscaling and performance of short lived applications • Can reduce peak memory footprint, increase app density and reduce costs • Java solution to Java problem, with no compromise 41