Java Performance Testing for Everyone - Shelley Lambert
Download as PPTX, PDF1 like583 views
This document discusses performance testing of Java applications and the tools used at AdoptOpenJDK. It provides an overview of the scope of testing done at AdoptOpenJDK across multiple Java implementations, platforms, and versions. It defines key performance metrics and the basic steps of performance testing. Tools used at AdoptOpenJDK are introduced, including PerfNext for running benchmarks, the Test Results Summary Service for analyzing and comparing results, and BumbleBench for simplifying microbenchmark implementation. The document encourages open collaboration to advance innovation in performance testing.
Java Performance Testing for Everyone - Shelley Lambert
- 1. Java™ Performance Testing for Everyone Shelley Lambert IBM Runtime Technologies Test Lead
- 2. Who Am I? Various Roles Developer / Test Lead Development Manager Yoga Teacher tuneupfitness.com/teacher/shelley-lambert Chief Food Forester ottawafoodforests.com nanabushfoodforests.com
- 3. The Scope AdoptOpenJDK • Projects: Eclipse OMR, Eclipse OpenJ9, AdoptOpenJDK • Ensuring Free and Verified Java™ for the Community • 6+ Jenkins servers • 18+ platforms • 6+ versions, 4+ implementations • 7+ test categories -> 100,000’s of tests • 18x6x4x100000 = ~43 million tests… nightly, around the world!
- 4. Where to start with performance? • The story of Java performance – No single recipe (Many factors: JVM implementation, hardware, application design) – JVMs evolve, performance improves • JVMs “complex, intricate, subtle” • Wouldn’t it be great if it were simple? -XX:goFaster, -XX:useLessResources
- 5. The Intersection • Necessary – developers need to know if the code they write affects performance – currently using diverse set of tools and approaches, home- made scripts, duplication, lost learning opportunities • Impossible – measuring performance often stated as “too hard” to do Necessary Impossible
- 6. Defining the “Impossible” • What is performance testing? – Often called ”Experimental science” – “Testing if a system accomplishes its designated functions within given constraints regarding processing time and throughput rate.”* • Good performance? Speed, resources or a blend – Modern language runtimes care about many different metrics • Throughput, Startup Time, Ramp-up Time, Compile Time • Footprint – Average Resident Set Size – Compilation Memory Consumption – Peak Resident Set Size * Witteveen, Albert. Performance testing - a practical guide (Kindle Locations 176-177).
- 7. What to measure Metric name What to measure? Constraints Inputs to vary Throughput # of transactions time Latency Time for single transaction # of transactions Workload (increases) Capacity # of simultaneous transactions Throughput or latency Parallel load on the system Utilization Use of resources workload Efficiency Throughput/utili zation Scalability Throughput or capacity Resources (added) Degradation Latency or throughput utilization Workload (increases) Explicit or implicit ‘inputs’ to normalize: HW, OS, system setup
- 8. Basic Steps • Set a goal – which metric(s) to improve • Measure – but how? tools? • Adjust – apply your experiments • Measure again – how exhaustive? • Verify goal – did the metrics improve? enough?
- 9. AdoptOpenJDK Testing github.com/AdoptOpenJDK/openjdk-tests • The wildly different ‘fruit’, how to make them easily consumable “Consolidate and Curate” functional openjdk perfjckexternalsystem testkitgen testNG, cmdlineteste r STF junit & others javatest Assorted benchmarks jtreg
- 10. Performance Benchmarks (Large-scale and Microbenchmarks at AdoptOpenJDK) bbenchacme-air libertydt jmhidlespark odm perf Assorted benchmarks … … … … Large scale Microbenchmarks
- 11. Introducing github.com/AdoptOpenJDK/openjdk-test-tools • PerfNext - configure, tune and launch performance benchmarks. • Test Results Summary Service (TRSS) - summarize and visualize different test results including perf results, push different sets of test results to a DB, search test and compare results across different platforms, report on differences between jobs • Future services – result analytics, test generation, core analytics, bug prediction
- 13. Track the progress of benchmark runs and verify their output
- 14. TRSS – Performance Comparison
- 15. TRSS – Performance Comparison
- 16. TRSS – Regression Analysis 1 month All data 7 days
- 17. BumbleBench “Microbenchmarks Simplified” github.com/AdoptOpenJDK/bumblebench • Writing a good microbench with an optimizing JIT running your code is hard – are you measuring what you think you are measuring? • BumbleBench is a Java framework that provide a hook point to implement the benchmark payload • Framework provides the outer timing loop, scoring infrastructure, etc.
- 18. Conclusion • Perf is hard (not impossible) – High resource requirements for full-scale testing – Microbenchmarks difficult to write – Data is noisy and subject to interpretation • Building tools to make perf easier – TRSS / PerfNext / BumbleBench • AdoptOpenJDK git repos: openjdk-tests, openjdk-test- tools, bumblebench • Coming soon -> trss.adoptopenjdk.net • Open Collaboration leads to greater Innovation – “Innovation is creativity with a job to do.” – John Emmerling
- 19. AdoptOpenJD K adoptopenjdk.net AdoptOpenJDK/openjdk-tests @adoptopenjdk eclipse.org/openj9 eclipse/openj9 @openj9 eclipse.org/omr eclipse.org/omr @eclipseomr Upcoming Talks: Performance Testing for Everyone AdoptOpenJDK: Ensuring Free Java for the Community Fuzzy Plans and Other Test Integrations Shaking Sticks and Testing OpenJDK Implementations Connect & Collaborate! Website Github Twitter 8thdaytesting.com smlambert @ShelleyMLambert
Editor's Notes
- #2: Performance Testing for Everyone Who is everyone? People like me… people who don’t have time to become performance analysis experts, but want some assurances
- #5: All the excuses for why something is not possible. History: the old joke in the early days of Java… “how do I write code that is faster?” … use a different language In our modern era, of sophisticated GC and JIT code… it’s a different story, you can still write Java is a funny language, god complex, do everything for everyone… Write once run everywhere and now we will perform magic under the covers (with GC/JIT code) that will take your average code and make it sing. How can we more easily run performance benchmarks against Java SDKs and analyze and compare results? What information is coming out of some common open-source benchmarks and why might it be interesting? How can you incorporate performance tests into your continuous delivery pipeline? This talk addresses all of these questions and more as it surveys the performance testing story at AdoptOpenJDK. In this presentation, the common user story of determining whether a code change caused a performance regression guides us through the delivery pipeline at AdoptOpenJDK, but also introduces how you can leverage the automation from that open-source projects for your own testing requirements.
- #6: How can we more easily run performance benchmarks against Java SDKs and analyze and compare results? What information is coming out of some common open-source benchmarks and why might it be interesting? How can you incorporate performance tests into your continuous delivery pipeline? This talk addresses all of these questions and more as it surveys the performance testing story at AdoptOpenJDK. In this presentation, the common user story of determining whether a code change caused a performance regression guides us through the delivery pipeline at AdoptOpenJDK, but also introduces how you can leverage the automation from that open-source projects for your own testing requirements.
- #7: Actual Metrics to Measure? Throughput, latency, capacity, utilization, efficiency, scalability, degradation Performance Testing for Everyone: Who is everyone? People like me… people who don’t have time to become performance analysis experts, but want some assurances
- #8: Throughput = transactions in given period of time Latency = time to process a single transaction as workload increases Capacity = how much parallelism in a system (related to throughput and latency) Utilization = efficient use of system resources Efficiency = throughput/utilization (cost accounting) Scalability = throughput or capacity as resources are added (target: perfect linear scaling) Degradation = how a throughput or latency are affected (system degrades) as workload increases Luckily, since we have stated we want to start somewhere and keep it simple, we can choose to focus on 2 important metrics (throughput and utilization)
- #9: Metrics 1 of 2 to start with… Toolkit is equipped with benchmarks
- #10: Talk about load/stress testing (from system tests) as a form of performance testing If a system accomplishes its designated functions within given constraints… Brief interlude to set the stage…
- #11: (Pattern is same) Hone in on performance testing (same pattern… the wildly different benchmarks, how to make them easily consumable) Large-scale open source benchmarks DayTrader3 / DayTrader7 (JavaEE stock trading) app server – flat profile, thousands of compiles Apache Spark HiBench (Big Data Analytics) Scala-based framework – more focused profiles Microbenchmarks Essence of past performance bugs API specific tests – array copy / object allocate / etc
- #20: Thanks for your attention! Let’s stay in touch!