Continuous Performance Testing
1 like255 views
The document discusses the importance of continuous performance testing in software development, highlighting various benchmarks used to assess system performance, including nanobenchmarks, microbenchmarks, component benchmarks, and system performance tests. It emphasizes that process maturity in performance testing leads to increased confidence and maintenance costs while outlining strategies for effective monitoring and regression testing. Additionally, it advocates for using production-grade tools and methodologies to ensure accurate performance assessments and rapid feedback cycles.
Continuous Performance Testing
- 1. Continuous Performance Testing Mark Price / @epickrram Performance Engineer Improbable.io
- 2. InfoQ.com: News & Community Site Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ cd-performance-testing • Over 1,000,000 software developers, architects and CTOs read the site world- wide every month • 250,000 senior developers subscribe to our weekly newsletter • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • 2 dedicated podcast channels: The InfoQ Podcast, with a focus on Architecture and The Engineering Culture Podcast, with a focus on building • 96 deep dives on innovative topics packed as downloadable emags and minibooks • Over 40 new content items per week
- 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon London www.qconlondon.com
- 4. The ideal System performance testing as a first-class citizen of the continuous delivery pipeline
- 5. Process
- 6. Process maturity A scientific and rigorous survey
- 7. Process maturity A scientific and rigorous survey
- 8. Process maturity “As part of QA, the whole team logs on to the system to make sure it scales”
- 9. Process maturity “We have some hand-rolled benchmarks that prove our code is fast”
- 10. Process maturity “We use a well-known testing framework for our benchmarks”
- 11. Process maturity “Our benchmarks are run as part of CI”
- 12. Process maturity “Trend visualisations of system performance are available”
- 13. Process maturity “There is a release gate on performance regression”
- 14. Increasing process maturity Implies: Higher maintenance cost Greater confidence
- 15. Scopes
- 16. Performance test scopes ● Nanobenchmarks ● Microbenchmarks ● Component Benchmarks ● System performance tests
- 17. Nanobenchmarks ● Determine the cost of something in the underlying platform or runtime ● How long does it take to retrieve System.nanoTime()? ● What is the overhead of retrieving AtomicLong vs long? ● Invocation times on the order of 10s of nanoseconds
- 18. Nanobenchmarks ● Susceptible to jitter in the runtime/OS ● Unlikely to need to regression test these... ● Unless called very frequently from your code
- 19. Message callback @Benchmark @BenchmarkMode(Mode.Throughput) @OutputTimeUnit(TimeUnit.SECONDS) public void singleCallback(final Blackhole blackhole) { callback.accept(blackhole); } @Benchmark @BenchmarkMode(Mode.Throughput) @OutputTimeUnit(TimeUnit.SECONDS) public void singleElementIterationCallback(final Blackhole blackhole) { for (Consumer<Blackhole> objectConsumer : callbackList) { objectConsumer.accept(blackhole); } }
- 20. Message callback
- 21. Microbenchmarks ● Test small, critical pieces of infrastructure or logic ● E.g message parsing, calculation logic ● These should be regression tests ● We own the code, so assume that we’re going to break it ● Same principle as unit & acceptance tests
- 22. Microbenchmarks ● Invaluable for use in optimising your code (if it is a bottleneck) ● Still susceptible to jitter in the runtime ● Execution times in the order of 100s of nanos/single-digit micros ● Beware bloat
- 23. Risk analysis - long vs double BigDecimal long double
- 24. Component benchmarks ● ‘Service’ or ‘component’ level benchmarks ● Whatever unit of value makes sense in the codebase ● Wire together a number of components on the critical path ● We can start to observe the behaviour of the JIT compiler (i.e. inlining)
- 25. Component benchmarks ● Execution times in the 10s - 100s of microseconds ● Useful for reasoning about maximum system performance ● Runtime jitter less of an issue, as things like GC/de-opts might start to enter the picture ● Candidate for regression testing
- 26. Matching Engine - no-ops are fast!
- 27. System performance tests ● Last line of defence against regressions ● Will catch host OS configuration changes ● Costly, requires hardware that mirrors production ● Useful for experimentation ● System recovery after failure ● Tools developed for monitoring here should make it to production
- 28. System performance tests ● Potentially the longest cycle-time ● Can provide an overview of infrastructure costs (e.g network latency) ● Red-line tests (at what point will the system fail catastrophically) ● Understand of interaction with host OS more important ● Regressions should be visible
- 30. Performance testing trade-offs Nanobenchmarks Microbenchmarks Component Benchmarks System Tests ● Slower feedback ● Hardware cost ● Maintenance cost ● KPI/SLA indicator ● Realism ● Faster feedback ● System jitter magnified ● Fewer moving parts ● Stability
- 31. Measurement
- 32. System jitter is a thing
- 33. Reducing runtime jitter Histogram of invocation times (via JMH) Run-to-run variation Large error values around average
- 35. Measurement apparatus Use a proven test-harness If you can’t: Understand coordinated omission Measure out-of-band Look for load-generator back-pressure
- 36. Production-grade tooling Monitoring and tooling used in your performance environment should be productionised
- 37. Containers and the cloud Measure the baseline of system jitter Network throughput & latency: understand what is an artifact of our system and what is the infrastructure End-to-end testing is more important here since there are many more factors at play adding to latency long-tail
- 38. Reporting
- 39. Charting “Let’s chart our benchmark results so we’ll see if there are regressions”
- 40. Charting
- 41. Charting
- 42. Charting
- 43. Charting Make a computer do the analysis We automated manual testing, we should automate regression analysis Then we can selectively display charts Explain the screen in one sentence, or break it down
- 44. Improvement
- 47. Virtuous cycle Measure Model ExecuteMeasure Compare Use the same tooling Track divergence
- 48. Regression tests If we find a performance issue, try to add a test that demonstrates the problem This helps in the investigation phase, and ensures regressions do not occur Be careful with assertions
- 49. In a nutshell...
- 50. Key points Use a known-good framework if possible If you have to roll your own: peer review, measure it, understand it Data volume can be oppressive, use or develop tooling to understand results Test with realistic data/load distribution
- 51. Key points Are we confident that our performance testing will catch regressions before they make it to production?
- 52. Thank you! ● @epickrram ● https://epickrram.blogspot.com ● recruitment@improbable.io
- 53. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ cd-performance-testing