Performance testing and_reporting_with_j_meter by Le Van Nghi

Performance testing and reporting with JMeter Introduced by: Nghi Nguyen Van ~ March, 1 st 2012 ~

Agenda Introduction What is performance testing, why is it important Performance testing and monitoring tools Performance testing with JMeter What to focus in a performance test report Tips and tricks Q & A

Purpose of this presentation: Introduce about performance testing activity in eXo SEA Share experience in do performance testing Share experience in working with Jmeter This presentation serves to: ● Leaders to understand about performance testing activity, reports ● Who is new in using Jmeter to gain knowledge about using Jmeter ● Who experienced in Jmeter want to gain, share & discuss about using Jmeter ● Testers who want to understand about building, using performance testing system Introduction

What is performance testing, why is it important?

Definition: Performance testing is a type of testing intended to determine the responsiveness throughput reliability and/or scalability of a system under a given workload What is performance testing

Performance testing is commonly conducted to accomplish the following: Assess production readiness Evaluate against performance criteria Compare performance characteristics of multiple systems or system configurations Find the source of performance problems Support system tuning Find throughput levels What is performance testing...

Performance test Load test Stress test Capacity test Types of Performance Testing

Application performance impacted by various of factors Last minute surprises are always bad Applications made up from hundred/thousand of components Components change day by day Performance issues are hard to fix Why is it so important?

Performance testing and monitoring tools

Imagine about a performance testing system This system must provide information about: Availability/Stability Response time Throughput Utilization

Can provide data: Availability/Stability Response time Throughput Commercial tools: LoadRunner (HP) IBM Rational Performance Tester (IBM) Visual Studio Load Test (Microsoft) Opensource tools: Apache JMeter (Apache Jakarta) The grinder OpenSTA (Open System Testing Architecture) Needed tools - Virtual users simulators

Data can provide: Percent of CPU utilization Amount of free memory Page file utilization Disk time Amount of free disk space Tools: OpenNMS Hyperic HQ Zabbix Perfmon (Windows) Needed tools – System monitoring

Can provide data: Detailed information about the application resource used Amount of used memory A mount of threads Openning files Monitoring tools: JvisualVM/VisualVM Jmxtrans + graphite Jprofiler HypericHQ Jmanage Javamelody Needed tools – application monitoring

A real performance testing system Application monitoring tool: jmxtrans, graphite Application measuring tool: JMeter System monitoring tool: OpenNMS

Performance testing with JMeter

The Apache JMeter™ desktop application is: - open source software - 100% pure Java application designed to load test functional behavior and measure performance - designed for testing Web Applicatio n JMeter features: - Can load and performance test many different server types - Complete portability and 100% Java purity - Full multithreading framework - Careful GUI design - Caching and offline analysis/replaying of test results. - Highly Extensible Apache JMeter introduction

+) Easy to use for both developer and tester record/modify with GUI understand & modify under text mode install the tool and distribute test scripts +) Good enough Rich elements that help doing performance test easily (especially web apps) Stable Good for both simple and advance test configuration Good with CI/automation test model Extendable via plugins +) Opensource +-) JMeter is a great tool but poor document Why JMeter is chosen/still be in used

Response time 90% line of response time (90 th percentile of response time) 90% of all the response time values less or equal than this value) Average response time Simple average of all values=(sum of all values from 1 st to Nth)/N Min and Max response time Error rate Throughput Performance testing notices – Basic parameters

Performance testing notices – Most reliable response time Most reliable response time = 90% line of response time (90 th percentile of response time)!

Performance testing tips - Do stability test Purpose Make sure that the test is reliable Gain better knowledge about application/test Give/confirm later tests' settings: Test duration Test cycles …

Performance testing tips - Do stability test How Choose a simple test Repeat the test for many times (e.g: >=6 times) Calculate difference from result of each test to their average Apply the performance test statements

Demo notices – Jenkins flow Jenkins flows: JMeter benchmark process Perf-suite Loop until end of setting file Various of test settings Easy to maintain/extend

Demo notices – JMeter script Introduce JMeter script How is a script used in a benchmark flow with jenkins Scripts: PLF_PERF_05_Login.jmx PLF_PERF_04_SocialRead-3.0.x.jmx PLF_PERF_03_ECMS-Public.jmx

JMeter notices – recording/modifying script Use selenium script -> this is for QA who have to repeat the scenario oftenly Save and backup the first recorded version, backup after every importance change Find and replace all the changeable values by Jmeter variables such as host address port number date and time account used in the scenario Id of components HTTP response assertions are needed to avoid something wrong

JMeter notices – script organization.p1

JMeter notices – script organization.p2 Test plan interface Jmeter properties usage can be used to make up an interface for a testscript Properties' value should be assigned to runtime variables at user defined variable table and use these variables instead This will help to make a script clear, easy to understand and maintain, especially with scripts need to be run with different configurations Ram-up, Loop duration, Think time, Thread count Rampup value helps a heavy test can be loaded smoother Loop duration: a test need to be ended with a suitable duration/loops count Think time (timer) should be used to simulate a real user Threads count will impact to test results, it is number of virtual users that you need to simulate Cookies, Session, Iteration Cookies manager is needed in most of the cases to use the same http session for each VU (iteration) Each round of a VU is an iteration All of each VU iterations can use the same http session via cookies manager setting

JMeter notices – script usage Test process (startup, warm-up, measure) Automate test rounds (jenkins & perf-suite) Confirm the test script with single VU (as a result of preheat) Perf test short duration & long duration Jmeter GUI mode vs Non-gui mode: Non-gui mode is preferred JTL file/Listeners: jtl file is needed/enough via the command jmeter -n -l output.jtl -t … listeners are not needed

Data collecting methods JMeter plugins and command line mode with JTL file Use suitable listeners in Jmeter GUI mode Merge results from multiple jtl files Monitoring tools (jmxtrans graphite, opennms/zabbix): collect via browsers

Data collecting method – JMeter-plugins command line JMeter plugins and command line mode with JTL file java -jar CMDRunner.jar --tool Reporter --input-jtl results.jtl --plugin-type <Plugin Type Classes> --generate-png/--generate-csv <file_name> [… other options...] Plugin Type Classes: AggregateReport = JMeter's native Aggregate Report , can be saved only as CSV ThreadsStateOverTime = Active Threads Over Time HitsPerSecond ResponseTimesDistribution ResponseTimesOverTime ResponseTimesPercentiles ThroughputOverTime TransactionsPerSecond

Data analysis and reporting Performance test statements Stability of cycles % of difference in response time Throughput distribution Application cycles Memory leak Session leak Resource utilization (see bottle necks) Error rate Throughput, 90% line of response time

[90% line of response time] is more reliable than [average response time] Variation in range of [-5%,5%] should be treated as normal for [response time] Variation out of range [-10%,10%] should be treated as an improvement/decline of [response time] Variation in range of [-10%,-5%) and ( 5%,10%] should be considered as a decline/improvement of [response time] Variation in range of [-3%,3%] should be treated as normal for [Throughput] Variation out of range [-5%,5%] should be treated as an improvement/decline of [Throughput] Variation in range of [-5%,-3%) and ( 3%,5%] should be considered as a decline/improvement of [Throughput] Data analysis and reporting- Performance test statements

As a result of the statements: Tests cycles on the same package and same test configuration should not be vary out of range [-5%, 5%] in comparison to their aggregation results The aggregation of cycles, itself, makes the test result more accurate, reduces error but unstable results still need to be checked. The stability of test rounds help us to give more accurate reports Data analysis and reporting – stability of cycles

% of difference in response time Throughput distribution Application cycles Data analysis and reporting – detail

Memory leak Session leak Data analysis and reporting – detail

Data analysis and reporting – resource utilization

What to focus in a performance test report

TC-PLF-3.5.1 PLF_PERF_04_SocialRead explanation https://wiki-int.exoplatform.org/display/exoReleases/TC-PLF-3.5.1+PLF_PERF_04_SocialRead A real performance test report explanation

Tips & tricks - A clear name for each performance test - non-GUI mode is more stable than GUI mode - Do not use listeners if not needed - Rampup is needed for heavy load - Assertion is needed to simulate a virtual user - Shutdown.sh -> save your effort while doing a long test - Unstable tests: think of data while an user run its scenario - If one user can not login, its later steps should not be counted - Backup after every important step you made to your script easily by cloning the jmx file - Speedup jmeter script modifying with text editors which support regex, e.g: kate

Links Http://code.google.com/p/jmeter-plugins/ http://jmeter.apache.org/ http://code.google.com/p/jmxtrans/wiki/IntroductionToJmxTrans?tm=6 https://wiki-int.exoplatform.org/display/QAF/Graphite https://wiki-int.exoplatform.org/display/TQA/Performance+test+suites+and+utilities https://wiki-int.exoplatform.org/display/TQA/PLF_PERF_04_SocialRead-3.0.x+-+PLF-3.0.7+-+DS2+stability+test https://wiki-int.exoplatform.org/display/TQA/PLF_PERF_04_SocialRead-3.5.x+-+PLF-3.5.0GA+-+DS2+stability+test https://wiki-int.exoplatform.org/display/exoReleases/TC-PLF-3.5.1_TOMCAT+Performance+Reports https://wiki-int.exoplatform.org/display/exoReleases/TC-PLF-3.5.1+PLF_PERF_04_SocialRead

Performance testing and_reporting_with_j_meter by Le Van Nghi

More Related Content

What's hot (20)

Similar to Performance testing and_reporting_with_j_meter by Le Van Nghi (20)

More from eXo Platform (19)

Recently uploaded (20)

Performance testing and_reporting_with_j_meter by Le Van Nghi

Editor's Notes