CONTINUOUS PERFORMANCE TESTING - ISTSTC (1).pdf
0 likes51 views
The document outlines continuous performance testing (CPT) and its importance in software development, detailing various types of performance testing such as load, stress, soak, and spike tests. It emphasizes the necessity of automation and integration with continuous integration/continuous delivery (CI/CD) pipelines for effective and efficient performance testing. Additionally, it covers evaluation metrics, tools and technologies required for performance testing, and the benefits of automated performance testing in maintaining system performance.
CONTINUOUS PERFORMANCE TESTING - ISTSTC (1).pdf
- 1. Mesut Güneş Principal Consultant @Modus Create, LLC. Continuous Performance Testing - (CPT) Motto: Performance is a living thing
- 2. 2 CONTINUOUS PERFORMANCE TESTING Performance is a living thing
- 3. 3 MESUT GÜNEŞ PRINCIPAL CONSULTANT AT MODUS CREATE in/gunesmes gunesmes story/gunesmes gunesmes testrisk.com gunesmes@gmail.com
- 4. WHO YOU ARE 4 ? Testers/QAs or SDETs Developer Performance engineer Did performance testing
- 5. 5 TALK INCLUDE THESE TOPICS Performance testing Type of performance testing (load, stress, soak, volume, capacity, spike) Automation, Tools, Technologies, Benefits of Automated Tests Integration Thresholds, Percentile, rps CI/CD API Orchestration . . . and more
- 6. 6 TABLE OF CONTENTS PERFORMANCE TESTING Concept of performance CONTINUOUS PERFORMANCE TESTING Automation in performance 01 03 02 04 05 METRICS Without data nothing measurable TOOLS AND TECHNOLOGIES What kind of tools we have for this purpose BENEFITS What kind of benefits we can have
- 8. PERFORMANCE TESTING BY DEFINITION 8 Performance Testing is + A non-functional testing + Testing the functionalities of the system + Both black-box and white-box testing + Defining how the system works under any loads + Applied under ○ different load levels ○ different load increments ○ different durations Defining the performance of the system by testing the functionalities of the system in terms of the non-functionalities
- 9. 9 PERFORMANCE TESTING BY PROCESS Performance testing should be also executed in a process. Performance engineers should ensure that they collected requirements, created correct amount of virtual users, created test scenarios and they can execute them. As a final they should check the result and the system, if required after fixing, re-execution may be needed. Planning Creating test Scripts Creating the Test Env. Executing the Tests Monitoring the Systems Analyzing & Optimizing
- 10. 3 criterias + Load - #VU + Increment of the load + Duration - Mins, Hours TYPES OF PERFORMANCE TESTING 10 Types of performance testing depend on the strategy that applied during the test by means of the loads, the duration and increment of the load
- 11. LOAD TEST 11 Load Test is + Basic form + Widely known-form + Under optimum load + Giving a result that users might be facing Load test is to understand the behavior of the system under a specific load. It is performing under specific amount of the load to check if the system gives the expected responses with keeping the load stable for a long term
- 12. STRESS TEST 12 Stress Test is + Testing the upper boundary of the system + Needing more resources + Needing more investigation for boundaries + Needing more attention + Prone to break the system, aka: soak, endurance testing + good for promotion like black friday, 11.11 Stress test is to test the upper boundary of the system. When the load is increased to the level of upper boundary, test aims to get the response of the system under stress.
- 13. SOAK-ENDURANCE TEST 13 Soak Test is + Testing the upper boundary of the system + Testing the system for a prolonged time + Needing more resources + Needing more investigation for boundaries + Needing more attention + Aims to break the system + Preparation for campaigns like black friday, 11.11 Soak test is to test the upper boundary of the system for a prolonged time. Soak is good if you are not sure the duration of the user for some occasions. Eliud Kipchoge broke the 2-hour-barrier and changed the history of modern marathons in Vienna in 2019. Image: https://images.app.goo.gl/6FCD6U95h8jFWuXX9
- 14. SPIKE TEST 14 Spike Test is + Testing the sudden increase and decrease of loads + Testing the scalability + Requiring more load generations + depends of the tools (load the system, then spike it) + Good for some occasion like simulating + push notifications for e-coms + announcing critical news Spike test is to test the behaviour of the system when the amount of load is suddenly increased and decreased. It aims to find system failures when loads is changing to the unexpected amount.
- 15. 15 CAPACITY TEST Capacity Test is + Testing the max user + Finding the behavior when more users join + Providing info about the loss of user data + Ex: gradually increase the number of concurrent users accessing a website and observe the server's performance Capacity testing is aiming to find the maximum amount of the user that the system can handle. By this test we can find the how many user can use the system.
- 16. 16 VOLUME TEST Volume Test is + Testing under huge amount of data receives + Any issue with caching + Database has any issue for indexing + Ex: "Black Friday" sale scenario with a massive surge of users accessing the website simultaneously, check the scalability Volume test is testing the performance when the system receives huge amount of data. With the volume testing, we can find issues for scalability.
- 19. 19 CONTINUOUS PERFORMANCE TESTING CPT is designing the performance test to run them spontaneously by tools and evaluation the result by tools to decide to go or not to go further as part of the CI/CD pipeline. CPT is ᐩ Part of CI/CD ᐩ Shift-left and/or shift-right approaches ᐩ Subset of the performance test scenarios ᐩ Designed to run spontaneously by tools ᐩ Designed to evaluate the result by tools ᐩ Designed not to break the system ᐩ Having decision metrics ᐩ Designed to create report ᐩ PERCEPTIONAL Reference: https://www.gocd.org/2016/06/08/Add-performance-testing-to-your-Continuous-Delivery-pipeline
- 20. 20 WHAT TYPES TO AUTOMATE Continuous performance testing can be applied to the all types of performance testing. However each types need different level of maturity and sanity. Automated Performance Testing is Easily applied to load testing Very hard to apply Soak-Endurance test Hard to apply stress test, spike test, but have benefits
- 21. 21 AUTOMATED LOAD TESTING Since the load test is aiming to test the system under a specific load and it is success is belong to the predefined metrics, load testing is the best suited performance testing type for CPT. Automated Load Testing is + Needing specific load level + Needing pre-defined metrics + Needing evaluation of the result + Needing exactly the same environment for every run + Needing tools and script knowledge + Easy to integrate CI/CD pipeline - everything is scripted + Reporting the performance of the system when something changes
- 22. 22 METRICS 03
- 23. 23 METRICS Some of the Metrics are + Response Metrics + Response time + Average response time + Peak response time + Wait time + Error rate Performance testing is meaningful when the metrics are understandable for each counterparts. These metrics are the key identifier for evaluating the results and deciding if the test is successful or failed. + Volume Metrics + Concurrent users + Requests per second + Throughput
- 24. 24 RESPONSE METRICS Variants of the Response Metrics are + Response time + Time to complete a request to the system + Time to First Byte (TtFB) - response starts + Time to Last Byte (TtLB) - response ends + Wait time is the time between sending request to TtFB Response time is the main measurement taken during the test. Then we calculate the average, mean, median, minimum, maximum values of it. + Average, Median, Peak response time + Average is mean of the whole responses + Median is the one in middle between min to max + Peak is the maximum value + Error Rate + # of Error / # of Request
- 25. VOLUME METRICS Volume related metrics are depends on the loads which are generated during the tests by the tools. These metrics show how much loads was sent to the system and how much response was receipt. Basically these are requests-per-seconds (RPS), concurrent user, throughput, hatch rate Variants of the Volume Metrics are + RPS is the number of request sending per second + Concurrent User is the number of virtual users which are testing the system + Hatch Rate is the number of virtual user adding to the system per second + Throughput is the data in kilobytes which are sent from the servers
- 26. 26 EVALUATION - NORMAL DISTRIBUTION + Enough data → Normal distribution + 2 SD (standard deviation), 95 confidence level + Remove outliers + Test the hypothesis H1: every metrics is similar with the historical data within 95 confidence level
- 27. 27 EVALUATION - THRESHOLDS - STATIC DATA Thresholds is designed to define fail/pass criteria. With it you can define max, min, average, median, throughput and other metrics.
- 28. 28 EVALUATION - HISTORICAL DATA - AI HELPS We have to evaluate the response metrics statistically. Keeping the historical data as a base and tracking the trends are critical.
- 29. 29 EVALUATION - HISTORICAL DATA - AI HELPS Metrics in the historical data shows us the steady-state of the system. We can compare the current metrics statistically with the history. Hypothesis 1: Current response is within the 95 confidence level Hypothesis 2: Current response is not within the 95 confidence level
- 31. 31 TOOLS FOR CPT Volume related metrics are depends on the loads which are generated during the tests by the tools. These metrics show how much loads was sent to the system and how much response was receipt. Basically these are requests-per-seconds (RPS), concurrent user, throughput, hatch rate Tools Needed for Automated Performance Testing are + Performance Testing tool + Databases + Reporting tools + Comparison tools + Orchestration Tools + Monitoring
- 32. 32 PERFORMANCE TESTING TOOLS We have a variety of tools in the industry, they changes from open source to commercial and/or commercial support; newly started to 20+ years old performance testing tools. Some of the well-know Performance Tools are + Jmeter - Apache project, v1.0 1998 + Gatling - Enterprise support by Frontline + K6 - Enterprise support by LoadImpact + Locust + and many others + LoadRunner by Microfocus (previously Mercury - HP) + Blazemeter - support multiple opensource tools + LoadUI Pro by Smartbear Opensource Commercial
- 33. PROPRIETARY & CONFIDENTIAL 33 AUTOMATION TOOLS We need to manage these tools to run the test efficiently. Every CI/CD run on a orchestration tool that provide every needed task by using different tools and scripts. Tools Needed for Automated Performance Testing are + Jenkins + TravisCI + TeamCity + CircleCI + Code Ship + Gitlab CI
- 34. PROPRIETARY & CONFIDENTIAL 34 ORCHESTRATION TOOLS We need to create test environments, some environment for running performance testing and also some environment to keeping data and reports. These environments should be created. Some of the well-know Orchestration Tools are + Docker Compose + Kubernetes + Prometheus + Mesosphere DC/OS + And others
- 35. PROPRIETARY & CONFIDENTIAL 35 MONITORING TOOLS We need to track, measure, and analyze the performance of their software applications, servers, networks, and other IT infrastructure components Some of the well-know Orchestration Tools are + New Relic + Dynatrace + AppDynamics + Datadog + Nagios + Zabbix + Prometheus + Grafana
- 36. 36 BENEFITS 05
- 37. When the benefits of continuous testing composed of the benefits of performance testing it becomes more delicious :) Some benefits of automated performance testing are + Quick feedback - shift-left approach + Giving current performance of the app + Creating awareness about the performance of the system + Initiate performance based development + Preventing last minute mistake + Giving history about the performance of the system + Making it easy to run performance testing on demand Image: https://images.app.goo.gl/TVy8DzqnVN4NhCHA8 BENEFITS
- 38. PROPRIETARY & CONFIDENTIAL 38 THANKS! Questions? References: + https://www.software.broadcom.com/hubfs/ESD_FY22/DevOps/ Continuous-Performance-Testing-White-Paper_BC-Agile-2022_ CE-2377_v5B.pdf + https://www.gocd.org/2016/06/08/Add-performance-testing-to-y our-Continuous-Delivery-pipeline/ + https://www.anztb.org/userfiles/files/PerformanceTestingQLD041 4.pdf + https://www.theseus.fi/bitstream/handle/10024/110354/Joonas_M oilanen_Mervi_Jeskanen.pdf?sequence=1 + The Art of Software Testing, http://barbie.uta.edu/~mehra/Book1_The%20Art%20of%20Softw are%20Testing.pdf + K6 manual, https://docs.k6.io/docs/welcome + https://github.com/gunesmes/testhive-django-unittest-behave-se lenium-newman-locust-jenkins + http://www.testrisk.com/2015/09/performance-testing-on-ci-locus t-is.html + http://www.testrisk.com/2016/01/performance-testing-on-ci-integr ation.html Connect Me on LinkedIn Download Presentation