Testing Experience Magazine Vol.12 Dec 2010
0 likes1,088 views
This document compares the response time accuracy of three performance testing tools: Tool A (open-source), Tool B (open-source), and Tool C (proprietary). A test was conducted loading static web pages on a server from 1 to 500 concurrent users using each tool. The response times and resource utilization on both client and server machines were recorded. Preliminary results found Tool A had the most consistent response times while Tool B had higher response times that increased with load. The full results are listed to compare the performance of each tool.
Testing Experience Magazine Vol.12 Dec 2010
- 1. ISSN 1866-5705 www.testingexperience.com free digital version print version 8,00 € printed in Germany 12 The Magazine for Professional Testers Open Source Tools December 2010 © diego cervo - Fotolia.com
- 2. Response Time Accuracy: open-source or Proprietary Performance Test Tool by Muhammad Dhiauddin Mohamed Suffian, Vivek Kumar & Redzuan Abdullah Performance testing is a critical test approach in achieving the it is important to demonstrate that there are differences in re- full coverage of software product testing, apart from functiona- sponse time between various performance testing tools and to lity and other non-functional aspects. Performance testing will understand the tools’ behaviors including their influence on the determine that the software under test is capable of performing machine resource utilization. We started off by planning the cor- within an acceptable time according to the end-users’ expecta- rect test setup for the study. tions. Our experiences in performance testing, particularly in the load testing of various types of software, ranging from mobile The test setup involved two identical machines that serve as cli- applications, desktop, web-based to cloud-based applications ent with performance testing tools installed and a server which and using different performance testing tools, have triggered us hosts the static web pages. These machines are connected via PC- to ask which tool actually gives the most accurate results. The to-PC LAN cable (commonly referred as cross cable). The reason answer is not simple, and it is not that the proprietary tool always for connecting it via cross cable is to minimize network factors gives the right and accurate response times compared to open- from influencing the performance test results. The diagram for source tools, or vice-versa. Thus, before answering the question, the setup is as follows: Figure 1: Test setup for comparison of performance testing tools As shown in the diagram, only three performance testing tools cookies. Running processes on this machine were kept to a mi- were selected and installed on the client machine. For the scope nimum, i.e. unnecessary processes and server-side logging were of the study, the tools are only referred as Tool A, Tool B and Tool disabled. Load testing was performed from 1 user access and in- C. While the test was running for a particular tool, the services creased to 100, 200, 300, 400 and 500 concurrent users with zero for the other remaining tools were stopped or disabled. As for ramped-up time. The most important thing for this experiment is the server, it only hosted static HTML pages with no cache and to reboot both the client and server machines for each test. 58 The Magazine for Professional Testers www.testingexperience.com
- 3. Server machine From the technical perspective, the hardware specification for both machines was as follows: Operating system: Windows Server 2003 Enterprise Edition SP1 CPU/processor: Intel Pentium D 3.4 GHz Java JDK: JDK 1.6.0 update 21 RAM/memory: 2 GB Web server: Internet Information Services 6 Hard disk storage: 80 GB HTML page size: 65.8 KB (page: 7 KB; image 1: 25.2 KB; image The configuration of the software specification was as follows: 2: 33.6 KB) Client machine The test was started with Tool A and with one simulating user ac- cessing the HTML page at the server machine using cross cable. At Operating system: Windows XP SP2 the same time, we recorded CPU, memory and disk I/O utilization Java JDK: JDK 1.6.0 update 21 for both client and server machines. The same approach was done Tool: Tool A (open-source); Tool B (open-source); Tool C (pro- three times per virtual user set until the number reached 500 vir- prietary) tual users. The same kinds of load test were performed against Tool B and Tool C. The metrics collected from this exercise were response time, average CPU utilization, memory utilization and disk I/O utilization. The details of the results obtained from this activity are listed below: a) Tool A No of Con- Response Time Resource Utilization (Client) Resource Utilization (Server) current Users (seconds) CPU Memory Disk I/O CPU Memory Disk I/O 1 0.103 0% 3% 22% 21% 22% 0% 1 0.100 0% 3% 22% 23% 22% 0% 1 0.103 0% 3% 22% 20% 22% 0% 100 0.258 6% 3% 22% 65% 23% 0% 100 0.29 5% 3% 22% 65% 23% 0% 100 0.231 1% 3% 22% 67% 22% 0% 200 0.198 7% 3% 22% 71% 23% 1% 200 0.231 7% 3% 22% 83% 23% 2% 200 0.538 4% 3% 22% 93% 23% 0% 300 0.337 3% 3% 22% 94% 23% 0% 300 0.265 10% 3% 22% 95% 23% 0% 300 0.229 3% 4% 22% 97% 23% 3% 400 0.169 7% 4% 22% 93% 23% 0% 400 0.534 8% 3% 22% 92% 23% 3% 400 0.278 6% 3% 22% 95% 23% 2% 500 0.394 14% 4% 22% 96% 22% 8% 500 0.35 14% 4% 22% 97% 22% 9% 500 0.306 14% 4% 22% 95% 23% 10% b) Tool B No of Con- Response Time Resource Utilization (Client) Resource Utilization (Server) current Users (seconds) CPU Memory Disk I/O CPU Memory Disk I/O 1 0.015 0% 4% 22% 17% 7% 2% 1 0.015 0% 4% 22% 17% 7% 2% 1 0.015 0% 4% 22% 17% 7% 2% 100 1.423 1% 4% 22% 99% 6% 12% 100 1.211 3% 4% 22% 99% 6% 7% 100 1.403 1% 4% 22% 99% 6% 48% 200 3.489 3% 4% 22% 99% 6% 55% 200 4.478 4% 4% 22% 99% 7% 53% 200 5.123 4% 4% 22% 98% 6% 31% www.testingexperience.com The Magazine for Professional Testers 59
- 4. 300 6.158 7% 4% 22% 99% 6% 14% 300 7.068 5% 4% 22% 99% 7% 33% 300 5.394 2% 4% 22% 99% 6% 69% 400 8.597 3% 4% 22% 99% 6% 32% 400 8.164 10% 4% 22% 97% 6% 32% 400 8.757 3% 4% 22% 98% 6% 36% 500 11.316 5% 4% 22% 98% 6% 47% 500 11.17 5% 4% 22% 98% 7% 27% 500 8.901 8% 4% 22% 97% 6% 28% c) Tool C No of Con- Response Time Resource Utilization (Client) Resource Utilization (Server) current Users (seconds) CPU Memory Disk I/O CPU Memory Disk I/O 1 0.047 0% 3% 22% 35% 19% 3% 1 0.078 0% 3% 22% 38% 19% 3% 1 0.047 0% 3% 22% 25% 20% 3% 100 1.487 3% 3% 22% 100% 19% 59% 100 2.174 3% 3% 22% 100% 19% 14% 100 1.52 1% 3% 22% 100% 19% 22% 200 3.007 3% 3% 22% 100% 19% 27% 200 3.614 2% 3% 22% 100% 18% 15% 200 4.021 5% 3% 22% 100% 19% 38% 300 5.997 2% 3% 22% 100% 18% 22% 300 5.947 3% 3% 22% 100% 17% 10% 300 4.979 3% 3% 22% 100% 17% 18% 400 6.272 11% 3% 22% 100% 22% 15% 400 7.042 2% 3% 22% 100% 17% 6% 400 7.332 3% 3% 22% 100% 17% 27% 500 7.771 4% 4% 22% 88% 18% 7% 500 7.106 7% 4% 22% 100% 18% 24% 500 7.604 7% 4% 22% 100% 16% 10% Based on the metrics collected from the three tools, we plotted times under different user loads for three rounds of performance the graphs to compare and understand the trends of the re- testing. The graphs representing the results detailed in the previ- sponse times. The graphs focus only on comparing the response ous tables are shown below: Figure 2: Performance (Load) Test – Round 1 Figure 3: Performance (Load) Test – Round 2 Figure 4: Performance (Load) Test – Round 3 60 The Magazine for Professional Testers www.testingexperience.com
- 5. From the graphs plotted, we could observe the differences in re- ed and proved that differences do exist. In our next research ef- sponse time between the three different tools after running a per- fort we will try to answer why this is the case. Some fundamental formance (load) test for the same HTML web page. This result has understandings that are already in place concern the language achieved our research objective, which was to prove that, when with which the tools have been developed, the architecture of the using different performance test tools, the response times we get respective tools, the recording mechanism for each tool, captur- are not similar or even close to each other, especially as the num- ing and simulating the load used for the performance test, and ber of concurrent users increases. As mentioned previously, Tool the method of calculating metrics gathered by each tool. Having A and Tool B are open-source tools while Tool C is a proprietary this research in place will allow further understanding of the be- tool, which leads us to another research question: “Why do differ- havior of various performance tools and could help in selecting ent performance testing tools give different response times when the most suitable tool for different contexts of testing. testing the same web page?”. In this article, we have demonstrat- Biography Dhiauddin is currently a Senior Engi- Vivek Kumar works as a Senior Test Redzuan is currently the Director of the neer and Test Team Lead for the test Engineer at one of the most reputable test department in one of the leading department in one of the leading R&D R&D agencies in Malaysia. He has more R&D agencies in Malaysia. He has more agencies in Malaysia. He has almost than 7 years of experience in Software than 14 years of working experience 6 years of experience in the software/ QA both in manual testing and in test in software development, technical system development and software automation. During his QA/testing support and software testing in both testing / quality assurance fields. With career he has been working in various local and multi-national organizations.. a working experience in automotive, domains, such as mobile telecommuni- Redzuan has vast knowledge in various banking and research & development cations, banking, agriculture, semantic programming languages, CMMI, Six companies, he obtained his technical technology, and ERP systems. He holds Sigma, telecommunication and project and management skills from various an Engineering degree in Electronics management with international certifi- types of project. He holds an M.Sc. & Instrumentation from the Dr. B. R. cations in TOGAF, ASP, Red Hat Certified in Real Time Software Engineering Ambedkar University, Agra (Formerly Engineer (RHCE), British Standard Infor- from the Centre for Advanced Soft- Agra University, Agra). He is a Certified mation Security Auditor (BS7799) and ware Engineering (CASE®), Universiti Tester- Foundation Level (CTFL) from Six Sigma Green Belt among others. Teknologi Malaysia. He is a Certified the International Software Testing Tester - Foundation Level (CTFL) and Qualification Board (ISTQB®). He has Certified Tester Advanced Level – Test undertaken and managed various Manager (CTAL-TM) of the International projects from different domains and Software Testing Qualification Board been involved in the complete project (ISTQB®). He also has vast knowledge in life cycle starting from project kick-off CMMI®, Test Process and Methodolo- to release to the customer. Vivek has a gies as well as Software Development sound knowledge in CMMI, STLC, SDLC, Life Cycle (SDLC). He has just completed automation testing and automation his Six Sigma Green Belt project on the framework (keyword driven and data test defect prediction model. He has driven) with Selenium, HP Quick Test applied and managed various testing Professional 10 and Rational Functional strategies in his projects, including Tester. His tireless dedication to the functional, performance, security, usa- advancement of software testing in bility and compatibility testing both general and to performance testing for system test and system integration in particular is often referred to as a test. His interests are in software engi- hobby and not only a job due to the neering and software testing, particu- enjoyment he gets from his efforts. larly in performance testing and test management. www.testingexperience.com The Magazine for Professional Testers 61