Response time difference analysis of performance testing tools
- 1. Performance testing : Analyzing Differences of Response Time between Performance Testing Tools By Spoorthi Sham 1PI14SSE12 12-05-2015 1CSPA Seminar
- 2. Performance Testing • Performance testing is the process of determining the speed or effectiveness of a computer, network, software program or device. • This process can involve quantitative tests done in a lab, such as measuring the response time or the number of MIPS (millions of instructions per second) at which a system functions. 12-05-2015 2CSPA Seminar
- 3. How is it performed? Tools used.. 12-05-2015 3CSPA Seminar
- 4. Issues with the tools • Several issues have been observed related to tools when conducting performance testing: tools compatibility with the software under test tools installation tools setup tools flexibility in doing test both for client and server side response time generated by the tools To demonstrate and prove that response time from different performance testing tools is different To suggest potential reasons or root cause behind response time differences To answer the question: “Why do different performance testing tools produce different response time?” Research Focus 12-05-2015 4CSPA Seminar
- 5. • Most previous work on performance testing tools comparison ignored on different result reported by each tools • There is no work so far to understand why they are different against tools • Each tool claims they are better than the others but none able to justify the performance testing results against the real world. Related Works 12-05-2015 5CSPA Seminar
- 6. Tool A •Open source tool purely developed on Java platform •Sits as desktop-based tool •Serves functional, load and stress testing which is extensible to write own test to suit the scenario. •Can simulate heavy load on the application, server and even the network •Able to give instant visual feedback and capable to do load and stress testing via distributed approach •Supports protocols such as HTTP, JMS, JDBC, FTP, SOAP as well as LDAP •Used across platforms and supports full multithreading framework •Allows caching and offline analysis with replaying of test results Tool B • Open source load testing tool • Developed using C++ language • Can perform heavy load tests using scripted HTTP and HTTPS • Feature-rich GUI-based web server benchmarking tool • Only runs on Windows- based platform • Performance scripts are recorded using own proprietary language • Support custom functions, variable scopes, and random or sequential lists Tool C • Proprietary tool (one of the established performance tools in the market) • Built on Eclipse and Java • Offers automated performance testing for web and served based application • Can be used across platforms (Windows, UNIX and Linux) • Capable to create code free tests, automate test data variation and enables insertion of custom Java code for flexible test customization. • Supports operating systems such as Windows, Linux, and z/OS Overview of Performance Testing Tools 12-05-2015 6CSPA Seminar
- 7. Test Environment Setup Hardware Specification (Both Machines) CPU/processor : Intel Pentium D 3.4 GHz RAM/memory : 2 GB HDD storage : 80 GB Network Card : Integrated 10/100/1000 Ethernet Server machine Operating system : Windows Server 2003 Enterprise Edition SP1 Java JDK : JDK 1.6.0 update 21 Web server : Internet Information Services 6 HTML page size : 65.8 KB (Page: 7 KB; Image 1: 25.2 KB; Image 2: 33.6 KB) Client machine Operating system : Windows XP SP2 Java JDK : JDK 1.6.0 update 21 Tool : Tool A (open source); Tool B (open source); Tool C (proprietary) 12-05-2015 7CSPA Seminar
- 8. Result of Tool A Findings 12-05-2015 8CSPA Seminar
- 9. Result of Tool B Findings 12-05-2015 9CSPA Seminar
- 10. Result of Tool C Findings 12-05-2015 10CSPA Seminar
- 11. Comparison of Performance Test Results between Tools Findings 12-05-2015 11CSPA Seminar
- 12. Comparison of Performance Test Results between Tools Findings 12-05-2015 12CSPA Seminar
- 13. Comparison of Performance Test Results between Tools Findings 12-05-2015 13CSPA Seminar
- 14. Potential Reasons for Response Time Differences • Some fundamental reasons: capturing and simulating the load used for the performance test method of calculating metrics gathered by each tool language to develop the tools architecture of the respective tools • Architecture differs greatly: Tool A and C developed by using Java and they require JVM to run so the value setting for Java Heap Size plays a role to generate the best user load without putting extra burden to the client Tool B architecture relies on web relay daemon facility allowing CORBA- based communication to be transmitted between machines during executing the performance test Findings 12-05-2015 14CSPA Seminar
- 15. Conclusion • Different performance testing tools do give a different response time. • Currently, there is no tool able to tell us if application is fast enough in term of user experience in a reality • It is crucial for performance testers to understand that there is no tool that is able to automate and tell us about the full picture of the application's performance that is going to be in a real world • It is back to human brain to analyze the information given and performance testing tools is just one of the way that can be used to achieve that. 12-05-2015 15CSPA Seminar
- 16. Thank You 12-05-2015 CSPA Seminar 16