Why Average Response Time is not a right measure of your web application's performance Nishant Verma
1 like3,449 views
This document discusses the limitations of using average response time to measure web application performance and introduces Apdex as a better metric. It explains that average response time does not account for outliers and can be distorted by extremely high response times. Apdex calculates a satisfaction score based on thresholds for satisfied, tolerating, and frustrated users, which provides more useful information for evaluating performance and user experience compared to average response time alone. The document recommends using Apdex instead of average response time to make "go/no-go" decisions and better understand the number of users affected by performance issues.
Why Average Response Time is not a right measure of your web application's performance Nishant Verma
- 1. + Why “Average” Response Time is not a right measure of your web application's performance? By- Nishant Verma
- 2. +For the next 15 minutes n Web Application Performance Testing n Average Response Time n Solution to Average Response Time n Beyond Average Response Time n Questions
- 3. +Web Application Performance Testing
- 4. + Web Application Performance Testing How to do Design Test/ Scenarios Establish Performance Acceptance Criteria Identify Environment Execute Test Analyze Reports / Tune/ Retests What we collect Response Time Throughput CPU Utilization Memory
- 5. +Average Response Time n Response Time is defined as the time it takes for each Web page to load. n How we capture? n Hit the server with a specified number of user (could be in pattern) and then capture the response time of the server. n Refer the plot of ART on next page.
- 6. +Average Response Time 0 100 200 300 400 500 600 Page 1 (in ms) Page 2 (in ms) Page 3 (in ms)
- 7. + Problems with Average Response Time n It’s a bad way of tracking website performance. n It doesn’t indicate or hint about the outliers. n We tend to average the number of samples than the quantity itself. n It represents a distorted picture in case of exceptional scenario with extremely high response time.
- 9. + Apdex n Created by Peter Sevcik of Apdex Alliance (apdex.org). n How is it calculated? n ApdexT = (# Satisfied requests + # Tolerating requests / 2) (# Total requests) APDEX: Application Performance Index
- 10. + How it scores over Average Response Time n It help takes us Go-No-Go decision. n It’s doesn’t hide outliers. It yells the number of users affected by the problem. n It doesn’t get distorted when there are extremely high response times. n It’s a measure towards Satisfaction score and can be derived based on threshold. It’s not a fix number.
- 11. + One last thing… Load Testing Stress Testing Soak Testing Performance Tuning
- 12. + Some Useful Readings n http://apdex.org/ n http://en.wikipedia.org/wiki/Apdex n http://www.apdex.org/overview.html n https://newrelic.com/docs/site/apdex n https://newrelic.com/docs/site/apdex-dissatisfaction
Editor's Notes
- #2: Hi I am Nishant Verma and today I am here to present a lightening talk for you folks for the next 15 minutes. So let’s start: (Slide 1)Not sure how many of you have read this quote before. But I would expect you folks to relate to this quote at the end of the 15 minutes.[ENTER]How many of you have previously worked on Performance Testing? (Show of hand would be fine!) How many of you are aware of Average Response Time? (Show of hands!)All right Cool!
- #3: So at the high level agenda looks like: What it is? How we do it? What we collect?What is it? And What’s wrong with Average Response Time?Solution - What is it? How to use it? - - - - - - And then in the end we will try to take some questions if time permits.
- #4: (Slide 3)So what is Performance Testing? [Click]It’s a type of testing with focus on non-functional aspects of the application. It’s done and owned by a tester. [Click] What we typically measure is [Click] Responsiveness of the application [Click] Scalability [Click] Throughput Why we do it? [Click] To establish a baseline of application performance [Click] To determine compliance.Response time, or Responsiveness, is simply how quickly an interactive system responds to user input.Throughput is the amount of transactions produced over time during a test. It’s also expressed as the amount of capacity that a website or application can handle.
- #5: How to do:So we start with designing the test scenarios which could be a business case .Then we establish the acceptance criteria (basically defining exit criteria)Identifying a test environment, sometimes we even create a separate Performance Environment with scale down version of actual production environment.Put a tested build preferably a release candidate build. Execution- How we do it ? Typically what we do is we make a load profile which could be just bombarding the app server with What we collect: Responsiveness:Response time, or Responsiveness, is simply how quickly an interactive system responds to user input. Basically, “Throughput” is the amount of transactions produced over time during a test. It’s also expressed as the amount of capacity that a website or application can handle.
- #6: 1. Response Time is defined as the time it takes for each Web page to load.2. There are bunch of tools in market like LR which suggests to go by 90% percentile.3. What we generally do is hit the server with a specified number of user (could be in pattern) and then capture the response time of the server.
- #8: Let’s say an E-commerce application like Flipkart
- #12: We need a performance Indicator Help take the go-no-go decision. Without looking at all possible URLs. Without parsing the request log to find any outliers.Indicator of the number of users affected by the performance problem.Indicator of over all health of the website, without missing outliers.
- #14: Let’s re look at the load scenario once again. And this time comparing it with Apdex Index.
- #15: This is just a tip of iceberg.Average Response Time or Apdex Index is just the tip of the so called Performance Testing ice berg….Load Testing – To verify application behavior under normal and peak load conditions.Stress Testing-To determine or validate an application’s behavior when it is pushed beyond normal or peak load conditions.Soak Testing provides a measure of a system's stability over an extended period of time.