Why Load Testing from the Cloud Doesn't Work
1 like628 views
The document discusses the challenges of load testing websites from the cloud, highlighting that traditional methods often fail to reflect real user experiences and performance variations. It emphasizes the importance of testing various components of the web application delivery chain, including third-party services and different geographic locations, to accurately assess website availability and response times. Case studies illustrate that relying solely on cloud testing can result in misleading data about actual user experiences, suggesting a need for comprehensive testing that includes last-mile performance and browser-specific response times.
Why Load Testing from the Cloud Doesn't Work
- 1. Why Load Testing your Website from the Cloud Doesn’t Work Imad Mouline - CTO, Compuware Gomez
- 2. Customer Expectations Are Rising For Web & Mobile Applications 47% of consumers expect a Web 58% of mobile phone page to load in 2 seconds or less users expect websites to load almost as quickly or faster on their mobile phone than their PC
- 3. The Impact of Web Performance on Page Abandonment 10 Performance improvement 5 (seconds) 6 secs 8 secs 2 secs 4 secs 0 -5 -8% -10 -25% -15 Percentage -33% change in -20 -38% page abandonment -25 -30 -35 -40 Source: Gomez 2010 Study of 500 Million End-User Interactions Across 200+ Web Sites
- 4. The Browser Is Becoming THE Integration Platform Number of hosts accessed directly by the browser, per user transaction, averaged across 3,000 companies 8.87
- 5. The Challenge Of Delivering Quality Web Experiences Systems …user is management NOT happy tools: “OK” The Web Application Delivery Chain 3rd Party/ Browsers Load Cloud Services Local ISP and devices Users Balancers • Inconsistent geo performance • Poorly Web • Bad performance under load • Network peering performing Servers • Blocking content delivery problems JavaScript • Incorrect geo-targeted content • Bandwidth • Inconsistent Mobile throttling CSS rendering Components • Inconsistent • Browser/device • Configuration • Network peering connectivity incompatibility App errors Servers problems • Page size too • Application • Outages Internet big DB design issues Major • Network resource • Conflicting Servers • Code defects shortage HTML tag ISP • Faulty content support • Insufficient Storage infrastructure transcoding • Too many • SMS routing / objects Mainframe latency issues • Content not • Configuration issues optimized for Network • Oversubscribed POP Content Delivery Mobile Networks Carrier device • Poor routing optimization • Low cache hit • Low cache hit rate rate
- 6. Pinpointing Problems and Optimizing Performance The Web Application Delivery Chain 3rd Party/ Browsers Load Cloud Services Local ISP and devices Users Balancers Web Servers Is it my Mobile Components Is it an Is it a Is it a App Servers data ISP or the 3rd party Internet browser or DB center? Internet? Major provider? device? Servers ISP Storage Mainframe Network Content Delivery Mobile Networks Carrier
- 7. Test Across the Entire Web Application Delivery Chain The Web Application Delivery Chain Load Testing 2.0 (Gomez) Load Testing 1.5 Load Testing 1.0 3rd Party/ Browsers Load Cloud Services Local ISP and devices Users Balancers Web Servers Mobile Components App Servers Internet DB Major Servers ISP Storage Mainframe Network Content Delivery Mobile Networks Carrier
- 8. Load Testing 1.0 Works… Some of the Time Company: Online presence for a popular TV show • Following episodes of the TV show the web site sees high traffic spikes • Goal was to achieve 1500 logins per minute • Load tested DB to improve performance in anticipation of another traffic spike 3rd Party/ Browsers Load Cloud Services Local ISP and devices Users Balancers Web Servers Mobile Components App Servers Internet DB Major Servers ISP Storage Mainframe Network Content Delivery Mobile Networks Carrier
- 9. Application Bottleneck Causes Response Time Issue • As users were added, the response time of step 3 (the login) climbed immediately • The test bottlenecked at 160 logins per minute (Goal 1500) • But quickly dropped off as users received server errors • New login query was not optimized and was bottlenecking the database servers’ CPUs
- 10. Application Bottleneck – Re-test •After tuning- application performance improved. •New Bottleneck occurred 1300 logins per minute. •Bandwidth limit reached at 90 Mbps Summary: •Problem found inside firewall •Fixes made for application issue •Retest shows second issue-bandwidth 1.0 1.5 2.0 •First test •Second test 1.0 1.5 2.0
- 11. Load Testing from the cloud misses the point… end users matter Company: Online Gaming Site Testing a new rollout in support of a new sports season • Support anticipated traffic increases • Load tested something using cloud and Last Mile to validate performance for real users in new geographies. 3rd Party/ Browsers Load Cloud Services Local ISP and devices Users Balancers Web Servers Mobile Components App Servers Internet DB Major Servers ISP Storage Mainframe Network Content Delivery Mobile Networks Carrier
- 12. View from the Cloud • First 20 minutes Cloud testing shows acceptable performance • After 2500 users Response time climbs Availability drops Error rate climbs
- 13. View from the Last Mile • Last Mile shows different story • Availability is terrible even at minimal load for real users Summary: Cloud-only testing may give misleading availability data Cloud starts with 100% availability Less than 25% for the Last Mile 1.0 1.5 2.0
- 14. Difference in browsers • Varied commercial browsers show different response times • Response time change as a function of load is dependent on the browser. Summary: Understanding response time under load requires the use of multiple browsers 1.0 1.5 2.0
- 15. Load testing from the cloud ignores real locations Company: Regional Online News Source • Began testing for the election season • Goal was to validate overall performance focusing in 2 key regions 3rd Party/ Browsers and Load Cloud Services Local ISP devices Users Balancers Web Servers Mobile Components App Internet Servers Major DB ISP Servers Storage Mainframe Content Delivery Mobile Network Networks Carrier
- 16. No Performance Issues Detected From Data-Center Increase and hold load and not exceed response times of 4 seconds and Success Rate of 99% There was only 1 page error and 11 errors total out of 60000+ transactions Page response times stayed under 4 seconds, outside of one brief blip 1.0 or 1.5 load testing shows tests passed
- 17. Last Mile Case Study: Primary Geographies Key geographies for this customer are New York and Pennsylvania. The response time never met the 4 second average goal Summary: Availability was Less Last Mile shows goal not than 99% reached Cloud can’t detect the end user issue 1.0 1.5 2.0
- 18. Don’t trust third parties to perform…test the whole WADC Company: Online Retailer • Several 3rd Parties now involved in serving up key content • Goal was to validate performance of entire application 3rd Party/ Browsers and Load Cloud Services Local ISP devices Users Balancers Web Servers Mobile Components App Internet Servers Major DB ISP Servers Storage Mainframe Content Delivery Mobile Network Networks Carrier
- 19. Understand 3rd Party Performance Under Load Load Test Response Time Per Page During the course of a Load Test, Page 3 is identified as the culprit Page Level Drill Down shows performance by Host Summary: Host & Object Level Trending 3rd party vendor did not • As the Load test have capacity progressed, the 3rd Only component to fail party search tool contributed to over 90% of the response 1.0 1.5 2.0 time
- 20. The Internet is global – where your customers are matters Company: International Hotel chain • New reservations system rollout • New global server load balancing rolled out across multiple data centers • Validate that system works globally 3rd Party/ Browsers and Load Cloud Services Local ISP devices Users Balancers Web Servers Mobile Components App Internet Servers Major DB ISP Servers Storage Mainframe Content Delivery Mobile Network Networks Carrier
- 21. Major Hotel Reservation System unavailable in 4 countries 0% availability in UK, Germany, Japan 99%+ availability in US, Canada, France Summary: Internal U.S. test looked good Distributed testing fails in key locations. 1.0 1.5 2.0 ?
- 22. Load Testing 2.0 shows you what your customer sees Company: eRetailer fashion • 100% virtual store • Daily sales spike driving 90% of revenue stream 3rd Party/ Browsers and Load Cloud Services Local ISP devices Users Balancers Web Servers Mobile Components App Internet Servers Major DB ISP Servers Storage Mainframe Content Delivery Mobile Network Networks Carrier
- 23. Load Testing with multiple browsers shows discrepancies Availability vastly different between browsers
- 24. Comparison of Performance across the country - Firefox Using Firefox browser – shows 100% availability for website Wide variations in response time based on geography
- 25. Comparison of Performance across the country – IE IE Browser : shows under 12 percent availability Availability and performance tied to geography
- 26. Page Element Downloads: IE Versus Firefox- Order Varies Explanation Summary: •Third party ad provider Only real locations can show modifying the DOM what really happens. •Depending on the load order of the third party the java 1.0 1.5 2.0 script in the ad would overwrite the DOM but only on IE
- 27. Gomez Load Testing: Most Accurate Load Test for User Experience Load Test 1.0 Load Test 1.5 Load Test 2.0 HTTP : Behind the HTTP : Data Centers Browser : Data Real World Firewall Centers Desktops Last Mile Traditional Client/ Datacenter Testing Server Test Only Gomez Spans Accuracy of End-User Incomplete Incomplete Indicative Most Accurate Response Time Accuracy of Invalid Indicative Indicative Most Accurate Application Availability Ability to drive large Yes-requires Best Better Good load volume substantial hardware Understand CDN No Misleading Misleading Most Impact Accurate Understand 3rd Party No Minimal Some Most (ads, feeds, etc…) Accurate Realistic object No No Yes Yes download Static Only Visibility behind the Best Good Good Good firewall
- 28. Questions Gomez Customers Enjoy Measurable Benefits Increased conversions 10% Reduced homepage load time from 11.3 seconds to 3.4 seconds Improved page load times 23% Saved 50%+ in staff and fees Reduced seven-step transaction time by 50% Reduced downtime 45% Achieved under 3 second response time and 99%+ availability Validated decision to consolidate three data centers For more information visit Gomez.com or contact us at +1 781.778.2700