Performance Testing: Eliminate System Outages and Save Millions
- 1. 1 © 2014 Method360, Inc. All rights reserved. Performance Testing: Eliminate System Outages and Save Millions PRESENT ED BY Sunny Hiran
- 2. 2 ©2014 Method360, Inc. All rightsreserved. Method360 Method360 PERFORMANCE TESTING Introduction SUNNY HIRAN Director Quality Assurance Through 2015, 80% of the outages impacting mission-critical services are expected to be caused by people and process issues, according to Gartner. More than half of these outages will be attributed to change, configuration, and other related issues. (Top Seven Considerations for Configuration Management for Virtual and Cloud Infrastructure). Only 22% of organizations surveyed by Gartner, however, have deployed the full complement of fault, performance, and configuration management capabilities. Performance and configuration capabilities are expected to lag through 2017 according to a recent Gartner research report. (I& O Teams Must Proactively Develop Three Core Network Management Disciplines)
- 3. 3 ©2014 Method360, Inc. All rightsreserved. Method360 Method360 The Stakes Are High In a matter of days if not hours, organizations can incur losses totaling tens of millions of dollars due to unplanned production outages that are the result of poor performance. Even if software conforms to business and functional requirements, in the absence of rigorous performance and scalability validation, credibility and competitiveness are at stake. Once lost, these are difficult to recover. PERFORMANCE TESTING
- 4. 4 ©2014 Method360, Inc. All rightsreserved. Method360 Method360 Agenda •Key Takeaways •Performance Nightmares •What is Performance Testing? •Locking down performance scope and understanding what to simulate •Tool options in the performance marketplace, including proven open source solutions •Key strategies that will ensure the responsiveness of your mission critical systems PERFORMANCE TESTING
- 5. 5 ©2014 Method360, Inc. All rightsreserved. Method360 Method360 Key Takeaways •Performance outages are preventable!! •Understand that performance outages impact all production support landscape models, both hosted and in-house solutions. (Demand that your hosted provider demonstrate how your system will perform under load & DR) •Performance failures have a real impact on organizations both in terms of credibility and lost revenue •Performance testing should be a mandatory testing activity for any mission critical solution PERFORMANCE TESTING
- 6. 6 ©2014 Method360, Inc. All rightsreserved. Method360 Method360 Performance Nightmares TWITTERGOES DOWN FOR 'MOST USERS' IN LONGEST OUTAGE SINCE IPO According to previous Status blog updates, it was the longest outage for Twitter's mobile and Web platforms in at least the past year, when most issues have been resolved in 25 minutes or less. Twitter had trouble in its early years with frequent outages and improved the stability of its service even as users and tweet volume skyrocketed, helping to convince users and investors about the viability of the microblogging service. GOOGLE APP ENGINE For four hours between10:30-14:30 EST on October 26th, 2013 Google App Engine failed to deliver about 50% of its requests. As a service used by hundreds of thousands of developers to create applications, this outage was felt heavily across the web. ROYAL BANK OF SCOTLAND The IT staff was responsible for system failures that affected 17 million customers of RBS, NatWest and Ulster Bank. The problem occurred during maintenance on systems which caused an error in their automated batch scheduler and processor. This prevented millions of customers from receiving or making payments, and lasted for more than a week! The outage cost RBS a whopping £125 million! PERFORMANCE TESTING
- 7. 7 ©2014 Method360, Inc. All rightsreserved. Method360 Method360 Performance Nightmares BANK OF AMERICA On September 14th, 2013 problems started with BofA's website with the message "some of our pages are temporarily unavailable" on the homepage. The issues only got worse. 72 hours later, the majority of users were unable to connect to Bank of America's website due to slowness and time-out failures. The problems weren’t resolved for another 24 hours BofA attributed the outages to month end traffic along with a code release. KOHL’S Last year, Kohl's ran a massive online special for Black Friday shoppers, offering early bird specials, 20% off sales prices, and free order shipping. The bargains started the day before Thanksgiving and ran until 3pm on Black Friday. However, given the surge in traffic, the Kohl's website experienced an outage for several hourson Thanksgiving evening. As the heaviest online traffic week of the year, a fewhours of downtime can be incredibly costly for online retailers. AMAZON WEB SERVICES (AWS) On October 22nd, 2013 a memory leak and failed monitoring system caused Reddit, Foursquare, Minecraft, Airbnb, Heroku, GitHub, imgur, Pocket, HipChat, Coursera and a number of others to go down. The outage lasted for six hours until service was restored. Finally, on Christmas eve, Netflix went down until Christmas morning due to an elastic load balancing failure in AWS. PERFORMANCE TESTING
- 8. 8 ©2014 Method360, Inc. All rightsreserved. Method360 Method360 What is Performance Testing and Why do it? PERFORMANCE TESTING “Performance testing is in general testing performed to determine how a system performs in terms of responsiveness and stability under a particular workload. It can also serve to investigate, measure, validate or verify other quality attributes of the system, such as scalability, reliability and resource usage.”
- 9. 9 ©2014 Method360, Inc. All rightsreserved. Method360 Method360 Performance Scope PERFORMANCE TESTING
- 10. 10 ©2014 Method360, Inc. All rightsreserved. Method360 Method360 Performance Tools & Monitoring Options Borland® Silk Performer® Web Load Performance Testing Tools MS VSTS Live Monitoring Tools PERFORMANCE TESTING
- 11. 11 ©2014 Method360, Inc. All rightsreserved. Method360 Testing the ability of the system to process required datavolumes Stressingthe system up to and beyond its breaking point Captures performance of a newsystemto a knownstandard Identify problems over a periodof timeby executing continuous repetitive transactions Exercisingfailure points and verifying backup systems can be deployed Measuring systemresponse times when measured under a specified load Performance Testing Services Load Stress Failover Baseline Endurance Volume Performance Strategies PERFORMANCE TESTING
- 12. 12 ©2014 Method360, Inc. All rightsreserved. Method360 Method360 Implementing Performance Testing •Library updates •Lessons learned and best practices •Ongoing validation and support 05_Closure •End-to-end load/ performance test executions •Monitoring, tuning & improvement •Defect Management •Test summary report 04_Test Execution •Test scripts development •Test data setup •Test environment setup •Test scenarios development 03_Test Artifacts Creation •Test strategy, plan, tools setup •Determine monitoring requirements 02_Preparation •Scope, tools, transaction distributions resourcing, feasibility analysis, LOE •Kickoff and KT sessions with business •Finalize success criteria 01_Planning PERFORMANCE TESTING
- 13. 13 ©2014 Method360, Inc. All rightsreserved. Method360 Method360 Q&A PERFORMANCE TESTING
- 14. 14 ©2014 Method360, Inc. All rightsreserved. Method360 Contact Info QUALITY ASSURANCE SUNNY HIRAN 415.535.5453 shiran@method360.com PERFORMANCE TESTING To learn more about how Method360’s solution offerings can benefit your company, please contact us:
- 15. 15 ©2014 Method360, Inc. All rightsreserved. Method360 Thank You END-TO-END SERVICES. HIGH VALUE SOLUTIONS. PERFORMANCE TESTING