IDC & Gomez Webinar --Best Practices: Protect Your Online Revenue Through Web Performance Testing
2 likes929 views
The document outlines strategies for optimizing application quality and performance to enhance business effectiveness in online revenue streams. It emphasizes the importance of a coordinated quality life-cycle approach, integrating IT and business through common language and collaborative metrics, while addressing the challenges posed by modern technologies and global teams. Best practices include establishing customer experience management as part of the culture, conducting end-to-end testing from the user's perspective, and leveraging automated testing for continuous improvement.
IDC & Gomez Webinar --Best Practices: Protect Your Online Revenue Through Web Performance Testing
- 1. Best Practices Protect your Online Revenue through Web Performance Testing Melinda Carol-Ballou - Application Life-Cycle Management pp y g & IT Executive Strategies, IDC Imad Mouline - CTO Gomez CTO,
- 2. Optimizing Application Quality and Performance to Drive the Business Melinda Ballou Application Life-Cycle Management IT Executive Strategies E ti St t i September 2009 Copyright 2009 IDC. Reproduction is forbidden unless authorized. All rights reserved.
- 3. Agenda Understanding the drivers and challenges for taking a life-cycle approach to application quality and performance Coordinating quality and performance from requirements building through development and deployment Establishing process maturity and organizational strategies for successful application deployments © 2009 IDC 3
- 4. Disruptive Trends Driving Automated Software Quality Adoption (ASQ) • Evolving, flexible development paradigm with services creation will increasingly demand technology and business collaboration and drive for service quality within and outside the firewall – strong agile emergence • Changing approaches to application testing as application packages evolve to new architectures • End-user experience and business impact challenges of emerging web technologies (rich Internet), virtualization, cloud computing and Web 2.0 • Global economic competition plus local compliance initiatives across geographies demand quality, adaptability and rigor • Complex sourcing/off-shoring necessitate strong teaming and effective testing and quality management; limited infrastructure creates barriers • License and delivery mechanisms with SaaS for new models, choices, customer support, speed adoption, limit complexity ( pp , p p , p y (human and systems) y ) © 2009 IDC 4
- 5. “Quality Gap”: High Cost of Failure Poor Quality = Increased Business Risk y ($$$$$) Lost Revenue Lost Customers Increased Costs Damaged Brand Lost P d ti it L t Productivity Lower P fit L Profits © 2009 IDC 5
- 6. Two Goals of Effective IT/Business Alignment Innovation: Maximize Upside Through Technology- Enabled Business Processes New Business Value Reduced Exposure Compliance: Minimize Downside Through Risk g Management © 2009 IDC 6
- 7. Key IT Challenges: Business- Criticality Today’s applications are high-visibility, and carry a high cost of failure: cost-of-failure: Customer “self-serve” Supplier/channel integration Internal applications that automate critical business processes “Network effect”: failure in one applicationcan mean loss of service in others. “Outside-in” Testing is Increasingly Important Due to Web Application Delivery Chain Complexity © 2009 IDC 7
- 8. Performance Problems: a Leading Cause of Failure for New Applications Q. When your software fails to satisfy users, the top 3 reasons are: Flawed specification p Under-estimated resources /time required Poor performance /capacity/response time Failure of app. to evolve with business needs Inability to integrate with other software Dissatisfaction with user interface Data quality issues Flawed implementation of a correct spec Quality of code /lack of testing Inefficient or incorrect configuration 0 5 10 15 20 25 30 35 Source: IDC n=467 (% of respondents) © 2009 IDC 8
- 9. IT Business Challenges: Silos, Gaps The need for a quality life-cycle is key since G2000 organizations are split across interdependent groups: Business/users stakeholders Developers QA professionals Operational staff Increased complexity for both technology and source (offshore, etc.) drives need for coordination Must extend the Quality life cycle across phases and life-cycle groups © 2009 IDC 9
- 10. Rise of Global Teams and Businesses European UI Team Dev/QA/ Business Unit Support APAC Dev/QA/ Business Unit Support US Business Content Sponsors Content Business Analysts Dev/QA/ Support © 2009 IDC 10
- 11. Coordinating across the Life-Cycle • Coordinating requirements, testing, and operational performance are key across core emerging technologies • Slow response times for key business areas are problematic • Organizations should target quality life-cycle approaches through requirements, unit t t system i t th h i t it test, t integration and pre- ti d deployment and performance testing and change management • Leveraging test automation for synthetic transaction monitoring enables users to manage user experience with applications • As business requirements change, a cogent life-cycle approach enables adaptive testing and monitoring for flexible responses © 2009 IDC 11
- 12. Establishing Process and Org Strategies Engage Upfront Business input with requirements and QA and change activities via organizational and process strategy “Customer advocates” – attuned to business and IT – Design a test plan based on how real users interact with your applications • Where they do it – LA, Edmonton, London, Bangkok, NY, etc • What they do – Key tasks, transactions, and business processes • How they do it – Browsers galore, mobile devices • When they do it (peak times, spikes and how frequently) Better estimate effort better coverage Early requirements/test/design helps reveal design p y q g p g problems – managed change key © 2009 IDC 12
- 13. Establishing Process and Org Strategies Involve LoB & Ops in projects earlier Refine test plans – Identify key geographies and third-party components to test – Help define third-party SLAs, and overall “business SLAs” Coordinate testing of external components and services delivered from outside the datacenter Anticipate deployment and configuration issues Leverage tools, tests and skill sets across the life-cycle to ensure functionality, availability, and performance y y p goals are met © 2009 IDC 13
- 14. Application Quality – Design and Delivery Quality Evaluation Matrix •Easy to Use •Easy to Use •Unpredictable U di t bl •Behaves as B h Behavior Designed Quality of Design Customer Experience •Difficult to Use •Difficult to Use •Unpredictable •Behaves as Behavior Beha ior Designed D i d Delivery © 2009 IDC 14
- 15. App Deployment & Support: Old View DEV Define SLAs, Test/ Provision Monitor Support Tune Issues Result: Little input into specifications or development Little leverage between development/ops of testing/monitoring investments Testing/tuning LATE in the cycle! © 2009 IDC 15
- 16. Closing the Loop: Leverage Skills & Tools for Agile, Iterative Approach •Shared goals, g , •Shared metrics, •Share tests, Design tools,and tools and skills Define •Shared systems Develop Support Deploy & Test & Monitor Tune © 2009 IDC 16
- 17. Establish a Common Language “Sales says “Call Center’s “Network statistics look great.” customers can t can’t swamped with “All servers are available.” All available find product in the Order Status database.” inquiries.” “All databases are available.” “99% of our monitored apps are OK” “Order “We’re not “Service levels are within limits.” Service limits. volume is meeting our first down.” call resolution metric today.” Business IT Bridge the communication gap! Establish a common language! © 2009 IDC 17
- 18. Bridging the Customer Experience Gap “Is the end-to-end Business Management M business process working?” Customer End-to-End Usage Scenarios g Experience “Can customers Management do what they Web, Voice Web Voice, Web Services, email FAX Services email, FAX… want to do?” do? Application “Can I proactively Management fix things before they impact the Server, Network & business?” Element Management © 2009 IDC 18
- 19. Closing a ‘Customer Experience’ Gap 1. Establish a common language between IT and the business, designate a liaison 2. Agree on business process measurement priorities 3. Implement testing/management for end-to-end p g g business processes – from the customer/end-user perspective 4. Eliminate silos and close the loop between development and operations 5. S Share goals, metrics, f feedback with the entire team: make Customer Experience Management part of the culture © 2009 IDC 19
- 20. Summary •Coordinate a Quality Life-Cycle approach from requirements through to operations to enable adaptive businesses •Schisms between business, development, testers and operations must be addressed •IT groups and the business must build a common language, common metrics, and common tools and practices •Focusing on the customer experience will help IT achieve this along with core incentives to collaboration •Making “customer experience” the culture is key to help bridge the gap and to enable successful collaboration across groups © 2009 IDC 20
- 21. Best Practices: Protecting your Online Revenue BP #1: Establish a Common Language between IT and the business BP #2: Make Customer Experience Management part of the culture BP #3: Implement testing & management for end‐to‐end business processes from the customer/end processes – from the customer/end‐ user perspective
- 22. BP #1: Establish a Common Language Between IT & the Business: Customer Satisfaction Index Apdex (Application Quickly isolate and prioritize Performance Index) P f I d ) opportunities for performance ii f f Single measurement of improvement based on real user satisfaction with customer satisfaction Web performance W b f Why is customer satisfaction h f Based on response time decreasing? thresholds What do users have in common? ? How do I compare against Satisfied Tolerating Frustrated my competitors? http://www.apdex.org 2007
- 23. Customer Satisfaction Index: An Example A l di if i d As load increases customer satisfaction decreases Customer satisfaction decreases Load increases
- 24. BP #2: Make Customer Experience Management Part of your Culture ‐ Design your Web site for a Multi‐browser World Will it work across all browsers? ill i k ll b ? Functions correctly Does not function correctly = brand damage
- 25. Make Customer Experience Management Part of your Culture ‐ Test & Monitor your Key Transactions Further analysis shows sequence 0 of However sequence 1, the the transaction - the login page - is authentication step, is unaffected impacted b a major i t d by j performance issue
- 26. Make Customer Experience Management Part of your Culture – Ensure that your Applications Will Scale Under Load Obama family exclusively wears leading fashion brand for inauguration ceremony Leading fashion brand’s web application crashes leading to short and long term loss of revenue and brand equity
- 27. BP #3: Implement Testing/Management for End‐to‐end Business Process from the Customer/End‐user Perspective Behind‐the‐ …user is firewall testing NOT happy tools: “OK” Users Load Balancing Web Servers App Servers The traditional view of Web Application Delivery The traditional view of Web Application Delivery DB Servers Storage Mobile Components Traditional zone of control
- 28. Critical to Test Across Entire Web Application Delivery Chain Behind‐the‐ …user is firewall testing NOT happy tools: “OK” The Web Application Delivery Chain 3rd Party/ Browsers Cloud Services Local ISP and Devices Users Load • Inconsistent geo performance • Network peering • Poorly Balancing • Bad performance under load Bad performance under load p problems p performing g • Blocking content delivery • Bandwidth throttling JavaScript Web • Inconsistent • Browser/device Servers • Incorrect geo‐targeted content connectivity incompatibility • Configuration • Page size App • Network peering too big Servers errors problems • Application •TToo many DB design issues • Outages Internet objects • Network resource • Low cache Servers • Code defects Major shortage hit rate • Insufficient ISP • Configuration issues • Faulty content Storage infrastructure transcoding • Oversubscribed POP Mobile • Poor routing optimization SMS routing / • SMS i / Components • Low cache hit rate latency issues Content Delivery Mobile Networks Carrier Traditional zone Zone of customer expectation of control
- 29. Load Testing 2.0 Performance Testing For A Web 2.0 World Self‐service testing gives you control to test as needed Test internal & external web T ti t l& t l b application components to ensure transactions perform under load Test cloud‐based applications & Test cloud‐based applications & services SaaS based Load Testing 2.0 solutions enable on‐demand, , scalable testing without associated hardware investment & maintenance costs Combine hi h volume C bi high l Test the full web application delivery chain all the way to your application load testing end users’ desktops with broad geographic experience testing from the outside-in
- 30. Gomez Reality Load XF: On‐Demand Realistic Load Testing from Browser to Data Center Backbone Last Mile Last Mile High volume load (HTTP, Browser) Real‐world load Find infrastructure breaking points Find user experience breaking points Define capacity headroom Accurately measure response time 3rd Party/ Browsers Cloud Services Local ISP and Devices Users Load 100,000+ Balancing consumer‐ grade d Web desktops Servers 168+ App 100+ countries Servers commercial‐ grade DB nodes & Internet 2,500+ data ISPs Servers Major centers ISP Storage Major mobile Mobile carriers around Components the globe Content Delivery Mobile Networks Carrier
- 31. Ensuring Performance of All 3rd Party Components Company: Online Retailer • Several 3rd Parties now involved in serving up key content • G l was to validate performance of entire application Goal t lid t f f ti li ti 3rd Party/ y/ Browsers Cloud Services Local ISP and devices Users Load * Balancing Web Servers App Servers DB Internet Servers Major ISP Storage Mobile Components Content Delivery Mobile Networks Carrier
- 32. Response Times Rise Due To 3rd Party Object Error The load increases The transaction rate throughout the test increases and then falls off as response times climb Errors are seen, all on a 3rd party object • 3 d party h d 3rd t hardware was i insufficient f overall d ffi i t for ll demands on application d li ti • Based on SLAs 3rd party had to improve performance to get paid
- 33. Ensuring Performance in Key Markets Company: Regional Online News Source • Began testing for the 2008 election season • G l was to validate overall performance f Goal t lid t ll f focusing i 2 k regions i in key i 3rd Party/ y/ Browsers Cloud Services Local ISP and devices Users Load Balancing * Web Servers App Servers DB Internet Servers Major ISP Storage Mobile Components Content Delivery Mobile Networks Carrier
- 34. No Performance Issues Detected From Data-Center Increase and hold load and not exceed response times of 4 seconds and Success There was only Rate of 99% 1 page error and 11 errors total out of 60000+ Page response transactions times stayed under 4 seconds, outside of one brief blip By traditional test standards the test passed
- 35. Performance Issues Detected From Real User Desktops Key geographies for this customer are New York and Pennsylvania Last Mile data showing substantial number of measurements greater than 4 seconds
- 36. The Gomez Platform: Web Application Experience Management Self‐ Self‐Service SaaS Portal Web Web Load Web Web Cross‐Browser and Performance Performance Performance Testing Testing Management Business Analysis • Compatibility High volume • High‐volume 20 Real user monitoring • Real‐user monitoring Web performance • Web performance • Functionality backbone load • Last Mile monitoring analytics • Performance • Real‐world • Backbone monitoring • Benchmarks Last Mile load • Business dashboard Web 2.0, RIA, Streaming, Mobile Web 2 0 RIA Streaming Mobile Dashboards, Recording & Alerting & APIs & Education & Metrics & Analytics Provisioning Diagnostics Data Feeds Best Practices World’s Most Comprehensive Testing Network 168+ countries 2,500+ ISPs 500+ 100+ 5,000+ 100,000+ combos of commercial nodes supported consumer‐grade browsers & O/S and data centers mobile devices desktops
- 37. Gomez Customers Enjoy Measurable Benefits Increased conversions 10% Reduced homepage load time from 11.3 seconds to 3 4 seconds 11 3 d t 3.4 d Improved page load times 23% Saved 50%+ in staff and fees Reduced seven‐step transaction time by 50% y Reduced downtime 45% Achieved under 3 second Achieved under 3 second response time and 99%+ availability Validated decision to consolidate Validated decision to consolidate three data centers
- 38. Appendix
- 39. Ensuring Performance Of Login Process Company: Online presence for a popular TV show • Following episodes of the TV show the web site sees high traffic spikes • Goal was to achieve 1500 logins per minute • Load tested DB to improve performance in anticipation of another traffic spike 3rd Party/ y/ Browsers Cloud Services Local ISP and devices Users Load Balancing 1 Web Servers App Servers DB Internet Servers Major ISP Storage Mobile Components Content Delivery Mobile Networks Carrier
- 40. Application Bottleneck Causes Immediate Response Time Issue • As users were added the added, response time of step 3 (the login) climbed immediately • The test bottlenecked at 160 logins per minute (Goal 1500) • But quickly dropped off as users received server errors • New login query was not optimized and was bottlenecking the database servers’ CPU ’ CPUs