Four Best Practices for Modern Performance Testing
Download as PPTX, PDF1 like5,989 views
This document discusses best practices for performance testing mobile applications. It begins by acknowledging that there is no single best practice, and that approaches need to be adapted to each situation. It then asks key questions about how mobile is different and how to adapt. The document outlines how performance is important for mobile due to high abandonment rates from poor performance. It also discusses how to assess the user and load profile, address network variability, utilize cloud-based testing, and measure device performance. The summary emphasizes adapting performance analysis and testing approaches to the mobile context.
Four Best Practices for Modern Performance Testing
- 1. © 2014 SOASTA. All rights reserved. June 8, 2015 1CONFIDENTIAL – Not for Distribution 6 5 4 3 2 1
- 2. © 2014 SOASTA. All rights reserved. June 8, 2015 2CONFIDENTIAL – Not for Distribution Lee Barnes – Founder and CTO of Utopia Solutions Tom Chavez– SOASTA Sr. Product Evangelist
- 3. © 2014 SOASTA. All rights reserved. June 8, 2015 3CONFIDENTIAL – Not for Distribution o Device/OS diversity o Network variability o Device performance o User location and volume
- 4. © Utopia Solutions Best Practices for Performance Testing Mobile Apps Lee Barnes, CTO Utopia Solutions
- 5. © Utopia Solutions Apology I don’t believe in “best practices”… I do believe in guidelines and better approaches for a given situation
- 6. © Utopia Solutions6 Key Questions How is mobile different? How can I adapt? Why is performance of mobile important?
- 7. © Utopia Solutions7 Why Should I Care? M-Commerce 66% OF SHOPPERSABANDO NTRANSACTIO NS 5 OUT OF 10 BECAUSE OF POORPERFORMAN CE 1DELA Y SECOND 7% DROP IN CONVERSIO NS equals a 2013 Mobile Commerce Insights study conducted by Jumio
- 8. © Utopia Solutions8 Still Not Convinced? 2013-2014 World Quality Report
- 9. © Utopia Solutions9 Evolution of Performance Testing 1990 2000 2010 2-Tier Client Server Web Mobile Challenges State of the Practice Testing Large Loads Complex / Integrated Systems Proprietary Protocols Skilled Resources In-Lab Testing Tool Maturation Open Source Tools Specialized Resources Cloud-based Testing
- 10. © Utopia Solutions Mobile Challenges & Adaptations
- 11. © Utopia Solutions11 Mobile Challenges – User Access Native App Mobile Site Full / Responsive Site Your users have options…
- 12. © Utopia Solutions12 Assess the User and Load Profile 1. Determine the user profile • Business processes • Native app, mobile site, full site • Preferred browser • Network conditions • Geographic location 2. Determine the load profile • Volume of users by business process and location • Volume of users by access type • Frequency / throughput Incorporate constant feedback from application logs and other monitoring solutions
- 13. © Utopia Solutions13 Mobile Challenges - Network Mobile User Connected User 3G 4G InATunnelG Network bandwidth and quality affect both end user experience and system load
- 14. © Utopia Solutions14 Assess Impact of Network Variability 1. Assess baseline performance (single user) Observe UX with real devices behind network virtualization solutions 2. Assess performance under load Put system under virtual user load Observe UX with real devices behind network virtualization solutions NV Virtual Users NV Real Devices
- 15. © Utopia Solutions15 Network Virtualization Tools Single User (Device and/or Emulator) • Apple Network Link Conditioner • Android Emulator • Charles Proxy Multiple Virtual Users • Charles Proxy
- 16. © Utopia Solutions16 Mobile Challenges – User Location and Volume A high volume of global users is difficult to reproduce in a lab…
- 17. © Utopia Solutions17 Utilize Cloud-based Testing Solutions • Achieve production level loads • Test entire infrastructure (vs. behind the firewall) • Distribute load geographically • Assess user impact on real devices
- 18. © Utopia Solutions18 Mobile Challenges – Device Performance Measuring System / Network Metrics is Not Enough… Memory CPU Battery Drain Graphics I / O
- 19. © Utopia Solutions19 Measuring Device Performance Critical device metrics: • CPU • Memory • Battery • Storage Mobile development platforms • Android DDMS • iOS Instruments Mobile test automation tools • TouchTest (SOASTA) Mobile lab / cloud solutions • deviceConnect (Mobile Labs) • Remote TestKit (NTT Resonant) Capture metrics while exercising app Include appropriate devices / configurations Incorporate automation for consistency Device Performance Measurement Solutions
- 20. © Utopia Solutions20 Summary Key takeaways… • Mobile users are not the same as connected users • Performance is more than backend / network • Push performance analysis to the left in the development cycle
- 21. © 2014 SOASTA. All rights reserved. June 8, 2015 21CONFIDENTIAL – Not for Distribution SOASTA PRODUCT DEMO
- 22. © 2014 SOASTA. All rights reserved. June 8, 2015 22CONFIDENTIAL – Not for Distribution o CI and Mobile Lab Scorecard • 2 days • Evaluate People, Process, Products, CI • TouchTest and CI POC with customer tests • Same mobile automation engineers that have implemented mobile test automation for Fortune 5 and Fortune 10 retailers
- 23. © 2014 SOASTA. All rights reserved. June 8, 2015 23CONFIDENTIAL – Not for Distribution UTOPIA o Lee Barnes o Founder and CTO o Utopia Solutions, Inc. o Email: lee.barnes@utopiasolutions.com o Twitter: twitter.com/USI_LeeBarnes o LinkedIn: linkedin.com/in/leebarnes o Blog: www.RaiseYourMQ.com SOASTA o Tom Chavez o Sr. Product Evangelist o SOASTA o Email: tchavez@soasta.com o Twitter: @TomChavez o LinkedIn: linkedin.com/in/tomchavez
Editor's Notes
- #2: Join us for this webinar that will introduce you to the latest mobile testing technology and processes implemented by Forbes Fortune 5 Companies and the Top 10 Internet Retailers, reducing time to market and giving back valuable time to your business with every test cycle. With the implementation of leading technology, people and processes, our customers have turned taxing four-week long test cycles to simple overnight automation. Give us an hour and let us show you the seven steps on the path to successful Mobile Test Automation. Topics we will cover will include: 1. Choosing the right device cloud 2. When to automate 3. How to get continuous results And four more!
- #3: Join us for this webinar that will introduce you to the latest mobile testing technology and processes implemented by Forbes Fortune 5 Companies and the Top 10 Internet Retailers, reducing time to market and giving back valuable time to your business with every test cycle. With the implementation of leading technology, people and processes, our customers have turned taxing four-week long test cycles to simple overnight automation. Give us an hour and let us show you the seven steps on the path to successful Mobile Test Automation. Topics we will cover will include: 1. Choosing the right device cloud 2. When to automate 3. How to get continuous results And four more!
- #4: #6 Know your Mobile Performance The remaining two steps deal with Mobile performance and knowing your edge. Mobile Performance will be dealt with in detail in a new Webinar by the same name on March 4th but it is important to cover some points in conjunction with these other steps. Very seldom do we have the time to get to look at how performance is affecting our mobile user experience. The intersection of real user data, mobile performance and functional testing is critical to the end user experience. Companies are using this data to create real user test cases and correlate performance to business metrics. It was recently discovered at a major retailer using our real user monitoring product ,mPulse, that the most common female user experience was directly accessing the shopping cart and checking out. It’s not hard to image why once you know the data. Women are smart shoppers, they put items in the cart to see what the final prices are then go to other sites to compare. Once they found the best price they are returning to finish the transaction. This retailer was not testing that path nor where they taking advantage by offering discounts when they returned. They are now. Testing performance with real user data is a game changer and is the topic of our next webinar on March 4th. The last step is
- #22: Mobile Readiness Scorecard Regardless of where you are on the path to automation you need people that can recommend next steps and help you create a plan to get to your goal. We have a 3 day Mobile Readiness Assessment put on by the same people that have built automation solutions for the Fortune 5 and Fortune 10 retailers. One of these customers have reduced a 4 week test cycle to an over night lights out process. It all starts with an assessment.
- #23: Mobile Readiness Scorecard Regardless of where you are on the path to automation you need people that can recommend next steps and help you create a plan to get to your goal. We have a 3 day Mobile Readiness Assessment put on by the same people that have built automation solutions for the Fortune 5 and Fortune 10 retailers. One of these customers have reduced a 4 week test cycle to an over night lights out process. It all starts with an assessment.
- #24: Mobile Readiness Scorecard Regardless of where you are on the path to automation you need people that can recommend next steps and help you create a plan to get to your goal. We have a 3 day Mobile Readiness Assessment put on by the same people that have built automation solutions for the Fortune 5 and Fortune 10 retailers. One of these customers have reduced a 4 week test cycle to an over night lights out process. It all starts with an assessment.