Testing Mobile App Performance
- 2. ! ! !!!!! ! !!!! Brad Stoner is a senior performance engineer at Neotys. Previously, he managed the load and performance team at H&R Block where he spent seven years leading his five-person team in the pursuit of improved application performance and quality. Brad and his team managed the performance testing process for more than fifty projects annually. He founded Sandbreak Digital Solutions, a consulting company specializing in web application performance testing, web page optimization, front end optimization, capacity testing, infrastructure validation, and cloud testing. In his twelve years of IT experience, Brad has held roles from systems engineering to operations management. Brad Stoner Neotys
- 3. www.neotys.com© 2015 Neotys. All Rights Reserved. Ensuring the Performance of Mobile Apps On Every Device and Network… StarCANADA 2015 Brad Stoner, Neotys USA
- 4. www.neotys.com© 2015 Neotys. All Rights Reserved. BIOGRAPHY NEOTYS USA Senior Performance Engineer H&R Block IT Manager – Load and Performance Team Sandbreak Digital Solutions LLC Former Founder/Principal Consultant Webinars on neotys.com
- 5. www.neotys.com© 2015 Neotys. All Rights Reserved. OBJECTIVES Review of the Problem Why is performance testing necessary? How does it apply mobile? Types of Mobile Testing How they relate to performance Mobile Testing Challenges Overview of Performance Testing Mobile specifics Suggested Approaches
- 6. www.neotys.com© 2015 Neotys. All Rights Reserved. PERFORMANCE MATTERS! From a Nice-to-Have to a MUST-Have 1 Second Could Cost Amazon $1.6 Billion In Sales US Online Businesses = $1.7 trillion in 2013 100K$ TV ADS WEBSITE CRASHES! The Obamacare Website Failed For The Same Reason The Soviet Union Did Failures Obamacare Apps Bank of America Facebook IPO Poor Performance = Lost Revenues Meanwhile… Entertainment, video & mobile usage are exploding! Website = Brand Secure Marketing Investments
- 7. www.neotys.com© 2015 Neotys. All Rights Reserved. MARKET DRIVERS: BUSINESS Web and Mobile Applications are Business Critical Organizations compete via the internet Software and Brand are synonymous Technology usage growing Failure and poor performance are catastrophic and costly Time To Market pressure increasing End user expectations are rising Investments in Software Quality are Up Load testing is a must-have Time pressure on dev & tests teams 18% 23% 28% CY 2012 CY 2013 CY 2014 % IT BUDGET FOR APP QUALITY Source: World Quality Report 2013-2014 0% 40% 20%
- 8. www.neotys.com© 2015 Neotys. All Rights Reserved. MOBILE IS GROWING! Unprecedented User Expectations Exploding Expansion of Devices • 1.2 billion new smartphones over the next 5 years • Smartphones over 40% of all handset shipments • 85% of users expect mobile app to load at least as quickly as desktop sites • 25% of users abandoned a mobile app after 3 second delay
- 9. www.neotys.com© 2015 Neotys. All Rights Reserved. PERFORMANCE PERSPECTIVE How can we tell where the problem is? Combinations of functional and performance testing but… “End User Experience” • Time between a client initiating a transaction and the time the result is rendered back in the device • Stand-alone application - EUE is determined completely on the device • “Other” application: - EUE can be measured on device but is sum of: - Round trip time from device to server and back - Device-side rendering Does it matter where the problem is? (device, server, network, etc.) • To the business…maybe not. • To you…maybe (probably) yes!!
- 10. www.neotys.com© 2015 Neotys. All Rights Reserved. CHALLENGES/CONSIDERATIONS – DEVICE TYPE DEVICE CHALLENGES Operating Systems App Technologies Screen Sizes iOS Android Windows Design of what content is above/below the fold When that content gets delivered Different content for different devices Web SocketHTML 5 Streaming Video/Audio Google SPDY
- 11. www.neotys.com© 2015 Neotys. All Rights Reserved. MOBILE APPLICATION TYPES WEB Browser-Based NATIVE Client Downloaded From App Store or Google Play • Usually connected to the internet - If not, “Standalone” • Generally faster (client-side) than browser app • Often a different application site than if accessed via desktop browser HYBRID Combo of Web and Native • Usually wrappers for web clients
- 12. www.neotys.com© 2015 Neotys. All Rights Reserved. TYPES OF MOBILE TESTING PLATFORM TESTING LOAD TESTING FUNCTIONAL TESTING FEATURES DEVICE PERFORMANCE • Does the app do what it’s expected to do? How does the app work on this device? • Supported? • CPU, Battery consumption How fast is it displayed? • Speed of use • Can 1000 users use this app? • What is the user experience under load? • What about 500 users on 3G and 500 on 4G LTE?
- 13. www.neotys.com© 2015 Neotys. All Rights Reserved. TESTING STANDALONE APPLICATIONS Functional / Platform Testing Will Suffice • All logic on the device itself • No back-end server Metrics to Correlate • Custom Timers • Device “Vitals” (CPU, Battery, Throughput) Automated or Manual? • Hard to do it manually for all different configurations!! • How to get Vitals manually Solution Requirements • Consider an automation tool / vendor that allows: - ON DEVICE testing - Multi-device testing - Vitals
- 14. www.neotys.com© 2015 Neotys. All Rights Reserved. TESTING NATIVE, WEB, OR HYBRID APPLICATIONS Remember: EUE = device-side rendering + server-side performance How do we test each? DEVICE-SIDE Performance impacted by concurrent user load Similar to browser-based apps but with nuances SERVER-SIDE Essentially a stand-alone application Measurements made similar to prior slide However performance impacted by server Have To Separate Device-Side From Server-Side How do we correlate the two?
- 15. www.neotys.com© 2015 Neotys. All Rights Reserved. LOAD AND PERFORMANCE TESTING – OVERVIEW What? • The process of putting demand on a system or device and measuring its response • Non-functional • Bridges, roller coasters, elevators… • Software applications: Generating network traffic • (Impossible to do manually!!) Why? • To avoid: o Loss of revenue o Customer “migration” o Decreased productivity • To understand an application’s o Scalability o Stability
- 16. www.neotys.com© 2015 Neotys. All Rights Reserved. LOAD / PERFORMANCE TESTING SOLUTIONS 1 2 3 4 HOW THEY WORK Record Usage Scenarios Measure Response Times Generate the Network Load Analyze Results - Network traffic between client and server - “Models” of real-world activity - Round-trip times from client to server and back
- 17. www.neotys.com© 2015 Neotys. All Rights Reserved. LOAD / PERFORMANCE TESTING SOLUTIONS Cloud Load Generators Controller On-Premise Load Generators Firewall Application Server Database Server LOAD TESTING SOLUTION APPLICATION INFRASTRUCTURE LOAD
- 18. www.neotys.com© 2015 Neotys. All Rights Reserved. LOAD / PERFORMANCE TESTING SOLUTIONS Cloud Load Generators Controller On-Premise Load Generators Firewall Application Server Database Server LOAD TESTING SOLUTION APPLICATION INFRASTRUCTURE LOAD Monitoring Engine Monitoring
- 19. www.neotys.com© 2015 Neotys. All Rights Reserved. DEVICE CHALLENGES FOR PERFORMANCE TESTING Recording the Application • Type of app - Native apps - Web apps through mobile browsers - Hybrid apps • Technologies used in the app - HTML 5 - Web Socket - Streaming Video/audio - Google SPDY Replay scenarios for specific devices and browsers • Different content is delivered depending on the specific device or browser • Different browsers and devices have different numbers of parallel connections that affect performance
- 20. www.neotys.com© 2015 Neotys. All Rights Reserved. CHALLENGES/CONSIDERATIONS – NETWORK CONDITIONS Bandwidth • How are users with different download and upload capacities experiencing performance on the app? Latency • How are network delays affecting the performance of the app? Packet Loss • How are users experiencing the app when switching cell towers, traveling, losing signal, etc. SHOULD WE EMULATE THESE CONDITIONS? YES!!!
- 21. www.neotys.com© 2015 Neotys. All Rights Reserved. LATENCY: THE REAL BOTTLENECK Bandwidth Isn’t Everything • Benefits of increasing bandwidth over HTTP caps out around 5 Mbps • Increasing bandwidth from 5 to 10 Mbps decreases page load time by only about 5% Latency is Worse for Mobile • In 2012, the average round trip time to Google was ~100ms worldwide and ~50- 60ms within the US • On mobile, RTT is around 100-1000ms • If a device has been idle and requires the radio to turn on first, add 1000-2000ms to that time Benefit of Adding an Extra Mbps Bandwidth PercentImprovement 20 0 10 30 40
- 22. www.neotys.com© 2015 Neotys. All Rights Reserved. PACKET LOSS INFLUENCE Packet Loss Consequences Packet retransmission End user experience degraded because packet loss increases jitter Errors could occur if the packet loss is in pure data transmission Packet Loss Inevitable Mobility of the user Quickly switching to the next cell tower Temporary lose the signal due to congestion of the current cell
- 23. www.neotys.com© 2015 Neotys. All Rights Reserved. CHALLENGES/CONSIDERATIONS - HIGH, DISTRIBUTED LOAD High Number of Users • Mobile applications usually have large user bases • High user growth creates need for highly scalable load generation infrastructure Geographical Distributed Load • Users are accessing apps from remote locations • Distance and location greatly affect performance Mix of Web & Mobile Users • Many mobile apps have a “partnered” web app with a shared backend • High load on the web app can affect performance on mobile TIP: USE CLOUD-BASED LOAD GENERATION
- 24. www.neotys.com© 2015 Neotys. All Rights Reserved. MOBILE PERFORMANCE TESTING – SOLUTION REQUIREMENTS Record from Any Device • From real devices and emulators • Native or browser-based applications Simulate Any Device or Browser • Ensures the right content is delivered for that device or browser • Simulates parallel connections for each specific device and browser Emulate Realistic Network Conditions • Bandwidth • Latency • Packet Loss Generate Load from the Cloud • Geographic realism • Test full delivery chain Load Command 3GH+ 4G WiFi APPMonitor Network Emulation & Device Simulation Cloud Load Generators On-premise Load Generators Database ServerApp Server Web Server
- 25. www.neotys.com© 2015 Neotys. All Rights Reserved. WHAT HAPPENED TO EUE? The Problem: • Performance solutions don’t provide on-device measures • Functional/platform solutions don’t separate round-trip or server-side measures HOLY GRAIL SOLUTION COMBINE THE TWO! (Manual stop watch approach not feasible) Automated Functional/Platform Automated Performance On-device measures Device Vitals “Round-Trip” measures INTEGRATED FOR RESULTS CORRELATION
- 26. www.neotys.com© 2015 Neotys. All Rights Reserved. LOAD + FUNCTIONAL TESTING • Automatically trigger functional test with the start of a load test • See real device rendering times and vitals while the backend of the application is under load • Results correlated for “end-to-end” Load 3GH+ 4G WiFi APPMonitor Cloud LGsOn-premise LGs Database ServerApp Server Web Server Controller Real Devices & Networks Functional Simulated Devices w/ Network Emulation Real Device Metrics 4G 3GH+ Real Device Metrics Under Load
- 27. www.neotys.com© 2015 Neotys. All Rights Reserved. COMPLETE MOBILE EUE Real Device Users Virtual Users Load Real Device Testing Tool Load Testing Tool Application Server Load End User Experience Round-Trip Time 00:00:00 00:00:30 00:01:00 00:01:30
- 28. www.neotys.com© 2015 Neotys. All Rights Reserved. SUMMARY Mobile devices are everywhere. Application Performance is critical. Performance usually means “End User Experience” Test with a combo of AUTOMATED & INTEGRATED solutions MEASURE & CORRELATE
- 29. www.neotys.com© 2015 Neotys. All Rights Reserved. QUESTIONS? Feel free to contact me: Email: brad.stoner@neotys.com