Managing Mobile App Performance Through Geo-Performance Visibility
0 likes88 views
The document discusses the importance of performance in mobile app development, highlighting that great mobile experiences require a focus on both user interface and performance throughout the development cycle. It emphasizes the need for geo-performance visibility and the use of real devices to simulate user experiences, along with advanced diagnostics and continuous monitoring for optimizing mobile app health. Case studies from companies like Telstra and Facebook illustrate the impact of performance monitoring and management on user experience.
Managing Mobile App Performance Through Geo-Performance Visibility
- 1. TESTINGMIND www.testingmind.com MANAGING MOBILE APP PERFORMANCE THROUGH GEO-PERFORMANCE VISIBILITY DR DANIEL CHAN #TAS18 Hong Kong November 7th , 2018
- 2. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong CONTENTS • Performance as a critical success factor for mobile apps • Common practice in mobile app performance assessment • Importance of geo-performance • Distributed performance assessment platform with centralised orchestration • Issue identification & classification through machine learning • Enabling agile mobile evolution 2
- 3. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong PERFORMANCE AS A CRITICAL SUCCESS FACTOR FOR MOBILE APPS • Great mobile experiences require great performance Engaging mobile experiences are half user interface, half performance • Performance must be a focus throughout the development cycle Mobile consumers often do not allow developers multiple chances • Complexity hurts performance Simplicity of architecture is mandatory to deliver great mobile performance • Offline support drives amazing performance Engineering client-side caching solutions is challenging but rewarding 3
- 4. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong PERFORMANCE IS NOT A PRIORITY FOR MOBILE DEVELOPMENT TEAMS 4 Source: Forrester’s Forrsights Developer Survey, Q1, 2013 based on North American and European mobile and mobile web app developers
- 5. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong Emulators • SDK or emulator softwares used to develop and test • Does not reproduce device conditions Tethers • Run automated scripts on devices connected to laptop • Not scalable and a very localized solution Device Cloud • Scale device testing for distributed teams • Does not reproduce real world conditions Experience Platform • Real devices, real user conditions, real cellular • Endpoint performance monitoring TEST APPROACH EVOLUTION
- 6. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong CLOUD DEVICE FARMS 6
- 7. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong VOICE OF DEVELOPERS 7 We have zero visibility into the network characteristics of our app NETWORK Real localization testing is very challenging LOCALIZATION Our app content resides almost completely in the cloud CLOUD Weak code is increasing data plan costs and network issues are draining the battery CODE QUALITY We have bandwidth issues in large parts of the world GEO-PERFORMANCE Over 50% of our poor ratings in the app store are due to network issues USER RATINGS
- 8. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong THE CHALLENGES 8 • Device diversity • Operating system diversity • Carrier networks • Apps on the device • App versions • Background processes • App clients • Client-server communication • Geo-performance visibility
- 9. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong 9 The permutations of the possible iOS versions and device types that users will access an app is fairly high. IOS FRAGMENTATION
- 10. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong ANDROID FRAGMENTATION How many mobile devices are running Android? An OpenSignal survey from 2015 puts the number of devices at 24,093 under 1,294 brands. How many to cover in your app testing is a really a question about device penetration in your target markets. 10
- 11. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong REAL DEVICES To simulate real user geo-performance exeperiences • Real devices, particularly newly released devices • Real SIM cards • Geographically distributed in different cities & countries 11
- 12. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong REMOTE CONTROL To orchestrate the remote devices centrally from a console • Provides access to devices connected to mobile networks around the world right from the web browser • Allows app testing in real world conditions - pre and post release 12
- 13. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong TEST AUTOMATION 13 Automation Frameworks
- 14. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong PERFORMANCE DIAGNOSTICS 14 To detect & categorize issues • Captures user experience issues across the CDN, servers, app, device or carriers • Identifies, isolates, and prioritizes performance fixes
- 15. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong CASE STUDY: TELSTRA CONDUCTING DRIVE TESTS TO UNDERSTAND NETWORK PERFORMANCE ON THE MOVE 15 Using mobile physical boxes allow Telstra to experience their network in a familiar real- world use case — while driving. Accurate geo locations, correlated video & audio feeds, and video & MOS quality scores allow Telstra to accurately identify areas in its network that offer poor user experience.
- 16. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong CONTINUOUS MONITORING To monitor mobile app health ◦ Monitors critical user experience KPIs ◦ Automatically provides alerts to irregularities - not just at the time of an incident but beforehand ◦ Assesses speech quality of voice enabled app ◦ Integrates with bug tracking services (e.g. JIRA, ServiceNow) 16
- 17. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong CASE STUDY: FACEBOOK ASSESSING PERFORMANCE IMPACTS OF RELEASES 17 Slows down Post Image times Slows down Search times Version 139.0.0.26.93 141.0.0.31.91 143.0.0.32.90142.0.0.29.92 Slows down Search times Slows down Search times Speeds up Post Image times Speeds up Video Playback Time Speeds up Search times
- 18. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong CASE STUDY: TELSTRA NETWORK PERFORMANCE MONITORING 18 Telstra continuously monitors real-world network performance through collecting KPI data for 25 popular apps every 10 minutes
- 19. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong ENABLING AGILE MOBILE EVOLUTION • Fast feedback is a natural complement to continuous delivery Continuous delivery & feedback from performance management tools & analytics framework define the new app development & delivery cycle • Focus on collecting 3 types pf mobile measures Focus on metrics that measure your app’s technical fitness, level of user engagement, and business results • The faster you can act on feedback, the better Feedback has a half-life: its value degrades with time, so the faster you can act on it, the better In the long run you should collect real-time user feedback on local devices and use it to predict the next best action your app should take 19
- 20. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong REAL-TIME TECHNICAL & ENGAGEMENT METRICS 20 Audience exposed To ads of sponsors Track Converted Audience Mobile Purchase Measure conversion attributed to exposed audience
- 21. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong AWS ARCHITECTURE 21 Availability Zone region Administration VPC EC2 instancesVPN gateway Security group Security group date center n Ubuntu 16.04 MacOS 10.12 date center 1 Ubuntu 16.04 MacOS 10.12 …… VPN connection Route 53 user Dedicated Customer VPC VPN gateway VPN connection VPC subnet VPC subnet Security group Unified Controller EC2 instances RDB VPN connection VPN connection
- 22. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong Thank You 22