Apple Edition: iOS Mobile Experience Report
5 likes3,603 views
The Crittercism Mobile Experience Benchmark report for iOS apps in September 2014 highlights the importance of app uptime and responsiveness, noting that apps should respond to user requests within one second and maintain a crash rate under 1%. It reveals that many iOS apps rely on multiple cloud services, which can affect their performance, with statistics showing significant latency and error rates in service calls. The report stresses the need for mobile app developers to optimize performance, considering geographic variations and the underlying services that apps depend on.
Apple Edition: iOS Mobile Experience Report
- 1. Crittercism Mobile Experience Benchmark – Apple Edition September 2014
- 2. Key Performance Takeaways for iOS Apps Mobile App Complexity App Responsiveness Mobile Experience Benchmark 2 Mobile app performance is challenging with over millions of permutations of variables to manage. To be competitive, your app should respond to user requests within 1 second. The reality is most apps don’t because they rely on unpredictable services. App Uptime (Crash) Your goal should be to have an app that crashes less than 1%. Most apps crash more. Geography Mobile apps should perform well in all geographies. However, location-based factors impact regional experience.
- 3. Introduction Mobile Experience Benchmark 3 iOS apps live in a complex environment, making it difficult to manage performance and ensure superior user experience. The pillars that comprise mobile app performance are: 1. App uptime 2. App responsiveness App uptime and responsiveness are also impacted by the Geography of the user. Crittercism is the leading mobile application performance management (mAPM) solution that processes over 30,000 requests per second from over 1B mobile users in real-time in over 120 countries. We’ve gathered rich iOS related performance data and are honored to share the first-ever Crittercism Mobile Experience Benchmark – Apple Edition report with the world. This report uses data from Crittercism customers actively optimizing their app performance. App performance across un-optimized applications is expected to be worse.
- 4. Mobile Performance Impacts Your Bottom Line Let’s look at how poor uptime could impact a company with $1B in annual mobile revenue*: • At 97% uptime, the company would lose $2.5M per month • At 99% uptime, the company would lose $0.8M per month • At 99.9% uptime, the company would lose $82,192 per month Mobile Experience Benchmark 4 *Examples of companies with significant mobile revenue include Starbucks which Business Insider estimates drives $1B through mobile and Domino’s which sees $700M in mobile sales according to Mobile Commerce Daily.
- 5. 5 iOS Mobile Apps Live in a Complex World
- 6. Mobile Experience Benchmark 6 Fragmentation adds Complexity Millions Devices OS versions Carriers of permutations, creating a complex ecosystem X X X = App versions
- 7. Connecting to 6+ Cloud Services Increases Complexity Average Number of Services Used By iOS App Mobile Experience Benchmark 7 Findings • The average iOS app depends on 6+ cloud services . Examples include: Facebook for logins, Amazon Web Services for storage, and Flurry for analytics • 52% use over 5 services Takeaway • iOS apps depend on many underlying cloud services, which means they not only need to be free of crashes, but the services they depend on must also work and be available
- 8. 8 Responsiveness Performance The pillars of mobile app performance are responsiveness, uptime and geography. Responsiveness of cloud services gives you insight into how much faster or slower your app is running than your benchmark or expectations. Responsiveness helps answer questions such as: • Which services have the highest error rates? • Which services are slowing down my app? This section will shed light on mobile app responsiveness and error rates by app and top services. Why this matters: iOS apps depend on many cloud services and responsiveness impacts end-user experience
- 9. Service Calls are Unpredictable Mobile Experience Benchmark 9 iOS apps depend on many services and service calls. A single app may make multiple services calls per end user interaction. Hence it is the aggregate latency and error rates across these service calls that determines the end user experience. The two metrics to focus on are error rate and latency. Users expect responses in under 1 second*, yet 5% of calls experience higher latency and 46% have over 500 ms latency. In addition, 17% of service calls have over 5% error rate and 13% have over 10% error rates. 46% of cloud services that iOS apps use have over a half second latency 17% of service calls from iOS apps have error rates over 5% *Based on data from Chimera Labs: http://chimera.labs.oreilly.com/books/1230000000545/ch10.html
- 10. Google Analytics is Best Performing Top Service Error Rates & Response Times by top iOS Services Mobile Experience Benchmark 10 Findings • Google Analytics has lowest error rate (0.1%); error rate is percent of errors per number of service calls made • Of top public services, Google Analytics has lowest average latency Takeaway • Even top services like Google Analytics and Facebook experience latency and error rates • Teams must manage every aspect of mobile app performance, including third-party services
- 11. T-Mobile is the Fastest iOS Carrier iOS Latency Rankings By Major US Carriers Mobile Experience Benchmark 11 Findings • T-Mobile is the fastest iOS carrier • Sprint and US Cellular are the slowest iOS carriers Takeaway • Responsiveness of iOS apps varies across the major carriers. Expect that your mobile team will need to adapt to performance issues driven by these carrier specific differences
- 12. 12 Geography Responsiveness and uptime are affected by the location of the end user. Factors to manage include: device and OS adoption, regional carriers and networks. A geographic performance lens helps answer questions such as: • Is my app slow in different geographies? • What is causing a poor experience for users in different countries, cities, or store locations? This section will summarize iOS app performance across countries and cities. Why this matters: Expanding apps to new geographies introduces location factors that impact user experience
- 13. Cleveland has Slowest iOS WiFi iOS WiFi Latency Rankings by Most Populous US City Mobile Experience Benchmark 13 Findings • Virginia Beach has the fastest iOS WiFi responsiveness • Cleveland has the slowest iOS WiFi responsiveness Takeaway • iOS users across US cities experience app responsiveness differently. In particular, WiFi speeds vary by city
- 14. Raleigh has Fastest iOS Carrier iOS Carrier Latency Rankings by Most Populous US City Mobile Experience Benchmark 14 Findings • Raleigh has the fastest iOS Carrier responsiveness • Kansas City has the slowest iOS Carrier responsiveness Takeaway • App owners should optimize apps and architectures (i.e. Content Delivery Networks and regional data centers) to account for geographic variance in response times
- 15. Yokohama and Tokyo are Fastest iOS WiFi Latency Rankings by Major World Cities Mobile Experience Benchmark 15 Findings • Yokohama & Tokyo have the fastest iOS WiFi responsiveness • Ho Chi Minh City has the slowest iOS WiFi responsiveness Takeaway • Be prepared to deal with regional wireless networks and latencies
- 16. Tokyo has Fastest Carrier Responsiveness iOS Carrier Latency Rankings by Major World Cities Mobile Experience Benchmark 16 Findings • Tokyo and Yokihama have the fastest iOS Carrier responsiveness • Bangkok has the slowest iOS Carrier responsiveness Takeaway • iOS users across world cities experience app responsiveness differently. Carrier speeds vary by world city • Be prepared to deal with regional carrier networks and latencies • App owners should optimize apps and architectures (i.e. Content Delivery Networks and regional data centers) to account for geographic variance in response times
- 17. France has Fastest WiFi Responsiveness iOS WiFi Latency Rankings by G20 Countries Mobile Experience Benchmark 17 Findings • France has the fastest iOS WiFi responsiveness • India has the slowest iOS WiFi responsiveness Takeaway • iOS users in different countries experience app responsiveness differently. WiFi speeds vary by world city
- 18. Japan has Fastest Carrier Responsiveness iOS Carrier Latency Rankings by G20 Countries Mobile Experience Benchmark 18 Findings • Japan has the fastest iOS Carrier responsiveness • China has the slowest iOS Carrier responsiveness Takeaway • iOS users in different countries experience app responsiveness differently. Carrier speeds vary by world city • Be prepared to deal with regional carrier networks and latencies • App owners should optimize apps and architectures (i.e. Content Delivery Networks and regional data centers) to account for geographic variance in response times
- 19. Uptime Performance In addition to responsiveness and geography, the other main culprit of app performance issues is app uptime. Mobile app uptime is defined by Crittercism as the percent of app usage that did not experience a crash. Uptime helps answer questions such as: • Which devices are causing my app to crash? • Does my app crash more on a particular OS version? This section will reveal acceptable uptime performance of mobile apps. It will also provide data across app categories and industries. Why this matters: Uptime performance is a fundamental measure of end-user experience
- 20. 46% of iOS Apps Have Over 1% Crash Rate Percent of iOS apps by crash uptime Mobile Experience Benchmark 20 Findings • 30% of apps have over a 2% crash rate • 46% of apps have over a 1% crash rate Takeaway • In the web world, the norm uptime is 99.9% • In the mobile world, however, your app should have a crash rate under 1% (uptime of 99%) • Apple lists Crashes and Bugs as the number one reason apps get rejected from the App Store 1% crash rate is based on customers actively using Crittercism. Apps in the wild can experience anywhere from 3-10% crash rate.
- 21. Apple: 4S Crashes The Most Among Top iPhones Apple OS and top device breakdown Mobile Experience Benchmark 21 Findings • iOS 7.1 & iOS 7 have 2.1% crash rates • iPhone 5s crashes least • iPad Mini crashes the most • Apps on iPhones are more stable than iPads Takeaway • Prior to iOS 8’s release, app developers have stabilized crash rates across iOS versions • Expect iPad users to experience stability issues more often than those using iPhones
- 22. Games Crash the Most Uptime by iOS App Store Category Mobile Experience Benchmark 22 Findings • Games and Photo & Video have the highest crash rates • Health & Fitness and Business apps have lowest crash rates • The Lifestyle category (which includes many ecommerce apps) has a 1.5% crash rate Takeaway • Graphics intensive apps such as Games and Photo & Video crash most • Apps that enable transactions crash less, but the cost of downtime is significant. Pay close attention to the performance of these apps
- 23. Improving mobile app performance is critical Enterprises will sharpen their focus from web to mobile, and leverage apps to increase revenue and productivity. In addition, users will demand better performance from mobile apps. Companies that have consistently delivered top user experience use a mobile APM solution to optimize their apps. To learn more about Crittercism’s mobile app performance management solution, visit WWW.CRITTERCISM.COM
- 24. Real Business ROI Crittercism has worked with thousands of customers across multiple industries to improve their mobile apps and experience. Below is a glimpse into the achievements of customers. Mobile Experience Benchmark 24 ISSUE BEFORE AFTER IMPACT App performance issues impact $100K’s of mobile revenue/hour Poor visibility into app issues affecting users App performance issues affecting user experience Reactive approach to issues Long time to resolution for service issues 2 weeks to resolve issues causing downtime 6% downtime <0.02% crash rate Rapid troubleshooting of service issues Resolve most issues in 2 days <1% downtime 5 Star app ratings Improved revenue 14X more efficient at resolving critical crashes 2X faster app releases 7x performance improvement 6x revenue improvement Monetize through mobile channel while creating best user experience 3% downtime across all platforms < 0.5% downtime across all platforms 83% improvement in uptime 2 stars à 4+ stars
- 25. Mobile Experience Benchmark 25 Glossary & Methodology • MAU: Monthly Active Users are the unique end-users on a mobile app during a given month • DAU: Daily Active Users are the unique end-users on a mobile app on a given day • Responsiveness: Time it takes for each service call to respond • Mobile app uptime: Availability of the mobile app; percentage of app loads that do not result in a crash * • Crash rate: Percentage of app loads that result in a crash * • Error rate: Percentage of HTTP or network errors an app experiences while making cloud service API calls • Service: One or more endpoints providing a function that can be accessed by clients over the web. Example: s3.amazonaws.com • Endpoint: Specific API on a cloud service. Examples include account creation, check-out, or in-app purchase Methodology We’ve gathered data from 1B users in over 120 countries and analyzed 3B+ events/day over a month. *Mobile app uptime/crash metrics in this report are most accurate when used within the context of the platforms they reside on as there are differences in how platforms report app loads.