Innovate 2013 session 1243 mobile testing.v3
1 like1,115 views
This document provides a summary of a presentation on addressing challenges in mobile application testing. It discusses how mobile application testing is different than traditional web testing due to factors like device fragmentation, new capabilities to test, and more network considerations. It also outlines what mobile testers need, including test automation, device cloud access, test planning and reporting tools, and the ability to test various parts of a mobile solution like the backend systems and network. The presentation was given by representatives from IBM and AT&T.
![46
Acknowledgements and disclaimers
© Copyright IBM Corporation 2013. All rights reserved.
– U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
IBM, the IBM logo, ibm.com, Rational, the Rational logo, Telelogic, the Telelogic logo, Green Hat, the Green Hat logo, and other IBM products
and services are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or
both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these
symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may
also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at “Copyright and
trademark information” at www.ibm.com/legal/copytrade.shtml
If you have mentioned trademarks that are not from IBM, please update and add the following lines:
[Insert any special third-party trademark names/attributions here]
Other company, product, or service names may be trademarks or service marks of others.
Availability: References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries
in which IBM operates.
The workshops, sessions and materials have been prepared by IBM or the session speakers and reflect their own views. They are provided
for informational purposes only, and are neither intended to, nor shall have the effect of being, legal or other guidance or advice to any
participant. While efforts were made to verify the completeness and accuracy of the information contained in this presentation, it is provided
AS-IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise
related to, this presentation or any other materials. Nothing contained in this presentation is intended to, nor shall have the effect of, creating
any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license
agreement governing the use of IBM software.
All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may
have achieved. Actual environmental costs and performance characteristics may vary by customer. Nothing contained in these materials is
intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue
growth or other results.](https://image.slidesharecdn.com/innovate2013-session1243mobiletesting-v3-130613083252-phpapp02/85/Innovate-2013-session-1243-mobile-testing-v3-46-320.jpg)
Innovate 2013 session 1243 mobile testing.v3
- 1. Addressing the challenges of mobile application testing Leigh Williamson (@leighwillia), IBM Distinguished Engineer Charles Rankin (@mtgofiend), IBM Rational CTO Team Balsu Thandu, Director AT&T Developer Program Session 1243 © 2013 IBM Corporation
- 2. 2 Please note the following IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion. Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion. Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.
- 3. 3 The Broadest Portfolio of Mobile Solutions The Deepest Set of Services Expertise New Industry Partnerships and Resources for Developers The Industry’s Most Comprehensive Mobile Portfolio
- 4. 4 The IBM MobileFirst offering portfolio AnalyticsSecurityManagement IBM and Partner Applications Banking Insurance Transport Telecom Government Industry Solutions HealthcareRetail Automotive Application & Data Platform Strategy&DesignServices Development&IntegrationServices Cloud & Managed Services Devices Network Servers
- 5. 5 Scan & Certify Instrument Test IntegrateObtain Insight Manage Deploy Design & Develop The IBM mobile application development lifecycle
- 6. 6 Quality is critical for mobile applications Customer facing (B2C) applications –‘Face of the Business’ –Engage customer personally and drive loyalty –User experience is key to brand perception Employee facing (B2E) applications –Increase worker productivity –Speed decision making and action –Poor user experience compromises investment No more 1 star mobile apps !!
- 7. 7 How is mobile application testing different? Device platforms, fragmentation, and growth – iOS, Android, BlackBerry, Windows Phone, Symbian, etc. – Multiple form factors and screen resolutions – Device refresh every 24 months New capabilities to test – Camera, GPS, direction, orientation, voice, etc. – Data and power consumption More network considerations – Multiple carriers, variable throughput and latency, disconnected use, network switching, etc. User experience is more critical
- 8. 8 Middle Tier ServerClient Tier Devices Back-end Data & Services Testing involves much more than just the mobile front-end Testing the end-to-end mobile solution
- 9. 9 What do mobile testers need?
- 10. 10 Test planning, reporting, and dashboarding Test case/suite/environment construction Integration with ALM Integration with multiple execution engines What do mobile testers need?
- 11. 11 Still most used mobile testing method Necessary for some validation Integration with test management Frequently key in outsourcing and crowd sourcing scenarios What do mobile testers need?
- 12. 12 Isolate front-end from back-end Control over data and interface Decrease expensive set-up costs What do mobile testers need?
- 13. 13 Network connections Server data Client data Injections and data flow analysis What do mobile testers need?
- 14. 14 Middle-tier and back-end Mobile client code Mobile network Mobile client experience What do mobile testers need?
- 15. 15 Device / platform agnostic Application instrumented automation Optical / image recognition automation What do mobile testers need?
- 16. 16 Remotely accessible Shareable across teams / geographies Access to diverse selection Device clouds minimize cost of ownership What do mobile testers need?
- 17. 17 IBM / Rational Testing Solution Middle Tier ServerClient Tier Devices Back-end Data & Services Test automation on device Mobile device cloud access RTW/RTVS Simulation of mid-tier or backend systems Test automation of mid-tier or back-end systems RQM Test Planning Configuration testing Test Construction Test Execution Team Collaboration Reporting & Dashboard IBM AppScan
- 18. 18 Scan & Certify Instrument Test IntegrateObtain Insight Manage Deploy Design & Develop The IBM mobile application development lifecycle
- 19. AT&T Application Resource Optimizer (ARO) Balsu K. Thandu Director, AT&T Developer Program
- 20. How Fast is Your App?
- 21. How Fast is Your App?
- 22. How Fast is Your App? “Zombies appear to be incapable of running. The fastest have been observed to move at a rate of barely one step per 1.5 seconds.” – Max Brooks, Zombie Survival Guide
- 23. How Important is Speed Anyway? Traditional Web (PC) Stats: •Yahoo! 400ms delay leads to 5-9% drop in traffic (1) •1 second of latency (2) – Customer satisfaction drops 16% – Conversion to sales drops 7% •10 Golden Principles of Successful Web Apps: – “Speed is always the most important feature” (3) “If speed is important on the web, it follows that speed is even more important for mobile” •71 % of users expect mobile sites to be as fast as desktop (4) (1) slideshare.net/stoyan/don't-make-me-wait-or-building-highperformance-web-applications (2) http://econsultancy.com/us/blog/10195-the-need-for-online-speed-in-america (3) 10 Golden Principles Of Successful Web Apps http://www.businessinsider.com/10-golden-principles-successful-web-apps-2010-3 (4) http://econsultancy.com/us/blog/9162-the-importance-of-speed-for-mobile-commerce
- 24. What Do Mobile Users Expect? High Performance Apps •Speed – 64% of mobile users expect pages/apps to load in <4 seconds (1) •Battery Life – Better Battery life (6.1 satisfied of 10) !!! (2) – 72% rate phone as very good or excellent • Only 21% rate battery life as very good or excellent (3) •Reasonable data usage – No one wants to hit their cap (1) http://www.keynote.com/docs/reports/Keynote-2012-Mobile-User-Survey.pdf (2) http://www.businessnewsdaily.com/2200-smartphone-customer-satisfaction-battery-life.html (3) Personal communication from Swiftkey study
- 25. What Do Mobile Users Get? Our Customers Expect More!!!! • Top Mobile frustrations “web page slow to load” (1) • Battery life top reported problem (2) (1) http://www.keynote.com/docs/reports/Keynote-2012-Mobile-User-Survey.pdf (2) http://www.businessnewsdaily.com/2200-smartphone-customer-satisfaction-battery-life.html
- 26. As a Developer, What Can I Do? • Learn How Mobile is Different • Optimize: – Learn the best practices – Test your application – Fix issues before your customers discover them • Get ahead of the game – be ready for industry standards and grading systems It can’t be MY developers, they’re the best • AT&T’s Application Resource Optimizer (ARO) – http://Developer.att.com/ARO – In 2012, <2% of tested apps required no optimization
- 27. How is Mobile Different? Internet IP applications Mobile applications Wireless State Machine Wireless State Machine Mobile apps often do not consider the uniqueness of the wireless network Optimized mobile applications improve user experience with improved battery life and perceived faster speed
- 28. It’s All Connected Application traffic triggers RRC State transitions: •RRC State transitions control
- 29. It’s All Connected Application traffic triggers RRC State transitions: •RRC State transitions control – Speed/User Experience
- 30. It’s All Connected Application traffic triggers RRC State transitions: •RRC State transitions control – Speed/User Experience – Device battery life
- 31. It’s All Connected Application traffic triggers RRC State transitions: •RRC State transitions control – Speed/User Experience – Device battery life – Network efficiency
- 32. AT&T Application Resource Optimizer: Develop Mobile Apps that run Faster, Smarter and Cleaner! AT&T Application Resource Optimizer (ARO) is a diagnostic tool created to help developers and testers understand how their app is behaving on any wireless network Enabling a richer network-connected smartphone experience for your customers • Improved battery life • Faster response times • Efficient network handling ARO is Free, Cross Platform and Open Source -an essential tool for all mobile testers and developers ARO Analyzer
- 33. How Does ARO Work? Transfer Trace Files Process TraceTest Your Application • PCAP only: • Network traffic collected • Native Collector adds: • Video • User Input • Peripheral Usage • CPU usage • Applies Radio and Power models to data • Graded against Best Practices • Learn how your application ACTUALLY behaves!
- 34. So, How Does ARO Make My App Faster? ARO is the test kit to identify areas where performance improvements exist Simple, common sense development best practices – Reducing connection times – Caching files – Eliminating Errors The fixes identified by ARO will tune your application to higher performance and speed
- 35. What Platforms Can I Test With? All of them!!! • All Platforms: Pcap/tcpdump network trace • Ex: Wi-Fi Hotspot (like Connectify) and Wireshark/NetMon to collect • iOS: Remote Virtual Interface uses Instruments to collect pcap over 3G/LTE • Android and Windows 8: Native Collector • Note: Android collector requires root
- 36. Everyone Wins With ARO! “Using ARO we were able to cut the network traffic of our iOS application down by ~30%!”
- 37. Everyone Wins With ARO! “You will find that the new offerings from Zynga will do much better with respect to battery and network usage.” Words With Friends 40% less energy 200% less signaling Faster, more efficient user experience
- 38. Everyone Wins With ARO! “Users concerned about data consumption and application performance.” After only 1 week of development 35% faster downloads 70GB/month less data usage 20% reduced battery usage
- 39. Everyone Wins With ARO • Faster Response Times • Improved Battery Life • Efficient Data Plan Usage • Enriched User Experience End User • App-specific Analysis • Key Areas to Improve • Checks performance in ways not usually available • Cross Platform and Network Agnostic Developer/Tester • Increased Network Availability • Signaling Load Reduction • Efficient Spectrum Usage • Improved Radio Availability Wireless Carrier
- 40. Rational and ARO a Winning Pair for Mobile Application Testing Working inside Rational, ARO can now be a part of your day to day testing processes.
- 41. © 2013 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. 41 AT&T Developer Program http://developer.att.com Communications Newsletter Email Blog Social Media Contests & Events Developer Summit Mobile App Hackathons Contests Fun Events Online API Platform AT&T Application Resource Optimizer (ARO) Development Resources Device Specs SDKs & Tools Community & Support Forums Webcasts Live Chat Technical Support
- 43. 43
- 44. 44 Daily Apple TV giveaway Complete your session surveys online each day at a conference kiosk or on your Innovate 2013 Portal! Each day that you complete all of that day’s session surveys, your name will be entered to win the daily Apple TV! On Wednesday be sure to complete your full conference evaluation to receive your free conference t-shirt!
- 45. 45 © Copyright IBM Corporation 2013. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in these materials may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. IBM, the IBM logo, Rational, the Rational logo, Telelogic, the Telelogic logo, and other IBM products and services are trademarks of the International Business Machines Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others.
- 46. 46 Acknowledgements and disclaimers © Copyright IBM Corporation 2013. All rights reserved. – U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. IBM, the IBM logo, ibm.com, Rational, the Rational logo, Telelogic, the Telelogic logo, Green Hat, the Green Hat logo, and other IBM products and services are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at “Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml If you have mentioned trademarks that are not from IBM, please update and add the following lines: [Insert any special third-party trademark names/attributions here] Other company, product, or service names may be trademarks or service marks of others. Availability: References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. The workshops, sessions and materials have been prepared by IBM or the session speakers and reflect their own views. They are provided for informational purposes only, and are neither intended to, nor shall have the effect of being, legal or other guidance or advice to any participant. While efforts were made to verify the completeness and accuracy of the information contained in this presentation, it is provided AS-IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this presentation or any other materials. Nothing contained in this presentation is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.