Designing a Mobile Device Lab - XBOSoft Webinar
1 like1,008 views
The document outlines a webinar hosted by Xbosoft focusing on mobile device testing and the creation of a mobile device lab to improve software quality. It highlights the challenges and strategies in mobile testing, including the importance of using real devices over emulators and the need for specialized testing environments. Key speakers include industry experts discussing their experiences and insights into mobile quality processes and analytics for user understanding.
Designing a Mobile Device Lab - XBOSoft Webinar
- 1. with Costa Avradopoulos Designing a Mobile Device Lab #MobileDeviceLab
- 2. XBOSoft Dedicated to software quality improvement Founded in 2006 We speed products to market with our expert: • Software QA consulting • Software testing Global team with offices in San Francisco,Beijing and Amsterdam “Thorough, accurate and fast” XBOSoft, Inc. All Rights Reserved. 2
- 3. House Rules Participants other than the speakers are muted Questions via the GoToWebinar control on the right side of your screen or through Twitter @XBOSoft Questions may be asked throughout the webinar - we’ll try to answer them at the end You will receive info on recording after the webinar XBOSoft, Inc. All Rights Reserved. 3 Webinar Hashtag: #MobileDeviceLab
- 4. Meet Our Speakers Philip Lew CEO and Founder, XBOSoft • Relevant specialties and passions o Software quality process, evaluation, measurement and improvement o Software quality in use / UX design o Mobile User Experience and usability o Cycling and travel XBOSoft, Inc. All Rights Reserved. 4 Karla Smith Marketing Manager, XBOSoft • Emails you about our industry hot topic webinars • Invites you to download our latest trends in software testing whitepapers XBOSoft Team
- 5. Guest Presenter Costa Avradopoulos XBOSoft, Inc. All Rights Reserved. 5 • Over 25 years’ experience in development, from requirements, to delivery, and process engineering • Roles in Development, QA, Automation, Product Management, VP, and CTO. Management of systems up to 15 million users in verticals such as Telecommunications, Financial, Wireless, Transportation, and Retail. • Much of career dedicated to the mobility space, culminating with a breakthrough patented mobile technology • Frequent speaker. Writing book due to be published soon: Winning Mobile Strategies – Bridging the Quality Gap • Principal for Avracom. Advises organizations in the creation of winning mobile strategies to develop best of breed applications. • Certified Six Sigma and TPI costa@avracom.com www.avracom.com
- 6. Why We’re Here XBOSoft, Inc. All Rights Reserved. 6 • Mobile applications – disruption in the market and in the SDLC • Mobile testing requires a different approach • A good strategy starts with designing a good mobile device lab • With thousands of variations, traditional approaches are insufficient
- 7. Agenda XBOSoft, Inc. All Rights Reserved. 7 • Mobile Testing Challenges • Defining Lab Requirements • Building a World Class Device Lab Out of Scope for Today: • Discussion on specific tools • Automation
- 8. XBOSoft, Inc. All Rights Reserved. 8 How Do You Define Mobile? 1 ABLE TO MOVE OR BE MOVED FREELY OR EASILY. 2 OF OR RELATING TO CELLULAR PHONES, HANDHELD COMPUTERS, AND SIMILAR TECHNOLOGY. Smartphones, Feature phones, tablets, laptops, ruggedized devices, medical devices, wearables, kiosks, ATMs…
- 9. XBOSoft, Inc. All Rights Reserved. 9 World Quality Report 2014 KEY FINDINGS: The QA function is adapting to business demands by streamlining and centralizing its structure, to achieve efficiency and cost optimization Significant investment in Test Environments may also be undermined by a lack of specialist expertise Testing’s late engagement in the application delivery lifecycle and its reliance on basic IT metrics is possibly applying a negative brake on this increasing maturity Mobile testing has increased in importance as a key discipline within the function, but lacks specialized methods, expertise and environment
- 10. XBOSoft, Inc. All Rights Reserved. 10 World Quality Report 2014 KEY FINDINGS (Cont.): Cloud adoption and cloud-based testing has slowed as organizations continue to manage concerns regarding data security and performance A higher proportion of overall IT budget is being invested in testing and focused on transformation projects Agile development is now widely adopted but still gives rise to problems for testing, particularly in relation to specific methodologies and expertise NOTE: Mobile Testing is a dedicated section in this report To obtain a copy of this report, email costa@avracom.com
- 11. Mobile Testing Areas of Focus - 2014 Source: World Quality Report, 6th edition 11 XBOSoft, Inc. All Rights Reserved.
- 12. The Cost of Poor Quality Escalates 65%OVER of all apps have a 1 star rating or less Top quality issues in last 12 months exceed $2 Billion in losses: • Security: breach at top 10 major retailer ($570M+ loss) • Performance/Outages: Top 3 social media company – 5 major outages in 2014, and 2 outages in 2015 • Security: breach at top 10 major retailer ($500M+ loss) • Functional: Top 3 bank, double payments issue 12 XBOSoft, Inc. All Rights Reserved.
- 13. Impact of Failure Cost of Poor Quality is Accelerated with Mobile Visibility is far greater, exposing your brand & reputation: • Social Media makes failures exponentially more visible, with potential to go viral • App store is a public report card - you cannot hide or cover up poor ratings A plethora of choices for customers = competitive risks • Near zero switching costs for customers • 60% of users will abandon your app/site if does not load in 3 seconds • Less than half will return to give a slow app/site a 2nd chance 13 XBOSoft, Inc. All Rights Reserved.
- 14. Mission Critical Low Criticality Mobile Testing Challenges Security Usability Performance Networks Fragmentation Due to involvement of GSM/ CDMA carriers identification of threats, risk and determining its business impact is crucial Thorough understanding of Visual, Accessibility and certification testing is a must Due to the limited hardware architecture of mobile devices, success and failure of application in market depends on performance of the app Testing to be performed on different networks like 3G, 4G, Wi-Fi and Edge, etc When you factor in different OS versions, unique hardware models, and carrier versions, the permutations are in the thousands Tools Specialized knowledge of tools specific to mobile testing Lack of Awareness There are many mobile tools and test approaches available in market, but all are not mature 14 XBOSoft, Inc. All Rights Reserved.
- 15. Internal Cloud Definitions of Cloud What do we mean by: PUBLIC CLOUD A mobile lab hosted by the testing tool provider. All cloud clients have access to the devices in the lab. A specific tester reserves the device he needs and uses it when its available. Accessible through the internet. PRIVATE CLOUD A mobile lab hosted by the testing tool provider dedicated to a specific client. The tester can reserve a device without having to share the devices with other clients. Accessible through the internet. ON-PREMISE CLOUD A secure mobile lab set up by the client itself. All the devices are connected to a server inside the company network. The company has complete control over what devices are procured and who uses them. External Cloud 15 XBOSoft, Inc. All Rights Reserved.
- 16. Shared Advantages • Enables better collaboration between Dev/QA • Both provide very robust lab/device management, supporting provisioning and maintenance • Resource management. Both provide a mechanism for scheduling/reserving devices • Improve efficiency of test execution • Enable continuous integration for DevOps 16 XBOSoft, Inc. All Rights Reserved.
- 17. Using an External Cloud Pros: • Scalability - easy access to hundreds of devices on demand • No need to maintain devices (provider does it) • Newer devices added to the pool at no extra cost • Flexible for more advanced solutions (ex. GPS) Cons: • Tester session performance tends to be slower, especially with higher bandwidth apps like video • Some security concerns –Testing done outside the firewall, while data is transferred back and forth through the internet • Different companies use the same devices with your app and potentially your private test data 17 XBOSoft, Inc. All Rights Reserved.
- 18. Using an On-premise Cloud Types: • Commercial administration via cart/hub • DIY lab Pros: • Higher performance real-time user sessions • Increased security over devices and test data • Allows users to pull devices from lab for hands-on tests • DIY is least expensive Cons: • Limited scalability • Maintaining devices requires a lab manager • Procuring/retiring devices can be challenging • DIY is most difficult to maintain and physically secure 18 XBOSoft, Inc. All Rights Reserved.
- 19. Emulators Benefits: • Most cost effective • Least infrastructure setup • Flexible and Scalable • Support for most test plans Constraints: • Cannot execute Usability, Performance, & Security test cases on emulators • Network and hardware interrupt testing cannot be done on emulators Mobile Testing Comparison Physical Devices in Lab Benefits: Covers all test types Safest option Should be used for most common device types Best for UAT / Usability Constraints: Very costly to support across all OS, device, and version combinations Procurement and setup of new devices can delay testing Physical Devices in Cloud Benefits: Cost effective Fastest start up Flexible and Scalable Can cover approx 90% of test plan Constraints: Latency during usage and longer test execution time Lower level of security excludes this as an option for some clients XBOSoft, Inc. All Rights Reserved.
- 20. Defining Lab Requirements XBOSoft, Inc. All Rights Reserved. 20
- 21. Use Real Mobile Devices the degree to which a test system matches a production system, including hardware, firmware, devices, software, appliances, networks, etc. TEST INTEGRITY Emulators/Simulators are tempting due to ease of use and low cost, however: Proven to cause significant defect leakage (stats & peers) Lowest level of test integrity Certain functions cannot be replicated 21 XBOSoft, Inc. All Rights Reserved.
- 22. Other Device Lab Considerations LAB SETUP WILL DEPEND ON: Test objectives on coverage Manual vs. automation Testing types Security Budget Scale Location of Teams 22 XBOSoft, Inc. All Rights Reserved.
- 23. POLL Approximately how many unique models of mobile devices exist today? A. 5,000 B. 10,000 C. 15,000 D. 20,000 E. 35,000 Assumptions: • We are actually looking for approximate total # of permutations • Therefore a device with the same model number, but on a different carrier, counts as more than 1 “model” or instance • Total active devices in use today globally, including feature phones, tablets, wearables, but excluding IoT/embedded (cars, kiosks, appliances) XBOSoft, Inc. All Rights Reserved. 23
- 24. Test Coverage Device type (handheld, tablet, kiosk) Screen size & resolution – dozens just in the top 80% Device models – 20,000+ unique variations exist! Operating system – 27 variations in the top 85% Connection type (USB, WiFi, Bluetooth, Cellular) Mobile Web vs. App Variance in carrier installations – 750 carriers globally Even using 80/20 rule, you are left with thousands of permutations! So what do we do? 24 XBOSoft, Inc. All Rights Reserved.
- 25. Use Analytics to Understand Your Users Implicit Analytics: Platform usage and user behavior Omniture Google Analytics Flurry TestFlight (monitor behavior) Mixpanel (native vs. mobile web) Localytics 25 XBOSoft, Inc. All Rights Reserved.
- 26. Use Analytics to Understand Your Users Xyo Applause (external, sentiment) Appsfire MeltwaterHootSuite Explicit Analytics: What your users say 26 XBOSoft, Inc. All Rights Reserved.
- 28. Customer Feedback: Great place to find those hidden test scenarios! 1. Social Media (Facebook, LinkedIn Groups, Twitter, etc.) 2. App reviews 3. App ratings / comments 4. Comments, forum posts and articles on the web 5. Complaints and support requests 6. Trending analytics (ex via Applause) Ensuring Proper Test Coverage – Rounding it Out XBOSoft, Inc. All Rights Reserved.
- 29. Lab Definition Exercise XBOSoft, Inc. All Rights Reserved. 29 First step – review analytics The data says our customers are: • 55% on iOS – 62% iPhone 6, 15% iPhone 6+, 23% iPad • 43% on Android – 45% on Samsung S5, 20% on S6, 20% Note 5, 15% Tab S (tablet) • 2% on Other (Windows/Blackberry)
- 30. Lab Definition Exercise XBOSoft, Inc. All Rights Reserved. 30 Additional considerations: • 8 screen sizes/resolutions in top 80% • 12 OS versions in top 80% • 4 major U.S. carriers • 1,000 functional test cases
- 31. Lab Definition Exercise XBOSoft, Inc. All Rights Reserved. 31 Tier iOS Android Other Carriers Screens Tests Execution Total 6 6 2 4 8 1,000 N/A Tier 1 4 4 2 3 5 1,000All tests Tier 2 3 3 0 2 3 750 Mission Critical Tier 3 2 2 0 1 2 400 Light regression Tier 4 1 1 1 1 1 100 Exploratory/ Smoke
- 32. Test Coverage Sample By Phase XBOSoft, Inc. All Rights Reserved. 32 Source: Perfecto Mobile
- 33. Test Coverage Sample By Index XBOSoft, Inc. All Rights Reserved. 33 Source: Perfecto Mobile
- 34. Test Coverage Sample By OS XBOSoft, Inc. All Rights Reserved. 34 Source: Perfecto Mobile
- 35. Building a World Class Device Lab XBOSoft, Inc. All Rights Reserved. 35
- 36. Overview of Device Lab Components Desktop Users U.S. or India Device Management Interface (View device inventory, reserve, etc.) Real-time Device Viewer (launch sessions) Remote Control protocol (access to launch apps, perform gestures, text, etc.) Device interface (USB hub, Desktop/Server, Proprietary Appliance) Real Devices, On-premise or Cloud 36 XBOSoft, Inc. All Rights Reserved.
- 37. U.S. Test Team Firewall Example On-premise Cloud setup India Test Team Mobile Devices on-premise Cloud Device access via desktop tool Wi-Fi Access Point Carrier Network Device access via desktop tool Mobile Devices on-premise Cloud Wi-Fi Access Point Carrier Network Firewall XBOSoft, Inc. All Rights Reserved. 37
- 38. U.S. Test Team Firewall Example Public Cloud setup India Test Team Mobile Devices Off-site Cloud B Device access via desktop tool Wi-Fi Access Point Carrier Network Device access via desktop tool Mobile Devices Off-site Cloud A Wi-Fi Access Point Carrier Network Firewall XBOSoft, Inc. All Rights Reserved. 38
- 39. Mobile Device Lab Recap XBOSoft, Inc. All Rights Reserved. 39 • Define mobile for your organization • Create thorough test objectives • Weigh all factors when defining lab requirements • Decide on Cloud vs. On-premise • Define test coverage by leveraging analytics • Monitor channels for continuous feedback • Use tier-based approach to optimize coverage
- 40. Post your questions on Twitter and we'll answer them @XBOSoft Join us to keep updated on all our webinars, reports and whitepapers: facebook.com/xbosoft +xbosoft linkedin.com/company/xbosoft We post regularly on our blog – check us out! http://xbosoft.com/software-quality-blog/ Download our free Whitepapers, available here: http://xbosoft.com/knowledge-center/ You will receive an email with information on today’s webinar slides and recording. Please email us with any further queries regarding our services or ideas for future webinars! Services@xbosoft.com Q+A www.xbosoft.com XBOSoft, Inc. All Rights Reserved.